“Catastrophic” hack on email provider destroys almost two decades of data
-
The comments on the article are about why immutable offline backups are required for this. What email provider (I've never heard of VFEmail) around the size of this organization could afford this. Especially to keep 20 years worth of backups?
I do and agree that immutable offline backups are critical for the day to day operations but this seems awkward that the provider (especially for their free plan) would have this.
-
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
-
They offer free email accounts, they might have had a crazy number of free users and that might have been the problem. Although with a 50MB storage limit, it would take a lot of users to rack up any amount of storage needs.
The smarter thing would have been to at least backup the Silver, Gold, and Platinum customers to offline.
-
@Reid-Cooper said in “Catastrophic” hack on email provider destroys almost two decades of data:
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
The highest tier offered by VFEmail is $50/year
So the pricing scheme for their clients seems very low for a "for profit" business.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
They offer free email accounts, they might have had a crazy number of free users and that might have been the problem. Although with a 50MB storage limit, it would take a lot of users to rack up any amount of storage needs.
The smarter thing would have been to at least backup the Silver, Gold, and Platinum customers to offline.
Most of those users who using the free plan probably used for junk mail only.
-
@black3dynamite said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
They offer free email accounts, they might have had a crazy number of free users and that might have been the problem. Although with a 50MB storage limit, it would take a lot of users to rack up any amount of storage needs.
The smarter thing would have been to at least backup the Silver, Gold, and Platinum customers to offline.
Most of those users who using the free plan probably used for junk mail only.
I would assume, yeah. At best, they aren't using it for storing much!
-
@black3dynamite said in “Catastrophic” hack on email provider destroys almost two decades of data:
Most of those users who using the free plan probably used for junk mail only.
It seems like a junk email provider. Has anyone here actually heard of and used this provider?
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@Reid-Cooper said in “Catastrophic” hack on email provider destroys almost two decades of data:
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
The highest tier offered by VFEmail is $50/year
So the pricing scheme for their clients seems very low for a "for profit" business.
$50 / year is $4.16/mo. That's more than Microsoft's Hosted Exchange plan with 50GB of storage, and this one is only 20GB.
That's not just more, it is absurdly expensive. It's more money per GB available than any mainstream commercial email service.
And other services are offering special expensive features. Like Google bundles other services, Microsoft offers Exchange.
This needs none of that and could have been running on all open source and free components. It wasn't, as he admitted, but it had no need for any extra costs and certainly shouldn't cost more than normal hosted email.
So for people at those tiers, he should have been rolling in money to pay for backups.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@black3dynamite said in “Catastrophic” hack on email provider destroys almost two decades of data:
Most of those users who using the free plan probably used for junk mail only.
It seems like a junk email provider. Has anyone here actually heard of and used this provider?
No, did not know them previously and as @Dashrender knows from my looking at the SQRL website, this website does not give me the confidence to have considered using them.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@Reid-Cooper said in “Catastrophic” hack on email provider destroys almost two decades of data:
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
The highest tier offered by VFEmail is $50/year
So the pricing scheme for their clients seems very low for a "for profit" business.
$50 / year is $4.16/mo. That's more than Microsoft's Hosted Exchange plan with 50GB of storage, and this one is only 20GB.
That's not just more, it is absurdly expensive. It's more money per GB available than any mainstream commercial email service.
And other services are offering special expensive features. Like Google bundles other services, Microsoft offers Exchange.
This needs none of that and could have been running on all open source and free components. It wasn't, as he admitted, but it had no need for any extra costs and certainly shouldn't cost more than normal hosted email.
So for people at those tiers, he should have been rolling in money to pay for backups.
But my point is, based on the fact that "they weren't running on all free and open source options" means that their capital was going to something (likely a pocket). Rather than being invested in basic services (backup storage).
-
Assuming the 20GB users all filled their mailboxes completely (trust me, they don't), you get 50 accounts per TB of storage. Assuming you keep one full backup of the current system offline, and one "one week old" backup, on Wasabi that is just $10/mo for every 50 accounts. Or just 5% of total costs. That's super low to get two offline, immutable copies.
In the real world, utilization would normally be way lower, like 10GB average. So you'd get closer to 100 accounts per TB of storage, for just 2-3% overhead on backups.
Very, very affordable.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@Reid-Cooper said in “Catastrophic” hack on email provider destroys almost two decades of data:
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
The highest tier offered by VFEmail is $50/year
So the pricing scheme for their clients seems very low for a "for profit" business.
$50 / year is $4.16/mo. That's more than Microsoft's Hosted Exchange plan with 50GB of storage, and this one is only 20GB.
That's not just more, it is absurdly expensive. It's more money per GB available than any mainstream commercial email service.
And other services are offering special expensive features. Like Google bundles other services, Microsoft offers Exchange.
This needs none of that and could have been running on all open source and free components. It wasn't, as he admitted, but it had no need for any extra costs and certainly shouldn't cost more than normal hosted email.
So for people at those tiers, he should have been rolling in money to pay for backups.
But my point is, based on the fact that "they weren't running on all free and open source options" means that their capital was going to something (likely a pocket). Rather than being invested in basic services (backup storage).
All by choice, all flaunting that there was excess money to not even be worried about spending it well. If you can run SQL Server, you are rich and could have trivially afforded backups.
-
My guess is that some central thing was hacked. Like the password repository.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
You mean something like LASTPASS can be hacked?! Oh the humanity!
-
Basic system policy on any hosted platform, generally will require you (by default) to update your passwords regularly. And if you chose to disable that feature (or never change your passwords because you're too lazy) then all of the damage is on you.
-
Now if he had any scale, he could have been using tapes even cheaper. But you have to have enough scale to get into them.
-
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely has.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
Basic system policy on any hosted platform, generally will require you (by default) to update your passwords regularly. And if you chose to disable that feature (or never change your passwords because you're too lazy) then all of the damage is on you.
That's actually a bad practice. Good practice is to not do that and be less lazy and disable insecure policies, follow the industry (and finally) NIST guidelines to low change but high security passwords, but to never share them.
We don't know anything about his password policies other than that he had passwords different on different machines. So it wasn't something like an AD breach where one password gives you everything. But that all systems were hacked suggests either that there was some central repo that was hit, or the systems were uniformally out of patching.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely has.
Yeah, if he had ~3,000 paid accounts, then that would have been the way to go. Cheaper than Wasabi at that scale.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely had.
Fixed a typo
