“Catastrophic” hack on email provider destroys almost two decades of data
-
Assuming the 20GB users all filled their mailboxes completely (trust me, they don't), you get 50 accounts per TB of storage. Assuming you keep one full backup of the current system offline, and one "one week old" backup, on Wasabi that is just $10/mo for every 50 accounts. Or just 5% of total costs. That's super low to get two offline, immutable copies.
In the real world, utilization would normally be way lower, like 10GB average. So you'd get closer to 100 accounts per TB of storage, for just 2-3% overhead on backups.
Very, very affordable.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@Reid-Cooper said in “Catastrophic” hack on email provider destroys almost two decades of data:
Offline backups should not be that expensive. And the cost would generally scale with the size of the vendor. If you have ten mailboxes, it costs 10 * X. If you have a million, it is a million * X. But presumably they bill by the mailbox, so the cost of immutable backups would remain relatively close to the same per mailbox. So no matter how big or small, backups should have been able to be taken offline unless they were priced so low that backups could never be afforded at any scale.
The highest tier offered by VFEmail is $50/year
So the pricing scheme for their clients seems very low for a "for profit" business.
$50 / year is $4.16/mo. That's more than Microsoft's Hosted Exchange plan with 50GB of storage, and this one is only 20GB.
That's not just more, it is absurdly expensive. It's more money per GB available than any mainstream commercial email service.
And other services are offering special expensive features. Like Google bundles other services, Microsoft offers Exchange.
This needs none of that and could have been running on all open source and free components. It wasn't, as he admitted, but it had no need for any extra costs and certainly shouldn't cost more than normal hosted email.
So for people at those tiers, he should have been rolling in money to pay for backups.
But my point is, based on the fact that "they weren't running on all free and open source options" means that their capital was going to something (likely a pocket). Rather than being invested in basic services (backup storage).
All by choice, all flaunting that there was excess money to not even be worried about spending it well. If you can run SQL Server, you are rich and could have trivially afforded backups.
-
My guess is that some central thing was hacked. Like the password repository.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
You mean something like LASTPASS can be hacked?! Oh the humanity!
-
Basic system policy on any hosted platform, generally will require you (by default) to update your passwords regularly. And if you chose to disable that feature (or never change your passwords because you're too lazy) then all of the damage is on you.
-
Now if he had any scale, he could have been using tapes even cheaper. But you have to have enough scale to get into them.
-
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely has.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
Basic system policy on any hosted platform, generally will require you (by default) to update your passwords regularly. And if you chose to disable that feature (or never change your passwords because you're too lazy) then all of the damage is on you.
That's actually a bad practice. Good practice is to not do that and be less lazy and disable insecure policies, follow the industry (and finally) NIST guidelines to low change but high security passwords, but to never share them.
We don't know anything about his password policies other than that he had passwords different on different machines. So it wasn't something like an AD breach where one password gives you everything. But that all systems were hacked suggests either that there was some central repo that was hit, or the systems were uniformally out of patching.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely has.
Yeah, if he had ~3,000 paid accounts, then that would have been the way to go. Cheaper than Wasabi at that scale.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
LTO-8 supports up to 30T compressed storage per tape.
That's an insane amount of storage for what this provider likely had.
Fixed a typo
-
Basically if you are going to run a service of this nature, you probably want to build in the cost of immutable backups from the beginning. Just assume it is a required cost and build around it. Don't look at it years later and say "how do I afford this." You wouldn't say "SMTP costs too much, we will skip that", right? So the same with fully separate backups.
That said, if he had a central repository of passwords that was cracked as someone mentioned, they might have shut down any storage accounts elsewhere.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
Compromised by a weak password or something, probably.
Hacked? Unlikely.
-
@DustinB3403 said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
You mean something like LASTPASS can be hacked?! Oh the humanity!
Never has been yet.
-
@JaredBusch said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
Compromised by a weak password or something, probably.
Hacked? Unlikely.
Depends. Might have been just a notepad or something.
-
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
@JaredBusch said in “Catastrophic” hack on email provider destroys almost two decades of data:
@scottalanmiller said in “Catastrophic” hack on email provider destroys almost two decades of data:
My guess is that some central thing was hacked. Like the password repository.
Compromised by a weak password or something, probably.
Hacked? Unlikely.
Depends. Might have been just a notepad or something.
That is not hacked, that is giving it away.
-
Maybe it was a disgruntled former (or current) employee.
Anyway, what do you guys mean with offline backups?
Do you mean backups that are stored somewhere not connected to the net, like backup tapes in safe?For instance, in DBMS world "offline backup" is something completely different.
-
@Pete-S said in “Catastrophic” hack on email provider destroys almost two decades of data:
Maybe it was a disgruntled former (or current) employee.
Anyway, what do you guys mean with offline backups?
Do you mean backups that are stored somewhere not connected to the net, like backup tapes in safe?For instance, in DBMS world "offline backup" is something completely different.
An air gapped backup system, would have no connection (or credentials) to the production workload. Only during the time that a backup is being produced would there be some connectivity. IE An LTO tape is loaded and being written.
During all other time periods the Tape, the Controller, the credentials and everything that gets you "to the backups" are physically disconnected from everything else that is "production".
-
The most basic explanation is that there is no physical connection to the backup medium.
In this case, there was a connection, and that the backups were not stored offline.
-
@DustinB3403 But isn't then a tape library the only practical offline backup solution? And if you had the credentials to that, you could erase the tapes anyway.
-
@Pete-S said in “Catastrophic” hack on email provider destroys almost two decades of data:
@DustinB3403 But isn't then a tape library the only practical offline backup solution? And if you had the credentials to that, you could erase the tapes anyway.
No, not really. You can offload to rotating disks (Portable USBs for example), have detached offline storage providers and not keep the credentials in the same place as your production credentials.
