AWS Catastrophic Data Loss

1337

Bad one for the customer but for Amazon it's nothing. They make billions anyway.

I'd like to know how it's even possible that none of their backup generators started when they are supposed to be tested on a regular basis.

PhlipElder

@Pete-S The generators kicked in but eventually failed. Something else up there.

wrx7m

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

dafyre

Everybody say it with me: Backups are important! Backups are important!

FATeknollogee

Cloud is 10000% reliable....it never fails, it's the cloud. Don't need no stinkin' backup, it's in the cloud!

PhlipElder

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

Dashrender

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

How many people backup their O365 systems? I am willing to bet VERY few!! yet, if MS were to have the same issue, customers would find themselves in a similar situation.

One (invalid) claim I see from time to time when migrating to the cloud - it saves money because backups are part of the solution... which we can see here is definitely not the case.

PhlipElder

@Dashrender said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

How many people backup their O365 systems? I am willing to bet VERY few!! yet, if MS were to have the same issue, customers would find themselves in a similar situation.

One (invalid) claim I see from time to time when migrating to the cloud - it saves money because backups are part of the solution... which we can see here is definitely not the case.

Veeam was one of the first ones on the block to back up O365. That's messaging that Microsoft has not made clear but I've seen in the grapevine as far as the customer being responsible to do so.

No. My sh#t on their sh#t means no sh#t if something takes a sh#t.

IRJ

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

dafyre

@PhlipElder said in AWS Catastrophic Data Loss:

No. My sh#t on their sh#t means no sh#t if something takes a sh#t.

I'm totally stealing this! ROFL!

IRJ

@PhlipElder said in AWS Catastrophic Data Loss:

Veeam was one of the first ones on the block to back up O365. That's messaging that Microsoft has not made clear but I've seen in the grapevine as far as the customer being responsible to do so.

Customer is always responsible for data.

PhlipElder

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

IRJ

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

That is actually one of the biggest things you look at before chosing a vendor. If you do any cloud training, you will hear about data and data all over again. Being able to export it in a valuable way is essential for on prem or cloud though. If you cant export on prem and use the data in a usuable way into another system, you have the same problem.

IRJ

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

wrx7m

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Microsoft's central US DC failure, I think it was last year or early this year, cause a substantial amount of data loss as well. Not sure if any HA setup could have saved them from what I recall.

Just because you don't know how to do it or that you are supposed to do it, doesn't mean it isn't your responsibility.
Depends on the services. That should also be factored into the cost of deployment.
Absolutely correct. If it is important, it should be backed up by the customer. The customer should also make sure that they are storing said backups in a way that can't be affected by issues that would also cause data to be lost in the production environment.

PhlipElder

@IRJ said in AWS Catastrophic Data Loss:

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

I feel like I'm back in the early 2000s when Microsoft released Small Business Server 2000 then Small Business Server 2003 with the business owner DIY message. We got a lot of calls as a result of that messaging over the years.

Then, there was the mess created by the "IT Consultant" that didn't know their butt from a hole in the ground. We cleaned up a lot of those over the years.

At least in the above cases we could work with some sort of box to get their data on a roll.

Today, that possibility is virtually nil.

That is, the business owner being knowledgeable enough to navigate the spaghetti of cloud services setup to get to a point where they are secure and backed up for one. For another, as mentioned above, how many folks know how to set up any cloud?

Then, toss into the mix the message about speed and agility and we have a deadly mix beyond the SBS messaging and failures in that we're talking orders of magnitude more folks losing their businesses as a result of one big FUBAR.

Ever been on the back of a bike holding a case of beer while the "driver" hit 200+ KPH? I have. Once. And lived to never, ever, ever, trust an arse like that again.

wrx7m

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

That is actually one of the biggest things you look at before chosing a vendor. If you do any cloud training, you will hear about data and data all over again. Being able to export it in a valuable way is essential for on prem or cloud though. If you cant export on prem and use the data in a usuable way into another system, you have the same problem.

Exactly. Are you going to be trapped there because you can't get data migrated/transferred out to another service/platform? A lot of people don't think about this. These people should not be making the decisions to go with any vendor without knowing what questions to ask and how to use those answers to make decisions. Sadly, most don't know until it bites them in the ass.

PhlipElder

@wrx7m said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@wrx7m said in AWS Catastrophic Data Loss:

This was one AZ, right? If so, you need to design your environment to span multiple AZs, if not regions. This is beginner AWS design theory.

A few things come to mind:
1: Just how many folks know how to architect a highly available solution in any cloud?
2: At what cost over and above the indicated method does the HA setup incur?
3: It does not matter where the data is, it should be backed up.

Very valid points, but that is the responsibility of the customer.
Let's look at IaaS (EC2 instances loses EBS volumes after power outage) vs a cloud hosted service like Office 365.

Services are supposed to have HA built into them. Infrastructure has no HA built into it.

As you mentioned, DATA is ALWAYS responsibility of the customer.

One wee problem: Many "cloud" providers provide absolutely no mechanism to get the data out. Or, in some cases if it can be, it's not in usable form.

That is actually one of the biggest things you look at before chosing a vendor. If you do any cloud training, you will hear about data and data all over again. Being able to export it in a valuable way is essential for on prem or cloud though. If you cant export on prem and use the data in a usuable way into another system, you have the same problem.

Exactly. Are you going to be trapped there because you can't get data migrated/transferred out to another service/platform? A lot of people don't think about this. These people should not be making the decisions to go with any vendor without knowing what questions to ask and how to use those answers to make decisions. Sadly, most don't know until it bites them in the ass.

That's because most outside of our little tech bubble don't know what they want or what can bite them in the arse. I'm thinking of a video I saw a while back. SE Asia I think. Folks like me would wear steal toed boots while the locals knew where to look. Snake would jump out along the path to bite someone. Scared the carp out of me the first time I watched that.

FATeknollogee

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

I feel like I'm back in the early 2000s when Microsoft released Small Business Server 2000 then Small Business Server 2003 with the business owner DIY message. We got a lot of calls as a result of that messaging over the years.

Then, there was the mess created by the "IT Consultant" that didn't know their butt from a hole in the ground. We cleaned up a lot of those over the years.

At least in the above cases we could work with some sort of box to get their data on a roll.

Today, that possibility is virtually nil.

That is, the business owner being knowledgeable enough to navigate the spaghetti of cloud services setup to get to a point where they are secure and backed up for one. For another, as mentioned above, how many folks know how to set up any cloud?

Then, toss into the mix the message about speed and agility and we have a deadly mix beyond the SBS messaging and failures in that we're talking orders of magnitude more folks losing their businesses as a result of one big FUBAR.

Ever been on the back of a bike holding a case of beer while the "driver" hit 200+ KPH? I have. Once. And lived to never, ever, ever, trust an arse like that again.

The powerful power of marketing
Cloud = :couple_with_heart: :kissing_face_with_smiling_eyes: :kissing_cat_face_with_closed_eyes:

PhlipElder

@FATeknollogee said in AWS Catastrophic Data Loss:

@PhlipElder said in AWS Catastrophic Data Loss:

@IRJ said in AWS Catastrophic Data Loss:

For IaaS, using a tool like terraform can help you transition from one platform to another as terraform is compatible with many cloud hosts.

I feel like I'm back in the early 2000s when Microsoft released Small Business Server 2000 then Small Business Server 2003 with the business owner DIY message. We got a lot of calls as a result of that messaging over the years.

Then, there was the mess created by the "IT Consultant" that didn't know their butt from a hole in the ground. We cleaned up a lot of those over the years.

At least in the above cases we could work with some sort of box to get their data on a roll.

Today, that possibility is virtually nil.

That is, the business owner being knowledgeable enough to navigate the spaghetti of cloud services setup to get to a point where they are secure and backed up for one. For another, as mentioned above, how many folks know how to set up any cloud?

Then, toss into the mix the message about speed and agility and we have a deadly mix beyond the SBS messaging and failures in that we're talking orders of magnitude more folks losing their businesses as a result of one big FUBAR.

Ever been on the back of a bike holding a case of beer while the "driver" hit 200+ KPH? I have. Once. And lived to never, ever, ever, trust an arse like that again.

The powerful power of marketing
Cloud = :couple_with_heart: :kissing_face_with_smiling_eyes: :kissing_cat_face_with_closed_eyes:

Yes.

The The Great Firewall of Marketing has prevented most of the negative stuff from surfacing.