Massive Azure Outage and No Support
-
@coliver said:
@scottalanmiller said:
@Dashrender said:
It's not just you who is down, right?
Just us. 100% down. Epic Azure failure. Took out everything in every region associated with us.
Sounds like they just took your account and switched it off. Does this include Office365? Or is that not hosted on Azure?
They did, but only Azure. O365 is fine. They made a mistake and gave us three accounts, didn't merge them like they were supposed to, shut off one or two of them so our account is still up and fine, but all of the services on it are gone. We never made more than one account. It's total MS screw up end to end.
Same issues caused an O365 outage two weeks ago and email data loss. I still am missing most of my email from the last two years.
-
So our account is on and "fine". But they associated our Azure stuff with another account that they turned off. So it gets really complicated. The core issue is that they associated our Azure systems with the wrong account.
-
@scottalanmiller said:
So our account is on and "fine". But they associated our Azure stuff with another account that they turned off. So it gets really complicated. The core issue is that they associated our Azure systems with the wrong account.
Oh man... that isn't good.
-
Wow ,..... Just,....wow.
-
What's worse is I've been warning that they were going to screw this up for months. For about six months we've been trying to get them to fix their account issues because we knew that they were getting this all messed up and were going to do something stupid. Their account managers kept saying that everything was "fixed" or everything was "fine."
Now we've been down for over eight hours and they refused to even talk to the technical people until the account was fixed. The account that was "fixed" for the last six months?
I am so not happy. That they didn't fix the account before this when I've been warning and warning them that this was risky makes this impossible to be an accident. They decided to play fast and loose with our account because they were lazy or didn't care. Everyone who works there leaves so quickly they probably thought (and were probably right) that it would be "someone else's problem" when it fell apart.
-
Did you have some customers hosted on Azure or was this exclusively NTG stuff?
-
@coliver said:
Did you have some customers hosted on Azure or was this exclusively NTG stuff?
Customers too.
-
@scottalanmiller said:
@coliver said:
Did you have some customers hosted on Azure or was this exclusively NTG stuff?
Customers too.
That really isn't good.
-
I'm told that they have decided that it is okay to turn them back on now. We will see.
-
@scottalanmiller said:
I'm told that they have decided that it is okay to turn them back on now. We will see.
Waiting,....with baited breath.
-
Stuff like this makes me glad we have (most) of our junk internal. So much trust to put in another company!
Edit: advantages to cloud are obvious, don't get me wrong.
-
I've run into the multiple accounts on a vendors backend problem before.
Hell Disney has this problem too. If you register at their streaming movies website it never asks you to create a username, of course this leads you to the assumption that your email address is your username. But if you setup an account on store.disney.com they ask for a user or an email address.
This combined with the fact that their authentication process actually all goes through GO.COM left me without access to my store account for 4 days. Additionally since I use LastPass, Lastpass wasn't able to log me in correctly because of the whole damned GO.COM problem, because not all logon prompts provide the correct information as to where the credentials are going, so LP didn't offer me the needed information.
-
@MattSpeller said:
Stuff like this makes me glad we have (most) of our junk internal. So much trust to put in another company!
Edit: advantages to cloud are obvious, don't get me wrong.
On-prem for the WIN!!!!
-
@MattSpeller said:
Stuff like this makes me glad we have (most) of our junk internal. So much trust to put in another company!
Edit: advantages to cloud are obvious, don't get me wrong.
In this case, from what I know... it's not the cloud that has teh issue... it's the Account manager and Billing department.
-
@gjacobse Account managers and billing departments and the janky support are all part and parcel of it. I'd argue it's 100% a cloud issue. Just not a technical IT issue.
-
@MattSpeller said:
@gjacobse Account managers and billing departments and the janky support are all part and parcel of it. I'd argue it's 100% a cloud issue. Just not a technical IT issue.
It's a Microsoft issue. Anyone seen something like this with Rackspace, Amazon, Digital Ocean, etc.? This is twice in a month with MS on an issue like this.
-
Now, to buy time, they are claiming that they are turning the systems on. But it has been fifteen minutes and not a single one is back on yet. We are at nine hours down now and no answers, no fixes, no nothing.
-
@scottalanmiller Totally agree it's MS that screwed up, just pointing out that this is a little talked about issue with cloud computing. Maybe it should be called "Trust in Other Companies" computing. The other companies you mention still have billing departments and account managers and they're all human and capable of (unintentionally) giving you a real bad day. How do you build reliability into the cloud from outside?
-
@MattSpeller said:
@scottalanmiller Totally agree it's MS that screwed up, just pointing out that this is a little talked about issue with cloud computing. Maybe it should be called "Trust in Other Companies" computing. The other companies you mention still have billing departments and account managers and they're all human and capable of (unintentionally) giving you a real bad day. How do you build reliability into the cloud from outside?
Well one option is crossing cloud boundaries, a lot of companies do this. They host half on Amazon and half on Rackspace, for example.
-
You can have multiple accounts with a single provider. Unfortunately Azure causes problems with this and you can't safely use that technique with Microsoft because of the whacky authentication stuff that they try to do. Their authentication is why this all happened. They get so weird about how that stuff works that even they can't figure out which account is which.
