Office 365 Email Gone After Forced Logoff
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, your problem statement is "Exchange mailbox popped off user". If you have no USP updates, then that would be on the MS side. If you have USP entries, what do they say?
Yup, and that's how it manifested. And quickly it popped off all users. But then turned out to be only partially off, it went from the portal, but the system returned. Turned out to be issues with the partner team and how they applied some account details. But no one at MS could find that, for hours. And none of it was visible to us.
-
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry, because something had to tell your domain that it no longer has Exchange. That feeds back into the portal.office.com site, removing the Outlook link. You can't nuke your service without it.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
So someone ganked the O365 partner account, which propagated to the ntg.co domain? That was the issue?
Yeah, somewhere on the backend out of our reach or visibility.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Again, what do you see in AAD?
No activity in the logs from this morning until after the issue had started. And all the first logs are then clearly us trying to figure out what was wrong. So whatever they did, wasn't recorded in there.
-
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
-
Not to be obtuse but why do Zimbra instead of Zoho or another?
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
I’d take Zoho docs over LibreOffice any day.
-
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@stacksofplates said in Office 365 Email Gone After Forced Logoff:
Not to be obtuse but why do Zimbra instead of Zoho or another?
We've used Zoho in the past and it is pretty good. But given how we use systems and the resources that we have available, Zimbra seems a better fit. If we were going back to SaaS, Zoho would be the first choice, for sure.
Oh yeah. I forgot you had the big Scale cluster. I was thinking you were running on a VPS.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?
Oh oh, it's magic!
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
Logs are empty for two days before the issue. So whatever MS did, was not logged in a way visible to us.
I posted on here at 9:20am, 12 minutes before the first log entry in AAD.
Microsoft targeted you for funsies, left no trace in AAD, and you are without Outlook?
Oh oh, it's magic!
We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...
So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
We were, it is back now. Took a good half of the day with support to get them to the point that they were able to turn it back on.
Hey at least you didn't lose anything.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@psx_defector said in Office 365 Email Gone After Forced Logoff:
There will still be USP entries. Because AAD will still need to get something from the billing portal, portal.office.com, so something had to have been applied.
Again, what do you see in AAD? Because as my previous screenshot shows, it says who did what and exactly the stuff done. There will always be an entry,...
So this assessment must not be true. There are no entries. So whether MS makes changes that aren't reflected tehre, or lets us make chances that are not reflected there, nothing was reflected there. So the underlying premise that AAD will always tell us what has happened and that we can troubleshoot this ourselves is fundamentally incorrect. There was no data visible to us. Perhaps MS logs data that is visible only to you, I do not know.
Thinking about this on the drive home, since you have a partner account, it might be in another area.
Instead of going round and round with your non-stop conspiracy shit, I'm giving up. Given you didn't even know about AAD, I'm going with you don't know what you are doing. You don't have the right info in front of you and I'm not about to do your job.
You keep thinking MS is out to get ya.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
Instead of going round and round with your non-stop conspiracy shit, I'm giving up.
To what are you referring? You asked me to provide logs. You said that all actions must be in the logs. You got the logs that showed nothing. You are calling "showing you the logs that you requested" a conspiracy?
I don't follow. Please explain where conspiracy comes into the picture. I did what you asked. You alone are freaking out that you think that there is some coordinated plan to make things not work and make MS look bad or something. I just told you how it worked. That you were not familiar with O365 and Azure logs is not a conspiracy.
-
@psx_defector said in Office 365 Email Gone After Forced Logoff:
You keep thinking MS is out to get ya.
Um, only you said that. I said they had a problem. You seem to feel that anyone having any service problem from MS must mean that there is a conspiracy?
Those are your logs that are empty. My contention is that Microsoft doesn't know how to support their platforms. It's not a conspiracy, I just think MS isn't very competent. And being 100% convinced that the logs can't be wrong, finding them empty, and then freaking out that it must be a coordinated effort by customers to modify the logs kind of proves my point.
Just because someone isn't a masterful engineer does not a conspiracy make.
-
@aaronstuder said in Office 365 Email Gone After Forced Logoff:
@scottalanmiller Why are you still using Office 365 if you have so many issues with it?
We managed to get migrated off of it faster than MS was able to fix it
I'm super thiankful for this outage as the timing was perfect to make the decision to drop O365 immediately with all of the decision makers in the right place. And MS' response here really solidifies the decision. Not how you want a vendor responding. Everyone makes mistakes, but MS didn't handle the embarrassment well at all.So we are over on Zimbra now and things are great!
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
And MS' response here really solidifies the decision
I'm not going to get into your little spat but this point needs made.
An employee's personal response is not a vendor response. Pull your head out of your ass.
-
@jaredbusch said in Office 365 Email Gone After Forced Logoff:
An employee's personal response is not a vendor response. Pull your head out of your ass.
They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.
He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."
Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.
The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.
-
@scottalanmiller said in Office 365 Email Gone After Forced Logoff:
@jaredbusch said in Office 365 Email Gone After Forced Logoff:
An employee's personal response is not a vendor response. Pull your head out of your ass.
They are when they attack customers in public to try to make them look bad or to make the vendor look good. He's a Microsoft representative using the platform to try to defend Microsoft. Microsoft has a responsibility for that.
He is as much a part of the vendor as anyone else. He is Microsoft's only known representative on the community. All vendor responses are made by "an employee." Vendors can't get free passes just because their responses are from "an employee."
Other MS reps are invited here and could speak up if they don't want this being Microsoft's only response. But that it is Microsoft's response is what it is.
The response made was in no way outside of the scope of being an MS representative. It was very much an internal style response from an internal resource.
You could probably make that argument if he had a green badge. I doubt MS has agreed for him to be a representative of their environment and even know that someone has spoken in here that works for them. It’s a lot like people on Twitter, etc that have profiles that say “my opinions are my own” etc. Thy don’t officially speak for the company.
