Need advise, restoring domain controller and email server.
-
I screwed up and need help!
We have two locations, each have its own domain controller and they are in the same forest. Site A DC just host DC's stuff, DHCP, DNS, and etc. Site B's DC is also a file server, print server, as well as all the dc server's stuff. We have an in house exchange 2010 server at Site A. All three servers are virtual ESXi guest running Server 2008R2, and we also have other servers join to the domains, but they are application servers.
This morning, I noticed the exchange server have only 5% of free space left, I tried to free up some space by deleting old administrators users inbox, but exchange will not let me saying I don't have the permission to do so. Following https://community.spiceworks.com/topic/386021-cannot-delete-ad-user-insufficent-rights-or-protected-from-accidental-deletion
"Tommy6473 Jan 14, 2014 at 4:11 PM 1ST POST
I had the same issue where "Protect object from accidental deletion" was NOT checked, and I still couldn't delete.Here was the fix I found: In AD Users and Computers, Go to the object's properties > security tab > click Advanced > click "Restore Defaults".
Close out the windows, then try deleting again."
It says to go to the AD, then the object, then security page and reset the security to default, then delete the users. I somehow went the AD security for the object's folder and change the security to "default". Suddenly, 90% of the exchange email inbox were gone from the exchange server.
After tracing my steps to figure out what happened, I restored DC1 to yesterday 7pm backup, then left it off. Use DC2 for DNS so users have at least have internet. I'm in the process of restoring Exchange to the yesterday's 7pm backup now, which will take several more hours before it finish.
My plan is to bring up DC1, then bring up Exchange, make sure they are good to go and have no issue, then I will probably have to rejoin all users computers and applications server to the domain.
Questions I have are,
Does my plan of action sounds alright? We only have 35 users, so rejoin domain won't be too bad.
What do I do with DC2?
How would DC2 react to this restore DC1?
How do I prevent the AD security settings on DC2 to go to DC1? Would they even sync?
How do I keep the file share settings on DC2?
Any things I am missing?Any and all advise are welcome.
Thank you -
Would it be a better idea to make DC2 seize the FSMO roles, then copy the security settings from the offline DC1 to DC2, then bring up the Exchange. Once Exchange is verify working, create another server and join it to the domain then promote it to DC. Then use DC2 to do a Force demote on a the offline DC1?
-
Well this is a pretty bad situation.
Personally I would open up a ticket with Microsoft for $250 and see what their recommendations are. -
Also, remember that all of your member server's running applications are just like your workstations. Depending upon if the security principle changed they may or may not need to be reacted to the restore domain
-
@Dashrender said in Need advise, restoring domain controller and email server.:
Well this is a pretty bad situation.
Personally I would open up a ticket with Microsoft for $250 and see what their recommendations are.URL ?
-
It's generic
-
@Dashrender said in Need advise, restoring domain controller and email server.:
Well this is a pretty bad situation.
Personally I would open up a ticket with Microsoft for $250 and see what their recommendations are.New price is now at $499.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
This morning, I noticed the exchange server have only 5% of free space left, I tried to free up some space by deleting old administrators users inbox, but exchange will not let me saying I don't have the permission to do so. Following https://community.spiceworks.com/topic/386021-cannot-delete-ad-user-insufficent-rights-or-protected-from-accidental-deletion
Strange, because you later say:
I'm in the process of restoring Exchange to the yesterday's 7pm backup now, which will take several more hours before it finish.
Which usually fixes the disk space issue. I'm guessing you are using Veeam or some other VM level backup rather than the proper application level backup which truncates the logs on the Exchange EDB.
On the plus side, assuming your restore works correctly and most of the stuff is in place, you might not need to do much more than that. But knowing the Exchange part of this, I'm guessing you also didn't backup your AD correctly either for an authoritative restore.
You are in for a rough night my friend. First and foremost, if the user objects have not been deleted, don't start now. Exchange ties into the GUID, so not having it will cause your restore to pretty much be useless. If you already deleted them, you are looking at some really messed up stuff. If you have good backups, you might have to straight up restore to a previous point in time completely, that means nuking everything. If you had a block level backup, you might be almost OK then, assuming you don't have too much of a delta between the machines.
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
-
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
I spent 90 minutes on the phone and for just Microsoft so say setup is unsupported.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
Contact Microsoft, even at the worst paying $500 will help you out immensely when you need to do some more advanced AD stuff to restore.
I spent 90 minutes on the phone and for just Microsoft so say setup is unsupported.
That's kind of how MS roles. Support is not their forte. Highly recommended if you feel that support is critical, MS is not the place to be depending on.
-
The conclusion was to abandon Site A's DC1. Make Site B's DC2 a primary DC and seize the FSMO since I screwed up by changing permission, screwed up even more by taking out DC1 and restore it from a VM image backup and MS couldn't help.
Then, I talked to my old boss and an idea came to me, "What if I just call MS for help with Exchange and nothing else." So I spent just over 3 hours on the phone with the support guy from India, Neel Kamal Sharma Engineer -Microsoft Enterprise Communication Support. He was VERY knowledgeable about Exchange. Then we found out the Exchange 2010 we have is only a SP2, which is not supported by MS. He upgrade it to SP3, which took over 90 minutes, then he reconnected the lost mailboxes, set the permissions, and Exchange was running fully again. I restored what I can from GFI archiver so minimal lost on emails.
Though this incident, I learned
-
All our VM guest servers are backed thru Barracuda, which takes snap shot then backup the server. Great for an application server. It does NOT work for DC because once you restore the DC, the GUID changes, and AD on that DC will be broken.
-
Our way of backing up Exchange created logs that continue to take up storage space since the backup does not delete the logs.
-
Hosting internal Exchange can be very dangerous thing, since one wrong permission change can wipe out your entire mailboxes.
and many other things.
I still got a few things I need to fix, but at least email is working now.
Then, I will be proposing some new recommendations to management from what I learn.
Thanks for all those who helped. -
-
Great to hear you got an MS tech who was willing to help.
This has been my experience as well. They seemed to bend over backwards to assist in resolving my issues.
Sadly - it seems Scott has not had this experience.
-
@Dashrender said in Need advise, restoring domain controller and email server.:
Great to hear you got an MS tech who was willing to help.
This has been my experience as well. They seemed to bend over backwards to assist in resolving my issues.
Sadly - it seems Scott has not had this experience.
It's like playing whack-a-mole, they do have some good people. The problem is, it's only some, and trying to find a good one is always difficult.
shibboleet
-
On the first call, I was trying to get the Active Directory permission restored from the VM image base backup, then I though I can bring the restored exchange back online. It is just not possible. I tried it, and AD failed immediately with just the restored DC1 being on. It's not the tech's fault for my screw ups.
Second call was to only focusing on Exchange Server, and that ended well.
-
you can definitely put yourself into a situation where they can't/won't help you.. but that's normally because you started doing things you shouldn't and you've actually made the call to support to late. This you only have yourself to blame.
I'm guessing if this type of thing ever happens again, the first thing you'll do is call and open a ticket, either with a support company or Microsoft direct.
-
@Harry-Lui said in Need advise, restoring domain controller and email server.:
- All our VM guest servers are backed thru Barracuda, which takes snap shot then backup the server. Great for an application server. It does NOT work for DC because once you restore the DC, the GUID changes, and AD on that DC will be broken.
All VM backup solutions work by making snapshots to freeze the disk state that is going to be backed up.
Why your restore changed some GUID is an unrelated issue.
I have backed up and restored a DC, many, many times over the years and never had this kind of problem.
Hell, I have done a backup and restore of a SBS server with zero issues other than the inbound email from the point in time of the snapshot being lost.
