Unsolved Autoupdates Killed My Windows Server 2008 R2
-
I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual.
I'll spare the play by play, but basically 3 updates installed and after that the server was stuck in a loop where it would try to roll back the updates for 3 hours, and then reboot only to repeat the reboot loop. I restored from backup and now if I log in to the desktop, it pretty much locks up the server. If I don't log in to the desktop on the server, all the services run fine.
Any ideas on what to try? I can get in to safe mode, and I've tried a selective boot where 3rd party services were disabled.
Going forward I'd like to virtualize the server. Does anyone know if you can do a P2V migration without logging in to the desktop? My other thought was to restore from backup to a virtual machine. Has anyone tried that before with Veeam Endpoint? It would be simple enough to build a new domain controller, but I'm affraid when I get to the last step of dcpromoing the old server out, I won't be able since I can't log in to it. I could hack it out of AD with ADSIedit, but I'd rather avoid that if I could.
-
Updates that hosed it:
After those updates tried to install, this is the screen it would stay on for 3 hours before rebooting:
-
Can you get to a Safe mode command prompt and manually uninstall each update? ... or does it freeze on you then too?
-
@dafyre I tried that. KB890830 and KB4015549 would uninstall. KB4014981 failed to uninstall with this error:
It was at that point that I decided to just restore from last night's OS backup.
-
@dafyre So far I haven't had any problems logging in to safe mode.
-
The Windows update log had a ton of these errors:
2017-04-13 14:24:10:022 428 1614 Agent WARNING: Failed to evaluate Installed rule, updateId = {6CFCF2A8-52A0-4DDC-AD90-71A7736E83CE}.200, hr = 80080005 2017-04-13 14:26:10:109 428 1614 Agent WARNING: Failed to evaluate Installable rule, updateId = {6CFCF2A8-52A0-4DDC-AD90-71A7736E83CE}.200, hr = 80080005 2017-04-13 14:30:10:787 428 1614 Agent WARNING: Failed to evaluate Installed rule, updateId = {0C91D9FF-FEE6-421C-ACCD-15582919F140}.200, hr = 80080005 2017-04-13 14:32:10:846 428 1614 Agent WARNING: Failed to evaluate Installable rule, updateId = {0C91D9FF-FEE6-421C-ACCD-15582919F140}.200, hr = 80080005 2017-04-13 14:36:11:259 428 1614 Agent WARNING: Failed to evaluate Installed rule, updateId = {89185EE9-622B-4D77-9DF7-FE3F2E2027EE}.200, hr = 80080005 2017-04-13 14:38:11:326 428 1614 Agent WARNING: Failed to evaluate Installable rule, updateId = {89185EE9-622B-4D77-9DF7-FE3F2E2027EE}.200, hr = 80080005 2017-04-13 14:42:11:656 428 1614 Agent WARNING: Failed to evaluate Installed rule, updateId = {728F10D0-EFA2-494E-B116-FFCACBCF094C}.200, hr = 80080005 2017-04-13 14:44:11:715 428 1614 Agent WARNING: Failed to evaluate Installable rule, updateId = {728F10D0-EFA2-494E-B116-FFCACBCF094C}.200, hr = 80080005 2017-04-13 14:48:13:563 428 1614 Agent WARNING: Failed to evaluate Installed rule, updateId = {67BCDD15-BA51-4465-BED2-6202AD4A9D34}.201, hr = 80080005 2017-04-13 14:50:13:621 428 1614 Agent WARNING: Failed to evaluate Installable rule, updateId = {67BCDD15-BA51-4465-BED2-6202AD4A9D34}.201, hr = 80080005 -
What happens if you disconnect a network cable and then log in?
-
@dafyre said in autoupdates killed my server:
What happens if you disconnect a network cable and then log in?
I haven't tried that. I read about some people having that issue with Windows 7 machines, but didn't think it would apply to the server since it always has a connection to itself. I can give it a rip next time I'm on site.
-
Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.
-
@dafyre said in autoupdates killed my server:
Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.
In hind sight I wish I had brought a copy of the backup with me so I could try to restore to something on my bench. That's a good idea.
-
@dafyre said in autoupdates killed my server:
Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.
But with the old one running you will get conflicts. Power one off, power one on.
Delete the NIC so it doesnt preserve the MAC after the convert. new IP and rename it, then you can have both running. -
@Mike-Davis said in autoupdates killed my server:
I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual.
You can sense the impending disaster right away.
-
@scottalanmiller said in Autoupdates Killed My Windows Server 2008 R2:
@Mike-Davis said in autoupdates killed my server:
I've had auto updates hose stuff before, but this one takes the cake. I took on a client with a Dell T310 server with Server 2008 R2 Standard on it. It's physical, not virtual.
You can sense the impending disaster right away.
That is why I always like to check backups before rebooting something that is unkown or not been rebooted in a while.
-
@Texkonc said in autoupdates killed my server:
That is why I always like to check backups before rebooting something that is unkown or not been rebooted in a while.
That's the odd thing. I do have backups and I was able to restore to a point 6 hours before the automatic updates went on. It isn't stuck in a boot loop, but I can't get to the desktop. It's possible that I haven't logged in to the desktop since the last auto update went on. I could try rolling back even further, but it would be better to do that on a test server.
-
@Texkonc said in Autoupdates Killed My Windows Server 2008 R2:
@dafyre said in autoupdates killed my server:
Only other thing I can think of would be to restore it to a hypervisor (and leave it disconnected) to see if that has any bearing on it... and then, you could restore older backups until you find one that works.
But with the old one running you will get conflicts. Power one off, power one on.
Delete the NIC so it doesnt preserve the MAC after the convert. new IP and rename it, then you can have both running.That's why I said leave it disconnected. So the VM doesn't try to take over... at least not right away!
-
Possibly failed update from before that are just sticking around and that is why the last company did not reboot the server? Maybe it is spiraled worse since you took over, but it was left over from the last guy.
-
@Mike-Davis does terminal server connection works?? what about windows offline updater http://download.wsusoffline.net/
-
Have you checked the CBS.log file in %windir%\logs\cbs
It will give more verbose errors usually. -
Update for all those that suggested ideas. I took @dafyre 's idea to restore it to a hyper visor. I went on site and I'm not sure why, but it took like 11 hours to copy the backups 1TB+ to an external USB drive. I brought that back to my office and started the restore. That took about 7 hours each time I did that. The first time I just restored the
drive. After messing around with bcdedit I still couldn't get the thing to boot. Veeam said that the M: drive was a system drive to, so I created another VM and restored the and M: drive. This time I could boot to the Dell system installer setup, but still couldn't boot the OS.Then I decided to restore to another physical Dell server I had on the bench. It booted no problem. Veeam boots you to Directory Services Restore mode and then you have to use msconfig to tell it to do a normal boot and you're good. I did that and it seemed like it was having the same issue where I logged in and it showed the desktop but wouldn't respond to the keyboard. The mouse moved, but wouldn't let me click. I just left it and came back an hour later. At that point it was fine. Not sure what the deal was.
There were a few variables to take in to consideration. Since the NIC was different, none of the network services came up. I also disabled a few things like CrashPlan because when the NIC does come online, I don't want it to try to backup to the cloud since it's a clone of the real server that is still on production.
At this point I'm going to try to P2V the thing.
-
NOT 100% Sure,
but if you go here:
C:\Windows\Installer
Can you try to sorted it by latest modified date, then run the MSI and see if some can be repaired on uninstalled ?
