Windows Server 2012 - Task Scheduler Issue
-
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
-
@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
Do you have backup jobs or VSS running at the same time? Could be a stun.
-
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.
These jobs, unfortunately, run queries against a SQL database that's local to the box.
It's definitely weird.
-
@wrx7m said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
Do you have backup jobs or VSS running at the same time? Could be a stun.
Not on this particular VM.
-
Upon further scrutiny of the Event Logs, I think I might be on to what is happening. I think it may be a clock issue.
We are an XCP-ng shop, and it seems that the XenServer guest agent is attempting to sync the clock every 30 minutes. It issues a sync which causes the time to shift anywhere from a few milliseconds to a few MINUTES.
Seems this is an issue other folks have complained about (though others seem to experience much wider variations in time).
The two jobs that were "skipped" on the 01/31 are scheduled to run at 7:00 PM. It looks like the system time was tweaked at 6:39 PM and 7:09 PM. So, in theory, the jobs should've fired, but this is definitely not helping things. These are events with the ID of 1, the "source" is Kernel-General, and the user is SYSTEM.
01/31 @ 6:39:18 PM:
The system time has changed to 2020-02-01T02:39:18.826000000Z from 2020-02-01T02:40:37.936724600Z.01/31 @ 7:09:17 PM:
The system time has changed to 2020-02-01T03:09:17.907000000Z from 2020-02-01T03:10:33.882678100Z.This eventually causes the Time Service to complain since it causes the time to be off by more than 5 seconds for a period of time.
Every time it seems to roll the clock back. I followed the guidance via the Citrix forum post linked above in modifying registry entries to stop it from doing this but it didn't seem to help. Supposedly modifying the following registry entries to be either "blank" or set to "Local" is supposed to stop the Xen agent time sync.
HKLM\Software\Citrix\XenTools\HostTime
HKLM\Software\Wow6432Node\Citrix\XenTools\HostTimeHowever, they were already blank in the registry. So I set them to "Local" and restarted all the services on the box that have Citrix or Xen in the name since I can't reboot this box without pre-planning (yay for 24/7 availability). I'm still seeing the Kernel-General Event ID 1's in the System event log. Fun...
The time on the hosts are correct, and my NTP hierarchy is as so:
Member Servers -> DCs -> Stratum 1 NTP servers hosted by Hurricane Electric (one in San Jose, CA and the other in New York, NY).
I combed through time stuff yesterday and the time on the DCs seems stable (and w32tm reports they are syncing properly), as well as on other hosts, but I'll need to give it a second look.
I am also planning on upgrading from XCP-ng 7.6 to 8.0 this weekend...I'll have to see if this behavior continues after that.
-
@anthonyh I made a post asking about how time synchronization with XenServer and VMs works.
https://mangolassi.it/topic/12085/time-synchronization-on-xenserver-hosts-and-vms -
@anthonyh I absolutely hate VM integration based time sync with the host across all platforms.
It is an absolute horrible idea that should have never been implemented.
-
The hosts and VMs were all using that same NTP server.
-
@JaredBusch said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh I absolutely hate VM integration based time sync with the host across all platforms.
It is an absolute horrible idea that should have never been implemented.
Agreed 100%!
-
The good news is that this does not seem to be happening on my Server 2016 DCs, and the Time Service event log entries report it's syncing with HE's stratum 1 servers regularly and without issue.
I do see corrections every now and then that may be coming from the XenServer guest agent, but it's only a couple of times every few days or so (sometimes longer), and they're at the millisecond level. Would prefer it to not do this at all (just let me manage time via NTP!), but I don't believe this would cause anything detrimental.
Minor adjustments like this:
The system time has changed to 2020-02-04T18:13:51.561000000Z from 2020-02-04T18:13:51.565339000Z.and
The system time has changed to 2020-01-24T20:05:36.893000000Z from 2020-01-24T20:05:36.892351300Z. -
There have been problems with Windows guests time variation up to many minutes and even hours.
5 minutes is all it takes to cause serious issues because then authentication fails.
https://discussions.citrix.com/topic/399684-xenserver-tools-timesync-disable/ -
There was an issue with US NTP servers multiple users reported during this week. Not sure if it might be the case here but still.
-
@Net-Runner said in Windows Server 2012 - Task Scheduler Issue:
There was an issue with US NTP servers multiple users reported during this week. Not sure if it might be the case here but still.
That's why anyone who relies on proper time sync should never use pool.ntp.org servers. They even spell it out on their website but nobody seems to read it.
