Windows Server 2012 - Task Scheduler Issue
-
What's killing me is this only impacts some tasks. Other tasks that run as the SYSTEM account that do similar functions (execute a PowerShell script that does a SQL query and ships the result to its respective destination) are fine.
I don't know how the underlyings of the Task Scheduler work, but a WAG says that something on the back end is corrupt/broken. I dunno if there is a "rebuild" Task Scheduler sort of function.
-
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:
Well, create a new task with the same settings and see if that gives the same issue.
Yeah, I think I may just do that. Though I'd really like to know 1) what happened (?!?!), and 2) if there is a better fix than to just re-do things, heh.
-
How many tasks are we talking about?
I would probably export them, delete and import just to be certain.
And make sure task history is enabled.
-
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
Has this only happened once?
Nope. It's been happening for at least a week or so. I've finally pulled a majority of my hair out and made a post here.
There is always the possibility that it is the task scheduling service that fails or one of it's dependencies.
You could create a new task that triggers a simple script that just writes the current time in a log file.
If the task service fails to run you would know and also be able to see when.There are some settings under the service as well what the service should do if it fails. Might want to check those as well.
If the Task Scheduler service was failing, wouldn't that be written to the Event Log? If so, it's not.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:
Well, create a new task with the same settings and see if that gives the same issue.
Yeah, I think I may just do that. Though I'd really like to know 1) what happened (?!?!), and 2) if there is a better fix than to just re-do things, heh.
Well, you could add into the script at the end, to remove the scheduled task and create it again, followed by a verification of such, and to email you if it fails.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
Has this only happened once?
Nope. It's been happening for at least a week or so. I've finally pulled a majority of my hair out and made a post here.
There is always the possibility that it is the task scheduling service that fails or one of it's dependencies.
You could create a new task that triggers a simple script that just writes the current time in a log file.
If the task service fails to run you would know and also be able to see when.There are some settings under the service as well what the service should do if it fails. Might want to check those as well.
If the Task Scheduler service was failing, wouldn't that be written to the Event Log? If so, it's not.
Well, we are kind of assuming that we are dealing with something that has changed or is not the way it should. Either a software bug, a change made by windows updates, insufficient privileges or something of that nature.
-
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
How many tasks are we talking about?
I would probably export them, delete and import just to be certain.
And make sure task history is enabled.
Looks like at least 4 tasks are having this issue (there may be some more, I need to do another pass through the scheduled tasks). I'm going to try an export/delete/import and see what happens...
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
How many tasks are we talking about?
I would probably export them, delete and import just to be certain.
And make sure task history is enabled.
Looks like at least 4 tasks are having this issue (there may be some more, I need to do another pass through the scheduled tasks). I'm going to try an export/delete/import and see what happens...
Don't import the old task. Create it new. Just set the same options, but new start date.
-
Wow, so I just went through all the scheduled tasks on this box.
There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
Wow, so I just went through all the scheduled tasks on this box.
There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.
Perhaps export them all - then find out how to re-create them using powershell. Then you'll have a backup for them incase you need to migrate this to another server, etc for the future.
-
@Dashrender said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
Wow, so I just went through all the scheduled tasks on this box.
There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.
Perhaps export them all - then find out how to re-create them using powershell. Then you'll have a backup for them incase you need to migrate this to another server, etc for the future.
Not a bad idea.
-
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Pete-S said in Windows Server 2012 - Task Scheduler Issue:
How many tasks are we talking about?
I would probably export them, delete and import just to be certain.
And make sure task history is enabled.
Looks like at least 4 tasks are having this issue (there may be some more, I need to do another pass through the scheduled tasks). I'm going to try an export/delete/import and see what happens...
Don't import the old task. Create it new. Just set the same options, but new start date.
Makes no sense. They will be exactly the same regardless. If you want another start date (which makes no difference) just change it in the xml.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Dashrender said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
Wow, so I just went through all the scheduled tasks on this box.
There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.
Perhaps export them all - then find out how to re-create them using powershell. Then you'll have a backup for them incase you need to migrate this to another server, etc for the future.
Not a bad idea.
You could write a simple function for a script to recreate them all based on the exported XMLs.
-
This is how you create tasks from the xml files with powershell.
-
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
-
@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.
-
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
Do you have backup jobs or VSS running at the same time? Could be a stun.
-
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.
These jobs, unfortunately, run queries against a SQL database that's local to the box.
It's definitely weird.
-
@wrx7m said in Windows Server 2012 - Task Scheduler Issue:
@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.
Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!
I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....
Do you have backup jobs or VSS running at the same time? Could be a stun.
Not on this particular VM.
-
Upon further scrutiny of the Event Logs, I think I might be on to what is happening. I think it may be a clock issue.
We are an XCP-ng shop, and it seems that the XenServer guest agent is attempting to sync the clock every 30 minutes. It issues a sync which causes the time to shift anywhere from a few milliseconds to a few MINUTES.
Seems this is an issue other folks have complained about (though others seem to experience much wider variations in time).
The two jobs that were "skipped" on the 01/31 are scheduled to run at 7:00 PM. It looks like the system time was tweaked at 6:39 PM and 7:09 PM. So, in theory, the jobs should've fired, but this is definitely not helping things. These are events with the ID of 1, the "source" is Kernel-General, and the user is SYSTEM.
01/31 @ 6:39:18 PM:
The system time has changed to 2020-02-01T02:39:18.826000000Z from 2020-02-01T02:40:37.936724600Z.01/31 @ 7:09:17 PM:
The system time has changed to 2020-02-01T03:09:17.907000000Z from 2020-02-01T03:10:33.882678100Z.This eventually causes the Time Service to complain since it causes the time to be off by more than 5 seconds for a period of time.
Every time it seems to roll the clock back. I followed the guidance via the Citrix forum post linked above in modifying registry entries to stop it from doing this but it didn't seem to help. Supposedly modifying the following registry entries to be either "blank" or set to "Local" is supposed to stop the Xen agent time sync.
HKLM\Software\Citrix\XenTools\HostTime
HKLM\Software\Wow6432Node\Citrix\XenTools\HostTimeHowever, they were already blank in the registry. So I set them to "Local" and restarted all the services on the box that have Citrix or Xen in the name since I can't reboot this box without pre-planning (yay for 24/7 availability). I'm still seeing the Kernel-General Event ID 1's in the System event log. Fun...
The time on the hosts are correct, and my NTP hierarchy is as so:
Member Servers -> DCs -> Stratum 1 NTP servers hosted by Hurricane Electric (one in San Jose, CA and the other in New York, NY).
I combed through time stuff yesterday and the time on the DCs seems stable (and w32tm reports they are syncing properly), as well as on other hosts, but I'll need to give it a second look.
I am also planning on upgrading from XCP-ng 7.6 to 8.0 this weekend...I'll have to see if this behavior continues after that.
