Posts made by anthonyh

anthonyh

@pete-s said in MariaDB/MySQL Table Repair Eating TONS of disk space:

@anthonyh said in MariaDB/MySQL Table Repair Eating TONS of disk space:

I have a MariaDB database that is of moderate size (~48 GB) and two of the tables within this database have corruption. One of the offending tables is reportedly ~25 GB in size. When running REPAIR TABLE, the process eats up all the remaining disk space (which admittedly wasn't a lot, 14 GB).

I started growing the disk and gave it an additional 50 GB. The repair process gobbled it up. Gave it 50 GB more, repair process gobbled it up. Gave it 100 GB more (for a total of 200 GB additional disk), and the repair process still gobbled it all up. I'm a little baffled as to why and am wondering if anyone here can shed some light as to why a 25GB table would need over 200GB of working space for a repair?

The database is using MyISAM storage (I know, I'm working on it) and the table in question stores data (document images) in longblob format. From what I understand, MyISAM table repairs basically involve reconstructing the table, so I wouldn't be surprised at needing at least an equal amount of free space to the size of the table. This behavior has me a little confused.

I assume there are no database clients trying to access the database during your repair.

I don't know the answer to your question but have you had a look at the different settings for repair table?
https://dev.mysql.com/doc/refman/5.7/en/repair-table-optimization.html

Check myisam_max_sort_file_size for instance. If it's larger than your disk then it could outgrow the available space.

You are correct. No clients are accessing the database. I'm doing the repair on a clone of the actual host.

I will investigate myisam_max_sort_file_size and report back.

anthonyh

I have a MariaDB database that is of moderate size (~48 GB) and two of the tables within this database have corruption. One of the offending tables is reportedly ~25 GB in size. When running REPAIR TABLE, the process eats up all the remaining disk space (which admittedly wasn't a lot, 14 GB).

I started growing the disk and gave it an additional 50 GB. The repair process gobbled it up. Gave it 50 GB more, repair process gobbled it up. Gave it 100 GB more (for a total of 200 GB additional disk), and the repair process still gobbled it all up. I'm a little baffled as to why and am wondering if anyone here can shed some light as to why a 25GB table would need over 200GB of working space for a repair?

The database is using MyISAM storage (I know, I'm working on it) and the table in question stores data (document images) in longblob format. From what I understand, MyISAM table repairs basically involve reconstructing the table, so I wouldn't be surprised at needing at least an equal amount of free space to the size of the table. This behavior has me a little confused.

anthonyh

The good news is that this does not seem to be happening on my Server 2016 DCs, and the Time Service event log entries report it's syncing with HE's stratum 1 servers regularly and without issue.

I do see corrections every now and then that may be coming from the XenServer guest agent, but it's only a couple of times every few days or so (sometimes longer), and they're at the millisecond level. Would prefer it to not do this at all (just let me manage time via NTP!), but I don't believe this would cause anything detrimental.

Minor adjustments like this:

The system time has changed to ‎2020‎-‎02‎-‎04T18:13:51.561000000Z from ‎2020‎-‎02‎-‎04T18:13:51.565339000Z.

and

The system time has changed to ‎2020‎-‎01‎-‎24T20:05:36.893000000Z from ‎2020‎-‎01‎-‎24T20:05:36.892351300Z.

anthonyh

@JaredBusch said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh I absolutely hate VM integration based time sync with the host across all platforms.

It is an absolute horrible idea that should have never been implemented.

Agreed 100%!

anthonyh

Upon further scrutiny of the Event Logs, I think I might be on to what is happening. I think it may be a clock issue.

We are an XCP-ng shop, and it seems that the XenServer guest agent is attempting to sync the clock every 30 minutes. It issues a sync which causes the time to shift anywhere from a few milliseconds to a few MINUTES.

https://discussions.citrix.com/topic/404819-xenapppvs-weired-time-sync-problems-after-ctx235405-xeniface-windows-io-driver-821123/

Seems this is an issue other folks have complained about (though others seem to experience much wider variations in time).

The two jobs that were "skipped" on the 01/31 are scheduled to run at 7:00 PM. It looks like the system time was tweaked at 6:39 PM and 7:09 PM. So, in theory, the jobs should've fired, but this is definitely not helping things. These are events with the ID of 1, the "source" is Kernel-General, and the user is SYSTEM.

01/31 @ 6:39:18 PM: The system time has changed to ‎2020‎-‎02‎-‎01T02:39:18.826000000Z from ‎2020‎-‎02‎-‎01T02:40:37.936724600Z.

01/31 @ 7:09:17 PM: The system time has changed to ‎2020‎-‎02‎-‎01T03:09:17.907000000Z from ‎2020‎-‎02‎-‎01T03:10:33.882678100Z.

This eventually causes the Time Service to complain since it causes the time to be off by more than 5 seconds for a period of time.

Every time it seems to roll the clock back. I followed the guidance via the Citrix forum post linked above in modifying registry entries to stop it from doing this but it didn't seem to help. Supposedly modifying the following registry entries to be either "blank" or set to "Local" is supposed to stop the Xen agent time sync.

HKLM\Software\Citrix\XenTools\HostTime
HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime

However, they were already blank in the registry. So I set them to "Local" and restarted all the services on the box that have Citrix or Xen in the name since I can't reboot this box without pre-planning (yay for 24/7 availability). I'm still seeing the Kernel-General Event ID 1's in the System event log. Fun...

The time on the hosts are correct, and my NTP hierarchy is as so:

Member Servers -> DCs -> Stratum 1 NTP servers hosted by Hurricane Electric (one in San Jose, CA and the other in New York, NY).

I combed through time stuff yesterday and the time on the DCs seems stable (and w32tm reports they are syncing properly), as well as on other hosts, but I'll need to give it a second look.

I am also planning on upgrading from XCP-ng 7.6 to 8.0 this weekend...I'll have to see if this behavior continues after that.

anthonyh

@wrx7m said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.

Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!

I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....

Do you have backup jobs or VSS running at the same time? Could be a stun.

Not on this particular VM.

anthonyh

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.

These jobs, unfortunately, run queries against a SQL database that's local to the box.

It's definitely weird.

anthonyh

So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.

Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!

I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....

anthonyh

@Dashrender said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

Wow, so I just went through all the scheduled tasks on this box.

There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.

Perhaps export them all - then find out how to re-create them using powershell. Then you'll have a backup for them incase you need to migrate this to another server, etc for the future.

Not a bad idea.

anthonyh

Wow, so I just went through all the scheduled tasks on this box.

There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.

anthonyh

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

How many tasks are we talking about?

I would probably export them, delete and import just to be certain.

And make sure task history is enabled.

Looks like at least 4 tasks are having this issue (there may be some more, I need to do another pass through the scheduled tasks). I'm going to try an export/delete/import and see what happens...

anthonyh

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh

Has this only happened once?

Nope. It's been happening for at least a week or so. I've finally pulled a majority of my hair out and made a post here.

There is always the possibility that it is the task scheduling service that fails or one of it's dependencies.

You could create a new task that triggers a simple script that just writes the current time in a log file.
If the task service fails to run you would know and also be able to see when.

There are some settings under the service as well what the service should do if it fails. Might want to check those as well.

If the Task Scheduler service was failing, wouldn't that be written to the Event Log? If so, it's not.

anthonyh

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

Well, create a new task with the same settings and see if that gives the same issue.

Yeah, I think I may just do that. Though I'd really like to know 1) what happened (?!?!), and 2) if there is a better fix than to just re-do things, heh.

anthonyh

What's killing me is this only impacts some tasks. Other tasks that run as the SYSTEM account that do similar functions (execute a PowerShell script that does a SQL query and ships the result to its respective destination) are fine.

I don't know how the underlyings of the Task Scheduler work, but a WAG says that something on the back end is corrupt/broken. I dunno if there is a "rebuild" Task Scheduler sort of function.

anthonyh

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

Is the system fully up to date as much as it can be?

Yep! That's actually what the reboot was for last Thursday. Windowwwwwws updatesssssss.

The problem has been happening before that, unfortunately.

anthonyh

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh

Has this only happened once?

Nope. It's been happening for at least a week or so. I've finally pulled a majority of my hair out and made a post here.

anthonyh

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:
@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

Depends on if they are running right now. Anyway, how long does the task take to complete? Do your scripts produce a log file?

They are all "Ready" as of this moment. They run nightly, so I should see them as "Running" right now.

Some of them do, some of them do not (I've been adding Start/Stop Transcript to them as I've needed to troubleshoot). For those that do log, the log's last modified date/time corresponds to the last time the Task Scheduler reports the task ran.

I just edited my original post to add that the script I talk about as an example does log, and the log has not been touched since Task Scheduler reported it ran successfully.

OK, post screenshot of each tab under that task's properties, including what's under the edit button on the "triggers" tab.

Just export the xml here, clean any sensitive info. Easier.

Here is the XML of the task I mentioned in my OP. I can do screenshots too if that's still desired by anyone.
<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.4" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2015-12-23T13:40:10.4492565</Date>
    <Author>[REDACTED]</Author>
  </RegistrationInfo>
  <Triggers>
    <CalendarTrigger>
      <StartBoundary>2015-12-23T21:00:00</StartBoundary>
      <Enabled>true</Enabled>
      <ScheduleByWeek>
        <DaysOfWeek>
          <Monday />
          <Tuesday />
          <Wednesday />
          <Thursday />
          <Friday />
        </DaysOfWeek>
        <WeeksInterval>1</WeeksInterval>
      </ScheduleByWeek>
    </CalendarTrigger>
  </Triggers>
  <Principals>
    <Principal id="Author">
      <UserId>S-1-5-18</UserId>
      <RunLevel>LeastPrivilege</RunLevel>
    </Principal>
  </Principals>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <DisallowStartOnRemoteAppSession>false</DisallowStartOnRemoteAppSession>
    <UseUnifiedSchedulingEngine>false</UseUnifiedSchedulingEngine>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>P3D</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>powershell</Command>
      <Arguments>-ExecutionPolicy Bypass -File "C:\scripts[REDACTED].ps1"</Arguments>
    </Exec>
  </Actions>
</Task>
Does it run if you right-click on the task and run it manually?

As per my OP:

"I can right click the offending jobs and run them manually no problem. "

anthonyh

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

What does the task history look like?

Looks empty.

anthonyh

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

Depends on if they are running right now. Anyway, how long does the task take to complete? Do your scripts produce a log file?

They are all "Ready" as of this moment. They run nightly, so I should see them as "Running" right now.

Some of them do, some of them do not (I've been adding Start/Stop Transcript to them as I've needed to troubleshoot). For those that do log, the log's last modified date/time corresponds to the last time the Task Scheduler reports the task ran.

I just edited my original post to add that the script I talk about as an example does log, and the log has not been touched since Task Scheduler reported it ran successfully.

OK, post screenshot of each tab under that task's properties, including what's under the edit button on the "triggers" tab.

Just export the xml here, clean any sensitive info. Easier.

Except that it is not really in clear text. Or do you what user "S-1-5-18" is?

FWIW it runs as the SYSTEM account.

anthonyh

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

@Pete-S said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

I am having an interesting problem with the Windows Task Scheduler under Server 2012. I have a couple dozen scheduled tasks that do various things via PowerShell scripts (mostly pull data from a SQL database then ship it off to their respective recipient). These scripts have been in place and scheduled for years. However, recently, about a half dozen of the scheduled tasks have plain stopped triggering and I have no idea why.

For example, I have a scheduled task that's set to run at 9PM every Monday, Tuesday, Wednesday, Thursday, Friday of every week. It's "Next Run Time" is tonight at 9PM. It's "Last Run Time" was 1/24/2020 at 9:01:20 PM. Whaa?? The "Last Run Result" was "The operation completed successfully. (0x0)".

I can right click the offending jobs and run them manually no problem. I've "poked" at the scheduled tasks, enabled/disabled them, the trigger, changed the time back/forth, even changed the "Configure for" option from the default "Windows Vista, Windows Server 2008" to "Windows Server 2012". One thing I have not tried exporting/importing the offending tasks.

I'm curious if any of y'all have experienced anything like this?

If the task doesn't complete, it will not run again. Also the task scheduling service have to be running for any tasks to be triggered.

Right. I have "Stop the task if it runs longer than" set to 3 days. So I should see them in the "Running" state if that were the case, no?

Depends on if they are running right now. Anyway, how long does the task take to complete? Do your scripts produce a log file?

They are all "Ready" as of this moment. They run nightly, so I should see them as "Running" right now.

Some of them do, some of them do not (I've been adding Start/Stop Transcript to them as I've needed to troubleshoot). For those that do log, the log's last modified date/time corresponds to the last time the Task Scheduler reports the task ran.

I just edited my original post to add that the script I talk about as an example does log, and the log has not been touched since Task Scheduler reported it ran successfully.

OK, post screenshot of each tab under that task's properties, including what's under the edit button on the "triggers" tab.

Just export the xml here, clean any sensitive info. Easier.

Here is the XML of the task I mentioned in my OP. I can do screenshots too if that's still desired by anyone.

<?xml version="1.0" encoding="UTF-16"?>
<Task version="1.4" xmlns="http://schemas.microsoft.com/windows/2004/02/mit/task">
  <RegistrationInfo>
    <Date>2015-12-23T13:40:10.4492565</Date>
    <Author>[REDACTED]</Author>
  </RegistrationInfo>
  <Triggers>
    <CalendarTrigger>
      <StartBoundary>2015-12-23T21:00:00</StartBoundary>
      <Enabled>true</Enabled>
      <ScheduleByWeek>
        <DaysOfWeek>
          <Monday />
          <Tuesday />
          <Wednesday />
          <Thursday />
          <Friday />
        </DaysOfWeek>
        <WeeksInterval>1</WeeksInterval>
      </ScheduleByWeek>
    </CalendarTrigger>
  </Triggers>
  <Principals>
    <Principal id="Author">
      <UserId>S-1-5-18</UserId>
      <RunLevel>LeastPrivilege</RunLevel>
    </Principal>
  </Principals>
  <Settings>
    <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>
    <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>
    <StopIfGoingOnBatteries>true</StopIfGoingOnBatteries>
    <AllowHardTerminate>true</AllowHardTerminate>
    <StartWhenAvailable>false</StartWhenAvailable>
    <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>
    <IdleSettings>
      <StopOnIdleEnd>true</StopOnIdleEnd>
      <RestartOnIdle>false</RestartOnIdle>
    </IdleSettings>
    <AllowStartOnDemand>true</AllowStartOnDemand>
    <Enabled>true</Enabled>
    <Hidden>false</Hidden>
    <RunOnlyIfIdle>false</RunOnlyIfIdle>
    <DisallowStartOnRemoteAppSession>false</DisallowStartOnRemoteAppSession>
    <UseUnifiedSchedulingEngine>false</UseUnifiedSchedulingEngine>
    <WakeToRun>false</WakeToRun>
    <ExecutionTimeLimit>P3D</ExecutionTimeLimit>
    <Priority>7</Priority>
  </Settings>
  <Actions Context="Author">
    <Exec>
      <Command>powershell</Command>
      <Arguments>-ExecutionPolicy Bypass -File "C:\scripts[REDACTED].ps1"</Arguments>
    </Exec>
  </Actions>
</Task>