We have a system that uses Apache Jackrabbit as an image (document) storage repository. We would really like to be able to pull documents for use with applications outside said system. The vendor of the system is, of course, not willing to volunteer how we can do this. So, I've been asked to reverse engineer it. I've looked at the database (MS-SQL) that's being used as storage and, yeah, I need to get into it from the Jackrabbit side...

Anyone have any pointers on resources to help me with this? At least a pointer on where to start?

It goes without saying, I have no clue what I'm doing.

Hey all - So sorry for going dead! We implemented a new system that basically "runs the organization" and it's been a mad house. We consolidated 6 separate databases into one system, and even after a year of doing data migration tests, UAT, etc., it's still a bit of a mess (which we anticipated, fortunately). In any rate, I am here to update this thread on the issue...

It seems that the issue was caused by my backup software. Because the backup jobs were taking so long to process, for one magical moment, the software would have snapshots created for every single VM. This was the cause of the issue. After beefing up our backup server and adding SSDs for the metadata database (deduplication), backup windows are MUCH better, jobs are not overlapping as much as before, and this issue has not happened since.

Thanks everyone for the help! I'm hoping to be able to participate a bit more moving forward. We'll see how that goes...

I heard through @scottalanmiller (thanks again, BTW!) that there are some pretty active XenServer users here, so hopefully some light can be shed on this weird issue that hast just started happening for me.

I received the following alert multiple times from my XenServer pool today:

Field Value

Name: No space left on device
Priority: 3
Class: SR
Object UUID: 545839f5-e2fc-e972-9391-d5641a60a567
Timestamp: 20151017T15:06:49Z
Message UUID: 4cdd6b52-3343-fd64-bd37-e37a59b1a793
Pool name: MyPool
Body: Run out of space while coalescing.

I've checked the SR and according to XenCenter it is approximately 50% consumed:

I checked the SAN itself and it shows even less disk space consumed. In any rate, all signs show I have plenty of disk space.

Is there any way to get more detail behind what was going on when this happened? I looked at /var/SMlog and didn't see anything out of the ordinary, but maybe I need to look again.

Any ideas? Thanks!

Well, crap. I think I answered my own question...

[zimbra@mail ~]$ zmprov gadl

Does the trick

I hacked together a PowerShell script that backs up a very large MS-SQL database, copies it to a remote location, verifies the copy is complete, then emails me a report to let me know whether or not the run was successful.

Script: https://pastebin.com/vWuCzEP3

The SQL backup is ~600 GB (it's a database of document images). The script gets to the Copy-Item step and then seems to just quit running. The copy does complete, but the remaining steps in the script are never executed. The source/destination backup files are never compared post copy, the source file is not deleted (which means I have no disk space to run the subsequent full backup :-D), I don't get the success/failure email, and it doesn't "clean up" after itself (delete the log files). All of the logging I do also seems to indicate that the script up and "stops" at the Copy-Item stage (no log entries after this step is executed).

The SQL backup takes ~1.5 hours to complete and the copy takes ~1.5 hours to complete. Am I hitting a PowerShell session timeout of some sort? If so, how can I extend the timeout?

QUICK ADD: I forgot to add that this script was working in the past. I don't recall when this started happening (maybe a month ago), but I suspect it has something to do with some threshold I've hit in terms of how long the script is running (the database backup was ~400 GB when I hacked this script together some 2 years ago).

QUICK ADD 2: The script actually runs longer as it also verifies that the backup is good with SQL before initiating the copy, and it looks like that takes an hour. So 1.5 hours to run the backup, another hour for SQL to verify the backup, then another 1.5 hours for the backup to be copied to a remote location. So we're looking at ~4 hours.

I've realized that there is other Zimbra maintenance that I need to schedule (most importantly upgrading from 8.6.0 to current). I'm going to do the shut down, rescan SR, and hope it coalesces when I do this work. I seem to be in OK shape for the moment. Alike is able to back it up and backups are good (I did a test restore successfully).

Fixed! Now using lastLogonTimestamp.

param (
    [string]$group,
    [string]$days = 30,
    [string]$test = "y"
)

if ( -not ($group)) {
    $scriptName = $MyInvocation.MyCommand.Name
    echo "Group parameter missing."
    echo "Script usage: $scriptName -group `'AD Group`' -days 30 -test NO"
    echo "If `"-days`" isn't specified the default is 30."
    echo "If `"-test NO`" isn't specified, no changes will be made."
    exit
}

echo "Disabling accounts in group $group that have no logged in for more than $days day(s)."
if ( $test -ne "NO") { echo "Running in **TEST** mode.  No changes will be made!" }

import-module activedirectory
$disableList = @(get-adgroupmember $group | select -expandproperty SamAccountName)

$expiration = (get-date).adddays(-$days)

foreach ($acct in $disableList) {
    $lastLogonTS = get-aduser $acct -properties lastlogontimestamp | select -expandproperty 
lastlogontimestamp
    $lastLogon = [datetime]::FromFileTime($lastLogonTS)
    if ($lastLogon -lt $expiration) {
        echo "$acct's last logon was on $lastLogon which was more than $days days ago and is eligible 
for deactivation."
        if ($test -eq "NO") {
            disable-adaccount -identity $acct
            echo "$acct disabled"
        }
    }
}

Thanks @scottalanmiller and @mlnews !

I'm the sys/network admin for a U.S. trial court on the west coast. We have 12 XenServer hosts (8 in a pool) and growing, and a smidge over 100 VMs in our environment. Many of our services are Linux based, which I'm fairly proud of. Virtualization, networking, storage administration, AD, email, MySQL/MSSQL, you name it I am in charge of it. Fortunately we have a great group of guys who manage our workstations and do end-user support, so I can focus on back end infrastructure stuffs.

I'm a geek at heart, but my 2 year old b/g twins limit what I can tinker with at home (in a good way).

I love friendly and healthy discussion about a certain topic. If I'm not posting for help (as in, CRAP this is broken how do I fix it?!), I'm posting because I want the opinions and experience of the members of the board.

What drives me nuts is when people post statements similarly to "well if you're not doing X then you're an idiot." That is NOT HELPFUL.

Also what drives me nuts is when people don't take the time to read the entire OP before responding. On SW more often than not I've had a lot of quick spastic replies to posts I've made and it's obvious they either didn't read the OP completely or did not understand what I was asking. They often feel like "points" posts. I then have to regurgitate my OP multiple times and it's frustrating.

I participate in communities like SW and ML because I am the first to admit that I do NOT know nor have experienced everything from every angle. Having input from the diversity of these communities helps me grow as an IT professional. And, who knows, maybe my post will help someone out in the future.

I am an Alike user and, although they have their own faults, the software has been doing a decent job backing up my environment. I believe we are pushing their software pretty hard. I have ~100 VMs across 19 hosts. Once we get over the initial deployment and first backup, things are relatively smooth after that. They've made some good progress with their software over the years. There were some rocky times in the Alike 3.x days, and we eventually found a sweet spot with a "special" build of Alike 3.5 which I was running for a good 6 months plus without any fuss (was really nice). When I originally trialed the software back in 2014 they were really good about extending it without question. I believe I trialed the software for a good 3-4 months, and during my trial their support team treated me like I was a paid customer. My only wish is that their support was more reachable via phone (they like to communicate via email/tickets, which I understand)...but that'd drive up the cost of the software. When I've had major issues they've jumped on a conference call with me and done remote sessions. I believe their software does everything that PHDVirtual/Unitrends does, so you shouldn't lose any features.

I'm currently in the process of transitioning from Alike 3.5 to 4.2, which basically requires you to scrap and re-do your backups. Fortunately for me, I have the hardware to run both in parallel. I'm still working through the initial backups with 4.2, which are running into some snags (not unexpected), but their support has been responsive and helpful. I am hopeful things will iron out in sometime this week.

@pete-s said in MariaDB/MySQL Table Repair Eating TONS of disk space:

@anthonyh said in MariaDB/MySQL Table Repair Eating TONS of disk space:

I have a MariaDB database that is of moderate size (~48 GB) and two of the tables within this database have corruption. One of the offending tables is reportedly ~25 GB in size. When running REPAIR TABLE, the process eats up all the remaining disk space (which admittedly wasn't a lot, 14 GB).

I started growing the disk and gave it an additional 50 GB. The repair process gobbled it up. Gave it 50 GB more, repair process gobbled it up. Gave it 100 GB more (for a total of 200 GB additional disk), and the repair process still gobbled it all up. I'm a little baffled as to why and am wondering if anyone here can shed some light as to why a 25GB table would need over 200GB of working space for a repair?

The database is using MyISAM storage (I know, I'm working on it) and the table in question stores data (document images) in longblob format. From what I understand, MyISAM table repairs basically involve reconstructing the table, so I wouldn't be surprised at needing at least an equal amount of free space to the size of the table. This behavior has me a little confused.

I assume there are no database clients trying to access the database during your repair.

I don't know the answer to your question but have you had a look at the different settings for repair table?
https://dev.mysql.com/doc/refman/5.7/en/repair-table-optimization.html

Check myisam_max_sort_file_size for instance. If it's larger than your disk then it could outgrow the available space.

You are correct. No clients are accessing the database. I'm doing the repair on a clone of the actual host.

I will investigate myisam_max_sort_file_size and report back.

I have a MariaDB database that is of moderate size (~48 GB) and two of the tables within this database have corruption. One of the offending tables is reportedly ~25 GB in size. When running REPAIR TABLE, the process eats up all the remaining disk space (which admittedly wasn't a lot, 14 GB).

I started growing the disk and gave it an additional 50 GB. The repair process gobbled it up. Gave it 50 GB more, repair process gobbled it up. Gave it 100 GB more (for a total of 200 GB additional disk), and the repair process still gobbled it all up. I'm a little baffled as to why and am wondering if anyone here can shed some light as to why a 25GB table would need over 200GB of working space for a repair?

The database is using MyISAM storage (I know, I'm working on it) and the table in question stores data (document images) in longblob format. From what I understand, MyISAM table repairs basically involve reconstructing the table, so I wouldn't be surprised at needing at least an equal amount of free space to the size of the table. This behavior has me a little confused.

The good news is that this does not seem to be happening on my Server 2016 DCs, and the Time Service event log entries report it's syncing with HE's stratum 1 servers regularly and without issue.

I do see corrections every now and then that may be coming from the XenServer guest agent, but it's only a couple of times every few days or so (sometimes longer), and they're at the millisecond level. Would prefer it to not do this at all (just let me manage time via NTP!), but I don't believe this would cause anything detrimental.

Minor adjustments like this:

The system time has changed to ‎2020‎-‎02‎-‎04T18:13:51.561000000Z from ‎2020‎-‎02‎-‎04T18:13:51.565339000Z.

and

The system time has changed to ‎2020‎-‎01‎-‎24T20:05:36.893000000Z from ‎2020‎-‎01‎-‎24T20:05:36.892351300Z.

@JaredBusch said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh I absolutely hate VM integration based time sync with the host across all platforms.

It is an absolute horrible idea that should have never been implemented.

Agreed 100%!

Upon further scrutiny of the Event Logs, I think I might be on to what is happening. I think it may be a clock issue.

We are an XCP-ng shop, and it seems that the XenServer guest agent is attempting to sync the clock every 30 minutes. It issues a sync which causes the time to shift anywhere from a few milliseconds to a few MINUTES.

https://discussions.citrix.com/topic/404819-xenapppvs-weired-time-sync-problems-after-ctx235405-xeniface-windows-io-driver-821123/

Seems this is an issue other folks have complained about (though others seem to experience much wider variations in time).

The two jobs that were "skipped" on the 01/31 are scheduled to run at 7:00 PM. It looks like the system time was tweaked at 6:39 PM and 7:09 PM. So, in theory, the jobs should've fired, but this is definitely not helping things. These are events with the ID of 1, the "source" is Kernel-General, and the user is SYSTEM.

01/31 @ 6:39:18 PM: The system time has changed to ‎2020‎-‎02‎-‎01T02:39:18.826000000Z from ‎2020‎-‎02‎-‎01T02:40:37.936724600Z.

01/31 @ 7:09:17 PM: The system time has changed to ‎2020‎-‎02‎-‎01T03:09:17.907000000Z from ‎2020‎-‎02‎-‎01T03:10:33.882678100Z.

This eventually causes the Time Service to complain since it causes the time to be off by more than 5 seconds for a period of time.

Every time it seems to roll the clock back. I followed the guidance via the Citrix forum post linked above in modifying registry entries to stop it from doing this but it didn't seem to help. Supposedly modifying the following registry entries to be either "blank" or set to "Local" is supposed to stop the Xen agent time sync.

HKLM\Software\Citrix\XenTools\HostTime
HKLM\Software\Wow6432Node\Citrix\XenTools\HostTime

However, they were already blank in the registry. So I set them to "Local" and restarted all the services on the box that have Citrix or Xen in the name since I can't reboot this box without pre-planning (yay for 24/7 availability). I'm still seeing the Kernel-General Event ID 1's in the System event log. Fun...

The time on the hosts are correct, and my NTP hierarchy is as so:

Member Servers -> DCs -> Stratum 1 NTP servers hosted by Hurricane Electric (one in San Jose, CA and the other in New York, NY).

I combed through time stuff yesterday and the time on the DCs seems stable (and w32tm reports they are syncing properly), as well as on other hosts, but I'll need to give it a second look.

I am also planning on upgrading from XCP-ng 7.6 to 8.0 this weekend...I'll have to see if this behavior continues after that.

@wrx7m said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.

Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!

I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....

Do you have backup jobs or VSS running at the same time? Could be a stun.

Not on this particular VM.

@Obsolesce said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh maybe schedule something else to log pings to that device when those tasks are supposed tobrun. Maybe it's taking a nap or something at that time. No idea, never seen that before, lots of variables.

These jobs, unfortunately, run queries against a SQL database that's local to the box.

It's definitely weird.

So, I ended up just re-creating the jobs manually. It wasn't nearly as cumbersome as I had imagined. I'd export the job and use the XML as reference.

Things seemed good, until Friday (1/31). Some of the jobs I re-created did not run that night. They have subsequently ran successfully though...just "skipped" Friday even though they're scheduled to run that day of the week (Monday - Friday)...so what the heck?!

I've been meaning to move these jobs to another box, but it's been low priority. I guess this is a sign that I should do that....

@Dashrender said in Windows Server 2012 - Task Scheduler Issue:

@anthonyh said in Windows Server 2012 - Task Scheduler Issue:

Wow, so I just went through all the scheduled tasks on this box.

There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.

Perhaps export them all - then find out how to re-create them using powershell. Then you'll have a backup for them incase you need to migrate this to another server, etc for the future.

Not a bad idea.

Wow, so I just went through all the scheduled tasks on this box.

There are 41 tasks total. Out of the 41, there are 13 that are in this weird "ignored" condition. Bleh. This is going to be fun re-creating these.