Best posts made by anthonyh

We have a system that uses Apache Jackrabbit as an image (document) storage repository. We would really like to be able to pull documents for use with applications outside said system. The vendor of the system is, of course, not willing to volunteer how we can do this. So, I've been asked to reverse engineer it. I've looked at the database (MS-SQL) that's being used as storage and, yeah, I need to get into it from the Jackrabbit side...

Anyone have any pointers on resources to help me with this? At least a pointer on where to start?

It goes without saying, I have no clue what I'm doing.

Hey all - So sorry for going dead! We implemented a new system that basically "runs the organization" and it's been a mad house. We consolidated 6 separate databases into one system, and even after a year of doing data migration tests, UAT, etc., it's still a bit of a mess (which we anticipated, fortunately). In any rate, I am here to update this thread on the issue...

It seems that the issue was caused by my backup software. Because the backup jobs were taking so long to process, for one magical moment, the software would have snapshots created for every single VM. This was the cause of the issue. After beefing up our backup server and adding SSDs for the metadata database (deduplication), backup windows are MUCH better, jobs are not overlapping as much as before, and this issue has not happened since.

Thanks everyone for the help! I'm hoping to be able to participate a bit more moving forward. We'll see how that goes...

I heard through @scottalanmiller (thanks again, BTW!) that there are some pretty active XenServer users here, so hopefully some light can be shed on this weird issue that hast just started happening for me.

I received the following alert multiple times from my XenServer pool today:

Field Value

Name: No space left on device
Priority: 3
Class: SR
Object UUID: 545839f5-e2fc-e972-9391-d5641a60a567
Timestamp: 20151017T15:06:49Z
Message UUID: 4cdd6b52-3343-fd64-bd37-e37a59b1a793
Pool name: MyPool
Body: Run out of space while coalescing.

I've checked the SR and according to XenCenter it is approximately 50% consumed:

I checked the SAN itself and it shows even less disk space consumed. In any rate, all signs show I have plenty of disk space.

Is there any way to get more detail behind what was going on when this happened? I looked at /var/SMlog and didn't see anything out of the ordinary, but maybe I need to look again.

Any ideas? Thanks!

Well, crap. I think I answered my own question...

[zimbra@mail ~]$ zmprov gadl

Does the trick

I hacked together a PowerShell script that backs up a very large MS-SQL database, copies it to a remote location, verifies the copy is complete, then emails me a report to let me know whether or not the run was successful.

Script: https://pastebin.com/vWuCzEP3

The SQL backup is ~600 GB (it's a database of document images). The script gets to the Copy-Item step and then seems to just quit running. The copy does complete, but the remaining steps in the script are never executed. The source/destination backup files are never compared post copy, the source file is not deleted (which means I have no disk space to run the subsequent full backup :-D), I don't get the success/failure email, and it doesn't "clean up" after itself (delete the log files). All of the logging I do also seems to indicate that the script up and "stops" at the Copy-Item stage (no log entries after this step is executed).

The SQL backup takes ~1.5 hours to complete and the copy takes ~1.5 hours to complete. Am I hitting a PowerShell session timeout of some sort? If so, how can I extend the timeout?

QUICK ADD: I forgot to add that this script was working in the past. I don't recall when this started happening (maybe a month ago), but I suspect it has something to do with some threshold I've hit in terms of how long the script is running (the database backup was ~400 GB when I hacked this script together some 2 years ago).

QUICK ADD 2: The script actually runs longer as it also verifies that the backup is good with SQL before initiating the copy, and it looks like that takes an hour. So 1.5 hours to run the backup, another hour for SQL to verify the backup, then another 1.5 hours for the backup to be copied to a remote location. So we're looking at ~4 hours.

I've realized that there is other Zimbra maintenance that I need to schedule (most importantly upgrading from 8.6.0 to current). I'm going to do the shut down, rescan SR, and hope it coalesces when I do this work. I seem to be in OK shape for the moment. Alike is able to back it up and backups are good (I did a test restore successfully).

Fixed! Now using lastLogonTimestamp.

param (
    [string]$group,
    [string]$days = 30,
    [string]$test = "y"
)

if ( -not ($group)) {
    $scriptName = $MyInvocation.MyCommand.Name
    echo "Group parameter missing."
    echo "Script usage: $scriptName -group `'AD Group`' -days 30 -test NO"
    echo "If `"-days`" isn't specified the default is 30."
    echo "If `"-test NO`" isn't specified, no changes will be made."
    exit
}

echo "Disabling accounts in group $group that have no logged in for more than $days day(s)."
if ( $test -ne "NO") { echo "Running in **TEST** mode.  No changes will be made!" }

import-module activedirectory
$disableList = @(get-adgroupmember $group | select -expandproperty SamAccountName)

$expiration = (get-date).adddays(-$days)

foreach ($acct in $disableList) {
    $lastLogonTS = get-aduser $acct -properties lastlogontimestamp | select -expandproperty 
lastlogontimestamp
    $lastLogon = [datetime]::FromFileTime($lastLogonTS)
    if ($lastLogon -lt $expiration) {
        echo "$acct's last logon was on $lastLogon which was more than $days days ago and is eligible 
for deactivation."
        if ($test -eq "NO") {
            disable-adaccount -identity $acct
            echo "$acct disabled"
        }
    }
}

Thanks @scottalanmiller and @mlnews !

I'm the sys/network admin for a U.S. trial court on the west coast. We have 12 XenServer hosts (8 in a pool) and growing, and a smidge over 100 VMs in our environment. Many of our services are Linux based, which I'm fairly proud of. Virtualization, networking, storage administration, AD, email, MySQL/MSSQL, you name it I am in charge of it. Fortunately we have a great group of guys who manage our workstations and do end-user support, so I can focus on back end infrastructure stuffs.

I'm a geek at heart, but my 2 year old b/g twins limit what I can tinker with at home (in a good way).

I love friendly and healthy discussion about a certain topic. If I'm not posting for help (as in, CRAP this is broken how do I fix it?!), I'm posting because I want the opinions and experience of the members of the board.

What drives me nuts is when people post statements similarly to "well if you're not doing X then you're an idiot." That is NOT HELPFUL.

Also what drives me nuts is when people don't take the time to read the entire OP before responding. On SW more often than not I've had a lot of quick spastic replies to posts I've made and it's obvious they either didn't read the OP completely or did not understand what I was asking. They often feel like "points" posts. I then have to regurgitate my OP multiple times and it's frustrating.

I participate in communities like SW and ML because I am the first to admit that I do NOT know nor have experienced everything from every angle. Having input from the diversity of these communities helps me grow as an IT professional. And, who knows, maybe my post will help someone out in the future.

I am an Alike user and, although they have their own faults, the software has been doing a decent job backing up my environment. I believe we are pushing their software pretty hard. I have ~100 VMs across 19 hosts. Once we get over the initial deployment and first backup, things are relatively smooth after that. They've made some good progress with their software over the years. There were some rocky times in the Alike 3.x days, and we eventually found a sweet spot with a "special" build of Alike 3.5 which I was running for a good 6 months plus without any fuss (was really nice). When I originally trialed the software back in 2014 they were really good about extending it without question. I believe I trialed the software for a good 3-4 months, and during my trial their support team treated me like I was a paid customer. My only wish is that their support was more reachable via phone (they like to communicate via email/tickets, which I understand)...but that'd drive up the cost of the software. When I've had major issues they've jumped on a conference call with me and done remote sessions. I believe their software does everything that PHDVirtual/Unitrends does, so you shouldn't lose any features.

I'm currently in the process of transitioning from Alike 3.5 to 4.2, which basically requires you to scrap and re-do your backups. Fortunately for me, I have the hardware to run both in parallel. I'm still working through the initial backups with 4.2, which are running into some snags (not unexpected), but their support has been responsive and helpful. I am hopeful things will iron out in sometime this week.

I run Fedora on my primary desktop at home and just updated to kernel 4.10. NVIDIA's later driver has a bug that causes it to fail to compile with this kernel. Posting the workaround in case anyone else runs into this issue.

Source: https://devtalk.nvidia.com/default/topic/995429/failed-installed-nvidia-with-kernel-4-10/

Extract the installer:

sudo chmod +x ./NVIDIA-Linux-x86_64-375.39.run
./NVIDIA-Linux-x86_64-375.39.run -x

CD into driver directory and download patch:

 cd ./NVIDIA-Linux-x86_64-375.39
 wget https://pkgs.rpmfusion.org/cgit/nonfree/nvidia-kmod.git/plain/kernel_4.10.patch

Apply patch:

 patch -p1 < kernel_4.10.patch

Run the driver installer:

sudo ./nvidia-installer

Driver should install successfully!

Holy crap! Am I an idiot or what?! The issue is my score line. It's wrong. It should be:

score  AH_KNOWBE4  -10.0

Not sure why the hell it took me so long to catch my mistake!

We run MSSQL on VMs under XenServer. We have two pretty hefty MSSQL VMs, and I have them configured so that TempDB, User DBs, Logs, and backups are all on their own virtual disks. These virtual disks all reside within the same SR, which is a virtual volume on our 3PAR SAN. Each MSSQL server is given it's own virtual volume.

I grow the respective virtual disks as needed, as well as the virtual volume. It has worked out well in our instance. If we didn't have the flexibility the 3PAR "virtual volume" model, I would probably be approaching this differently.

As for the "number of TempDB files per logical processor", I have this implemented on our two large production MSSQL servers. Whether or not it improved performance I cannot say for fact. I can tell you that it did not cause any harm for us. I have 8 TempDB files that are of equal size (since the rule of thumb is to not exceed 8 if you have more than 8 logical processors), and I grow them as needed. I believe there is a trace flag that you can set (in 2012 anyway) that will make auto-growth grow all files evenly, but I'm paranoid and handle it manually.

https://support.microsoft.com/en-us/help/2154845/recommendations-to-reduce-allocation-contention-in-sql-server-tempdb-database

Of course, every MSSQL environment is different. So your resulst may vary.

@scottalanmiller said in MSSQL tempdb - your location:

@anthonyh said in MSSQL tempdb - your location:

From what I've been reading regarding TempDB in a RAM disk is that it's not recommended these days. The way the MSSQL engine works (if configured properly) is it uses all the RAM of the server (gets complicated after 64 GB RAM if you're using the Standard edition, but even then it can use more). So, in theory, TempDB should be in RAM as much as the server allows. It will only "spill" to disk if there is not enough RAM to complete whatever TempDB operation is happening at the time.

From what I'm reading, the recommendation these days is to put TempDB on a local SSD and/or beef up the amount of RAM the MSSQL server has.

https://www.brentozar.com/archive/2014/12/sql-server-2012-standard-edition-max-server-memory-mb/

Good to know. Basically.... RAM disk but managed by SQL Server, not by you.

That's how I understand it. The key is configuring the MSSQL service to consume the appropriate amount of RAM. I'm trying to dig up the "rule of thumb" I used when configuring our production servers. I believe it was Total RAM - 10 GB. However this was based on our prod VMs having 96 GB RAM assigned to them.

I also enabled "lock pages in memory" which helped performance wise as well:

An official MS article: https://support.microsoft.com/en-us/help/2659143/how-to-enable-the-locked-pages-feature-in-sql-server-2012

A better article: https://sqlserverperformance.wordpress.com/2011/02/14/sql-server-and-the-lock-pages-in-memory-right-in-windows-server/

This should prevent the OS from paging out SQL data, so less swapping will occur. You don't want to do this if you have not configured the memory limit of the MSSQL service as by default it's set to consume everything.

I should disclaim that I do not consider myself a SQL expert in any way and all my experience is with MSSQL 2012 under Server 2012 R2. Soo your experience may vary. I'd like to think I've got decent Google-Fu and an ability to eventually figure out which buttons to push in which order.

@scottalanmiller said in Firewalls & Restricting Outbound Traffic:

@anthonyh said in Firewalls & Restricting Outbound Traffic:

@scottalanmiller said in Firewalls & Restricting Outbound Traffic:

@anthonyh said in Firewalls & Restricting Outbound Traffic:

@scottalanmiller said in Firewalls & Restricting Outbound Traffic:

@anthonyh said in Firewalls & Restricting Outbound Traffic:

@JaredBusch said in Firewalls & Restricting Outbound Traffic:

@scottalanmiller said in Firewalls & Restricting Outbound Traffic:

@anthonyh said in Firewalls & Restricting Outbound Traffic:

@JaredBusch said in Firewalls & Restricting Outbound Traffic:

@anthonyh said in Firewalls & Restricting Outbound Traffic:

Ok, so perhaps the discussion should be...which ports would you blanket block?

25. That's it. And it is blocked on every network I have ever had access to the core router of.

You wouldn't want to force DNS at least, too? I'm liking the idea that DNS requests must be made by my DCs. Maybe it's not necessary.

Are there cases for that? Okay. But you already control that in other ways. So I'm unclear what benefit you think that this will provide since you are talking about malware, not your users. Does malware looking things up on your DC provide some value to you?

You cannot force client DNS 100% of the time to be what you want in a BYOD environment. If DNS control is a desire, blocking DNS at the router is the simpler method.

It is unrelated to the blanket blocking discussion as it is a decision the business needs to decide if they need to make or not.

What's the difference between blocking DNS at the router vs firewall? The idea being only permitted DNS servers would be allowed to perform DNS requests to the outside world. Everything on the inside would need to use the on premis DNS servers (aka our DCs).

RIght, if you allow unmanaged devices onto your network, then it could make sense to use LAN security to control access to DNS. But I'd ask.. why do you let uncontrolled devices onto your network?

Basically how I see this is an attempt at LAN based security, while also allowing skipping LAN management for the worst of both worlds mixed together.

Where did I say I let unmanaged devices onto my network?

That was the reason that Jared suggested for why you'd want to block DNS POrt 53. If you don't have unmanaged things on your network, what would blocking it get for you? It only makes sense if you have unmanaged things on the network.

It would force any prospective clients on the network to only be able to use my DNS servers. Sure, everything on the (non-guest) network is managed, and I do my best to keep them all clean, but things happen and I know I'm not the worlds perfect sysadmin.

I guess it's dumb after all.

It would force them not to use Google or whatever. But it would not make them point to your AD. So it would break their access. Which might be what you want, but I'd guess not.

Yes, that'd be what I want. If DNS on a given host is ill-configured, it doesn't work. Exactly the behavior I'd expect.

@krisleslie The EdgeRouter line's OS is based on Vyatta OS which you can run on x86 hardware. Check this out: https://vyos.io/

In case anyone needs it, here is the solution. Looks like the instructions I followed to change the hostname did not include the proxy service. So to cover all bases, use the following commands after following the article here: https://wiki.zimbra.com/wiki/ZmSetServerName

zmprov ms `zmhostname` zimbraReverseProxyUpstreamLoginServers new.hostname.com
zmprov ms `zmhostname` zimbraReverseProxyUpstreamEwsServers new.hostname.com
zmprov mcf zimbraReverseProxyUpstreamLoginServers new.hostname.com
zmprov mcf zimbraReverseProxyUpstreamEwsServers new.hostname.com
/opt/zimbra/libexec/zmproxyconfgen
zmproxyctl restart

Just a side note: I can confirm that @Dashrender 's suggestion of creating a CNAME works like a charm. I created a new software deployment GPO and added the source via the CNAME record and deployment was successful.

I stumbled across this by complete coincidence. I was looking at our mail server logs for an unrelated issue and noticed that messages from ML were getting rejected.

Hold the phone...I'm supposed to be receiving email alerts from ML?!

Here is the "reject" pulled from my mail server log (with slight modification):

Sep  1 08:33:24 mail postfix/smtpd[15760]: NOQUEUE: reject: RCPT from so254-54.mailgun.net[198.61.254.54]: 450 4.1.8 <[email protected]>: Sender address rejected: Domain not found; from=<[email protected]> to=<[email protected]> proto=ESMTP helo=<so254-54.mailgun.net>

Looks like it's bouncing the message(s) because "mg.mangolassi.it" does not exist. This is because we have "reject_unknown_sender_domain" set in our Postfix config.

To make sure it wasn't our mail server doing something funky, I did some DNS lookups outside of the scope our mail server sees, and sure enough, there is no A or MX record for this domain.

Not sure if this is a known issue or not, but wanted to share just in case.

Without knowing the details behind ML's infrastructure, I think simply adding an A record that points "mg.mangolassi.it" to the IP(s) that "mangolassi.it" points to will fulfill this spam check. You can then configure the web server to redirect "mg.mangolassi.it" to "mangolassi.it" if someone ever decides to type it into a web browser. Creating an A or MX record to just about anywhere would be sufficient, I believe, as that will constitute as the domain "existing" to this spam check, but I think this is the most thoughtful approach (to me, anyway).

Just FYI!

Manning the fort at home due to a sick wife and sticking in work here and there as I can. I'm thankful for having a flexible job!