"Did you know that your website is down?"
-
Yesterday, at about 4pm EST, one of our engineers at NTG was notified by a client that our website was inaccessible. After some tests from several locations, we found he was right- all that came up was a blank page. So three of us dive into VSphere and jump into a Lync conference call with a screen share to determine what it happening. We jump into console access to the server and start digging into a the files that are our website. The first line bunch of lines is just one huge block of garbled text. After scrolling a little lower, we run into a line that states '//Silence is Golden.' Yup, we got hacked.
We pull out this text and keep going to see the extent of the damage. 'Sheesh, the site is running slow. What is going on in here?' After doing some looking, we realize that there are over 36 thousand emails queued to be sent out just sending back rejection errors due to being flagged spam. Okay, time to do some rollbacks. What all is on here?
A database for the hosting. apparently. Since databases don't really play well with external backups, we do some file level restores on our sites, disable postfix, and write up the problems to be looked at when we have the authority to do a full scale baremetal restore after backing up that database. How far back can we go?
Unitrends has been passed through several hands and setups over time by the time we got here. We can't find a solid retention policy and have difficulty finding a decent backup. We can go as far back as a month, so that is what we do. We look back into some of the files for our website and there are traces of the infection being already in there. Luckily, we do not store sensitive data there,nor do we ask for sensitive data there, so nothing was ever compromised on that end.
In conclusion, we had a website that had been compromised for a little while, a server acting as a spambot, questionable backups, and a large headache. Let this serve as a warning to us all on what happens when you think you can just set it and forget it. Do not forget to give your machines and servers a once-over every once in a while
-
The NTG site was hacked or one of the sites hosted for the client was hacked?
-
The NTG site.
-
@bill-kindle Fun fun, right? That was a fun several hours...
-
Thank goodness we have an awesome response team and a wide range of skills. Dealing with databases, Linux, Wordpress, file system, unitrends, etc all at once.
-
@scottalanmiller Yea...about that backup system...@art_of_shred, we need to do some work...lol
-
@ajstringham sure, as soon as we know what the best plan is...
-
@art_of_shred Sounds good.
