@scottalanmiller said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:

So we heard from customers of Protek Support in Salt Lake City that the MSP has been hit with ransomware that has gone on to hit all of their clients as well. From what we understand, they are currently on four days of customers being without their files and they aren't cleaning them up yet. We would suspect that their internal systems have been hit and they are tied up dealing with that.

Pretty good timing considering we just posted about this MSP Risk a few days ago.

How do MSPs survive this kind of level of destruction? Are clients talking to each other? Are clients going on to talk to other MSPs and look for assistance when their main support is gone?

We rarely think about how the MSP itself would be offline indefinitely and potentially unable to function in the case of a breach like this. But in this case, it looks like the MPS has been impacted to such a degree that they aren't even able to start helping customers yet. Four days with no action is a lifetime to an impacted business. Something like a hundred customers down for a whole week with no end in sight, it sounds like.

Each customer is going to need every machine - desktops, servers, storage, etc. to be totally wiped, reloaded, and restored. Imagine the manpower necessary to do that.

WiPro outsourcing giant breach: https://krebsonsecurity.com/2019/04/experts-breach-at-it-outsourcing-giant-wipro/

PCM MSP Breach: https://krebsonsecurity.com/2019/06/breach-at-cloud-solution-provider-pcm-inc/

Ongoing mess: https://www.insynq.com/support/#status

^^^ Note the word meticulous in the "we've cleaned things out" paragraph. SMH

CCH Walters Kluwer: https://www.accountingtoday.com/news/the-wolters-kluwer-cch-outage-what-happened

Maersk: Saved by a physical DC that was off in Africa after a power outage.

MSPs: Vulnerabilities in RMM/PSA software allowed compromise a while back.

Bing Search: MSP Breach

Privileged Access Workstation is the only way to go today. There needs to be an air-gap between systems being used to manage clients/customers and the MSP's day to day production systems.

There is no excuse for not segmenting operations, administration, cloud services systems, backup systems, and more. None. Nada. Zippo. Zilch.

Oh, and this:

Courtesy of Malware-Traffic-Analysis. It's virtually always the human.

@nadnerB said in Random Thread - Anything Goes:

Oh by, the person behind that CAT better be careful! :0D

@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:

@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:

@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:

All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".

There's always going to be that risk or one absentminded click.

Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.

However if your admin machine is owned, you have bigger issues to start with.

Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.

There are several ways to implement with the simplest being the main machine having two VMs installed on it. One for day-to-day and one for client/systems management. Nothing is done on the machine itself with all designated tasks being done in their respective VM.

We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.

@nadnerB said in Random Thread - Anything Goes:

I'm going to go with this being factual. It's not like anything on the Internet is made up or anything

Had a FL Warden grab me and pull me back from what I was doing. I'd crossed over the little 1.5" or 2" bar that was about 3" off the ground so looked more like a suggestion than an actual DO NOT GO HERE type of thing.

There was a good 6' to 7' or more 'gator lying there that I wanted to get some pics of.

I'm sure if he could he would have called me a fucking ID10T right there but he was pretty polite about it. My buddy just stood there laughing. He was probably waiting for the bite. Bugger ...

@kamidon said in Spec'ing a new computer from Dell or?:

@Dashrender said in Spec'ing a new computer from Dell or?:

@kamidon said in Spec'ing a new computer from Dell or?:

@scottalanmiller said in Spec'ing a new computer from Dell or?:

@gjacobse said in Spec'ing a new computer from Dell or?:

Looking at an i5 Desktop with 16GB of ram pushes the envelop at $800

Why look at Intel when genuine AMD is crushing them? With better price/performance and shipping nearly 80% of CPUs today, Intel seems like an odd choice.

DAMN...80%?
In what market though? Surely not the consumer market. And most servers I've at least seen are always Intel. Perhaps the east coast is different?
I seldom come across an amd processor in the consumer market, it's extremely rare.

Edit: And in terms of speed, I mean AMD just doesn't seem to cut it. Maybe as a server because of the muilti core speeds. https://cpu.userbenchmark.com/Compare/Intel-Core-i9-9900K-vs-AMD-Ryzen-9-3900X/4028vs4044

How many people are running i9's though? For consumers wouldn't it be better to look at i5 vs Ryzen, or i7 vs Ryzen?

No...because AMD's software is garbage, their performance is spotty despite benchmarks that are decent.
But this is anecdotal and obviously my opinion.

Now it has been a while, I wouldn't mind trying out a system with another AMD proc, but at least at this time, Intel is cheaper and faster...still...so from a consumer perspective, it doesn't seem viable to go with AMD.

Ah, tweaked a memory ... indeed, the drivers were an issue for the AMD chipsets especially the chipset software based RAID setups. Lots of broken arrays.

The problems we have had on the Intel side have been with in-box drivers in the Windows OS, though the last time that happened was Windows 7 on the Intel NIC driver ($15K dual Intel Xeon trading box with pausing mouse cursor was a heart-stopping moment) on the user side.

@Dashrender said in Random Thread - Anything Goes:

@PhlipElder said in Random Thread - Anything Goes:

@RojoLoco said in Random Thread - Anything Goes:
EDIT: BTW, talking to Floridians at the event they thought I was nuts that I thought it was hilarious but they told me it was fairly common and that gators were very resilient. They also told me to never get out of the vehicle again as the gator could have been right p*ssed waiting to swallow me. :0)

Sounds about like hitting a deer here, expect the whole swallowing thing.

We were heading in to town early one morning. We'd just turned on to the two lane highway after a black RAM 1500 went by.

We were about 10-15 car lengths behind him coming up to speed.

We have a massive Canadian Bull Moose and his haram living around us.

The bugger decided to cross the highway in front of the RAM in front of us. All I could see was the truck's brake lights. That's it. The truck was engulfed by the moose.

The driver was fortunate to have missed the moose just barely.

When we caught up to him in town I asked him if he needed to change his pants after clarifying that we were behind him. Yeah. He was still shaking a bit too.

Deer weren't too bad this fall. It was weird. Moose not too bad either. It was a relatively quiet fall for animal hits around us. We're 15 minutes from the nearest town to the south and about 30 minutes from the city to the east of us.

@FATeknollogee said in Spec'ing a new computer from Dell or?:

@PhlipElder said in Spec'ing a new computer from Dell or?:

The goal is to run the system with two virtual machines:

• VM0: Domain joined and work apps
• VM1: Not domain joined with all personal apps

This user is a power user with the need to keep things separate. The OS will be Windows 10 Enterprise 64-bit.

For a personal rig, we'd flip the workstation board and Xeon to a desktop mATX board and a Core i5 and the NVMe drives to 760p or 600p series. Cost would be substantially reduced.

Just curious...why can't the o/s be F30 with both VM's running in Virt-Manager?

F30 being Fedora?

All of our clients are Windows houses and are licensed as such with Windows 10 E3 and soon E5 to gain access to Advanced Threat Protection.

Everything is tied in to Active Directory and Group Policy for management and security.

We've built GRID on FreeBSD and security soft-appliances on OpenBSD but that was quite a while ago.

There's no point to *NIX on the user side for us and our clients. It would be way too painful. Dealing with Macs can be a challenge enough especially when Apple is not so kind to their integration components for Microsoft networks.

@nadnerB said in Random Thread - Anything Goes:

The three rules of boost:
1: Something will break
2: See #1
3: See #1

@kamidon said in Spec'ing a new computer from Dell or?:

@PhlipElder Where's the slot Proc?! Lol. Now those are biiiiiig.
Think they sat in ISA slots right? Or no maybe it was just a special slot for the processor...can't remember

It was a custom slot that allowed for four or eight CPU configurations without needing too much real estate to make it happen. I was working for a former employer in those days, so unfortunately don't have any samples.

With AMD's EPYC Rome single socket CPU setups absolutely crushing Intel's dual and even quad :astonished_face: processor rigs real estate won't be too much of an issue for the next few years.

The trend, in my mind, was back towards daughter cards for CPUs and memory slots given the density direction Intel was heading in.

AMD packing such a huge punch in one socket has put Intel on notice and Ice Lake is probably not going to be the answer they need especially if it does not have PCIe Gen 4.

@scottalanmiller said in Random Thread - Anything Goes:

Anyone been to Rackspace's website recently? I forgot that they even existed. I went to their website and realized why...

https://www.rackspace.com

This looks like an AI generated page of gibberish if you said "make a generic MSP page that says nothing."

I'm seeing rumblings that business continuity for anyone on their Hosted Exchange is a total loss. So, if they are not Outlook enabled and thus able to export their mailbox to a .PST file, or other such client that can do so, they are totally and completely hooped.

Sacrosanct Edict #1: One shall never lose the data. One shall save the data. One shall always back up the data. One shall always test restore the data. One shall lose job if lose the data.
^^^
In this case, one has to wonder how long RS is going to be around as the rest of their businesses get hit with the lack of trust.

Now, the G00g lost a freaking huge amount of mailboxes something like a decade ago. We had a customer impacted by that. No redress. None. Back then there wasn't a disaster recovery structure in place where they could back up their data short of making regular .PST files. So, gone. It was all in the G.

@kamidon said in Spec'ing a new computer from Dell or?:

@PhlipElder Finally, we're going to see an epic rise in AMD's market share (more so than the already growing trend).
AMD will be ryzing up epically!

Microsoft's ongoing AMD EPYC Rome Azure services announcements are not helping Intel at all. NVIDIA may take a bit of a hit here too as it looks like they will be utilizing AMD/ATI for graphics subsystems for their VM offerings too.

STH (Serve The Home) is a pretty good balanced resource. They've been gaga over AMD EPYC Rome.

@scottalanmiller said in Random Thread - Anything Goes:

@Dashrender said in Random Thread - Anything Goes:

@dafyre said in Random Thread - Anything Goes:

@PhlipElder said in Random Thread - Anything Goes:

@scottalanmiller said in Random Thread - Anything Goes:

Anyone been to Rackspace's website recently? I forgot that they even existed. I went to their website and realized why...

https://www.rackspace.com

This looks like an AI generated page of gibberish if you said "make a generic MSP page that says nothing."

I'm seeing rumblings that business continuity for anyone on their Hosted Exchange is a total loss. So, if they are not Outlook enabled and thus able to export their mailbox to a .PST file, or other such client that can do so, they are totally and completely hooped.

Sacrosanct Edict #1: One shall never lose the data. One shall save the data. One shall always back up the data. One shall always test restore the data. One shall lose job if lose the data.
^^^
In this case, one has to wonder how long RS is going to be around as the rest of their businesses get hit with the lack of trust.

Now, the G00g lost a freaking huge amount of mailboxes something like a decade ago. We had a customer impacted by that. No redress. None. Back then there wasn't a disaster recovery structure in place where they could back up their data short of making regular .PST files. So, gone. It was all in the G.

For personal backups, I'm using MailStore Home (https://www.mailstore.com/en/products/mailstore-home/). They have some business offerings that might be worth looking at regardless of your provider.

Thanks for this.

I have a client that doesn’t want to do M365 backups because they think a cloud provider won’t lose data.

Remind them that Microsoft lost MY data!!

Backstory?

@kamidon said in Spec'ing a new computer from Dell or?:

@PhlipElder Finally, we're going to see an epic rise in AMD's market share (more so than the already growing trend).
AMD will be ryzing up epically!

I'm just waiting for an integrated AMD EPYC Rome single socket barebones that utilizes ESDFF .L with 32 bays to allow for 1PB in 1U or 2PB in 2U. We're not a SuperMicro fan here, so we shall see which vendor drops in first.

@nadnerB said in Random Thread - Anything Goes:

WD = Water Displacement
It was designed to displace water on the inside of distributors as the points and rotor would short out causing no starts or rough idle and/or running.

It apparently does work to remove the clouding on headlight lenses that have lost their UV protective coating. We've not tried it though.

@Dashrender said in Spec'ing a new computer from Dell or?:

@PhlipElder said in Spec'ing a new computer from Dell or?:

Tecra

Has Toshiba turned a corner? Granted I haven't used them since the early 2000's, but I hated them back then..

They'z in big troubles. No new machines for a long time now. They were in the news on and off because of the mess as I recall.

The Z50-C had a plastic bottom to cut costs though still the magnesium top. It was still one CPU generation behind the Dell units when it was released.

We had a lot of Z50-C CPU fans go wonky with buzzing noises too. We've replaced a lot of them. Fortunately, we can call Toshiba, we're a long standing partner, and get them as field replaceable units. Otherwise it's take it in. That's a PITA.

@nadnerB said in Random Thread - Anything Goes:

@PhlipElder not sure it’s from Aus. That one is left hand drive, and I don’t recognise the license plate format.

Yikes. I reversed that. I stand corrected on the location of the truck, but the motto still applies as far as the investment where it counts.

Bling is ... bling. ;0)

https://www.zdnet.com/article/ransomware-hits-hundreds-of-dentist-offices-in-the-us/
https://krebsonsecurity.com/2019/08/ransomware-bites-dental-data-backup-firm/

@nadnerB said in Random Thread - Anything Goes:

Robertson = Square
"Invented" in Canada.
It's the best all around head design because it allows for the torque to remain in the screw when the driver is in some way off-tilted from perpendicular.

A drywall canon and deck canon won't allow for anything less than perpendicular so Phillips wins the day there.

Qualifier: I've done lots of subcontractor work.

@Pete-S said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

We've worked with a variety of hosting solution providers. Most start with a base of one backup done per 24 hours with a fee to restore if required.

Some have a built-in backup feature that we can then set up for the VMs we have our cloud desktop clients running in. It can be set up to run relatively often. They charge a fee for that one.

Start with once per day.

As far as the "how" what is the underlying virtualization platform?

Our hosting solutions are set up to use Veeam at the host level.

StarWind's Virtual Tape Library (VTL) can be used to augment the backup in another DC with Veeam's Cloud Connect being another option to tie in to get the backup data out of the production DC.

As far as expectations go, we're in the process of setting up a BaaS and DRaaS service based on Veeam. Backups and DR will be multi-site with one goal to be a two to four week no-delete option available.

In our investigations of BaaS/DRaaS providers none were able, or wanted, to answer the, "How do you back up our backup data to protect against failures in your system?" question.

As we are are getting into SaaS and not infrastructure, I think our primary concern are being able to restore the customers data in case something bad happens that's our fault or responsibility - for instance software bugs, hackers, ransomware, multiple hardware failures etc.

We are not as concerned with being able to restore the customers data in case they screw up, as we are if we screw up. That said, if we can without to much investment, we might be able to add something here. Have to think about that one. In either case we will provide some way for the customer to export and backup their data.

For now we run on xen (xcp-ng). The goal is to be able to restore the infrastructure with automation, so I don't expect us to really need a lot of host based backups. We have a lot more testing to do on this.

From what I can gather right now, I think we will backup to disk storage on-prem. Then from there we will go to tape. Tape will be moved off site once a week. We will do incremental backups to the cloud or another site so we can restore completely using off-site tape and the incremental backups.

This will allow us to restore from on-prem disk in most cases. If we are hacked or infected we can restore from on-site tape. In case of a fire or something we can restore from off site tape and incremental backups.

There are some keys to providing a customer facing solution:

• Customer facing network(s) are not in any way connected to the hosting company's day to day network (DtDN)
• Privileged Access Workstation structures are in place to keep things separate
• Backups are air-gapped in some way to protect against catastrophic failure or encryption event
• Customer resources are on separate equipment from DtDN

Ultimately, the entire solution set for DtDN, Support, and Customer Facing networks should be segmented completely from each other with significant protections in place to keep them that way.

• iNSYNQ
• 
• 
• Wolters Kluwer/CCH
• Maersk
• PCM
• WiPro
• Hosting company (UK 123 something?) lost everything due to backups being wiped
• Secure mail hosting company lost everything when perp took everything out right through the backups
• ETC

@Pete-S said in Backup strategy for customer data?:

@PhlipElder said in Backup strategy for customer data?:

hosting company's day to day network

With day to day network, do you mean the hosting company's own internal IT, for managing their own company?
Or do you mean the hosting company's management network for managing the hosting infrastructure?

DtDN = Sales, HR, Financing, ETC where folks blindly click on things and get hit by drive-by web sites.

Management would be with PAW (Privileged Access Workstation) and segmented away from the DtDN with absolutely no crossover between them.