Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah
-
@scottalanmiller said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
Another company follow up while I'm thinking about them. Their social media has remained totally silent since Feb 4th. And their blog that was always a twice weekly thing has been silent since a few days before that. The news reports definitely continue to seem to have been real, contrary to what they stated.
Goes to their blog one week before the compromise.
The title of the article is correct. Just remove the dash
-
@IRJ said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@scottalanmiller said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
Another company follow up while I'm thinking about them. Their social media has remained totally silent since Feb 4th. And their blog that was always a twice weekly thing has been silent since a few days before that. The news reports definitely continue to seem to have been real, contrary to what they stated.
Goes to their blog one week before the compromise.
The title of the article is correct. Just remove the dash
Oh now that is funny crap!
-
@IRJ remove the dash and add a colon after "mistakes"...
-
@scottalanmiller said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
So we heard from customers of Protek Support in Salt Lake City that the MSP has been hit with ransomware that has gone on to hit all of their clients as well. From what we understand, they are currently on four days of customers being without their files and they aren't cleaning them up yet. We would suspect that their internal systems have been hit and they are tied up dealing with that.
Pretty good timing considering we just posted about this MSP Risk a few days ago.
How do MSPs survive this kind of level of destruction? Are clients talking to each other? Are clients going on to talk to other MSPs and look for assistance when their main support is gone?
We rarely think about how the MSP itself would be offline indefinitely and potentially unable to function in the case of a breach like this. But in this case, it looks like the MPS has been impacted to such a degree that they aren't even able to start helping customers yet. Four days with no action is a lifetime to an impacted business. Something like a hundred customers down for a whole week with no end in sight, it sounds like.
Each customer is going to need every machine - desktops, servers, storage, etc. to be totally wiped, reloaded, and restored. Imagine the manpower necessary to do that.
WiPro outsourcing giant breach: https://krebsonsecurity.com/2019/04/experts-breach-at-it-outsourcing-giant-wipro/
PCM MSP Breach: https://krebsonsecurity.com/2019/06/breach-at-cloud-solution-provider-pcm-inc/
Ongoing mess: https://www.insynq.com/support/#status
^^^ Note the word meticulous in the "we've cleaned things out" paragraph. SMHCCH Walters Kluwer: https://www.accountingtoday.com/news/the-wolters-kluwer-cch-outage-what-happened
Maersk: Saved by a physical DC that was off in Africa after a power outage.
MSPs: Vulnerabilities in RMM/PSA software allowed compromise a while back.
Bing Search: MSP Breach
Privileged Access Workstation is the only way to go today. There needs to be an air-gap between systems being used to manage clients/customers and the MSP's day to day production systems.
There is no excuse for not segmenting operations, administration, cloud services systems, backup systems, and more. None. Nada. Zippo. Zilch.
Oh, and this:
Courtesy of Malware-Traffic-Analysis. It's virtually always the human. -
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
Privileged Access Workstation is the only way to go today. There needs to be an air-gap between systems being used to manage clients/customers and the MSP's day to day production systems.
Agreed. The MSP doesn't need to be connected to their customer 24x7.
-
For the couple of sites I manage some stuff for, I have separate VMs (on my end) for each of them. Each VM has only the VPN client for the client I am connecting to.
Edit: Passwords are different for each VM, as well as each VPN connection as well.
-
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
For the couple of sites I manage some stuff for, I have separate VMs (on my end) for each of them. Each VM has only the VPN client for the client I am connecting to.
Edit: Passwords are different for each VM, as well as each VPN connection as well.
Catch is, the VMs need to be accessed from a system that has nothing to do with day-to-day operations. None.
-
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
For the couple of sites I manage some stuff for, I have separate VMs (on my end) for each of them. Each VM has only the VPN client for the client I am connecting to.
Edit: Passwords are different for each VM, as well as each VPN connection as well.
Catch is, the VMs need to be accessed from a system that has nothing to do with day-to-day operations. None.
I think I get your thinking... but for clarity's sake: Why?
-
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
For the couple of sites I manage some stuff for, I have separate VMs (on my end) for each of them. Each VM has only the VPN client for the client I am connecting to.
Edit: Passwords are different for each VM, as well as each VPN connection as well.
Catch is, the VMs need to be accessed from a system that has nothing to do with day-to-day operations. None.
I think I get your thinking... but for clarity's sake: Why?
The whole point of a Privileged Access Workstation setup is to keep things separate.
As soon as I manage client systems from this system I'm sitting at there is no guarantee of protection.
The idea is to keep an air-gap in there.
I remember a story of a small MSP that not too long ago got compromised. As it turns out, the perps that got in on their system managed to get in to the balance of the MSP's clients. Why? They had saved the passwords in the RDP files for their client's servers. Duh.
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
-
Basically you're saying that every admin at an MSP should have two machines - one for managing clients, and one for MSP related email/web surfing, etc.
-
@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
Basically you're saying that every admin at an MSP should have two machines - one for managing clients, and one for MSP related email/web surfing, etc.
https://blogs.technet.microsoft.com/datacentersecurity/2017/10/13/privileged-access-workstationpaw/
-
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
-
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.
-
@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.
It will be, because a human always has to use it at the end of the day. A used system is an at risk system. Less risk, certainly. There is good design and bad design. Good useage and bad useage. But any useful machine is at risk by nature of being useful.
-
@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.
There are several ways to implement with the simplest being the main machine having two VMs installed on it. One for day-to-day and one for client/systems management. Nothing is done on the machine itself with all designated tasks being done in their respective VM.
We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.
-
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
There are several ways to implement with the simplest being the main machine having two VMs installed on it. One for day-to-day and one for client/systems management. Nothing is done on the machine itself with all designated tasks being done in their respective VM.
We do this for certain tasks, it is air gapped in that nothing can move between the main desktop and the VM. But the human is still there, of course.
-
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.
This can be a great use case for small, limited machines like Raspberry Pi. Or super high power desktops that can run a lot of desktop VMs for you locally. And a place where, if you can, using Linux makes it work so much better. No limitations on count, don't need a GUI in all cases. Can run on ARM if you want.
-
@scottalanmiller said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.
This can be a great use case for small, limited machines like Raspberry Pi. Or super high power desktops that can run a lot of desktop VMs for you locally. And a place where, if you can, using Linux makes it work so much better. No limitations on count, don't need a GUI in all cases. Can run on ARM if you want.
We're getting there but not yet. The idea that's coming down the pipe is a boutique build we have designed that has everything mounted on small aluminium plate and is essentially wireless except power. It's a fun side project.
-
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.
There are several ways to implement with the simplest being the main machine having two VMs installed on it. One for day-to-day and one for client/systems management. Nothing is done on the machine itself with all designated tasks being done in their respective VM.
We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.
There is security leakage between VMs on a client machine for instance over clipboard.
Have a look at Qubes. https://www.qubes-os.org/
It's probably the best implementation of security separation to date.
-
@Pete-S said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@Dashrender said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@dafyre said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
@PhlipElder said in Protek Support MSP Ransomware Hits Customers in Salt Lake City, Utah:
All it takes is one absentminded click or drive-by that's completely shielded from us as we go about the day to day stuff and it's done. Game over. Say, "Bubbye".
There's always going to be that risk or one absentminded click.
Granted an Air-gapped PWA is a good way to handle it.... but so is not saving passwords in RDP files (I don't do this), and if you use an app like MobaXterm that can encrypt the files for you, use a good pass phrase.
However if your admin machine is owned, you have bigger issues to start with.
Well, the idea is that the air-gapped machine won't ever be in a situation to become compromised, is my guess. I haven't had a chance to look at the MS link Philip sent earlier.
There are several ways to implement with the simplest being the main machine having two VMs installed on it. One for day-to-day and one for client/systems management. Nothing is done on the machine itself with all designated tasks being done in their respective VM.
We have a number of laptops that came back from client refreshes. So, we're using them as our dedicated management machines. Asus makes a great external USB3 DisplayLink and DisplayPort external monitor that allows for two screens. That makes the work easier.
There is security leakage between VMs on a client machine for instance over clipboard.
Have a look at Qubes. https://www.qubes-os.org/
It's probably the best implementation of security separation to date.
not all have clipboards, that would be a big leak if they had that on.
