Remote management of VMs hosted in colocation
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
-
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
-
Cyber security is just like physical security. Nothing is 100% secure if it's usable.
Isn't the point of security to delay the attacker long enough so he can be detected? That's why we need layers and some kind of intrusion detection.
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
-
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
-
@dashrender said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.
-
@dashrender said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
But that is not what I was arguing. Thus, as I mentioned before, strawman.
-
@pete-s said in Remote management of VMs hosted in colocation:
Cyber security is just like physical security. Nothing is 100% secure if it's usable.
Isn't the point of security to delay the attacker long enough so he can be detected? That's why we need layers and some kind of intrusion detection.
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
For the management account log in? Not yet.
-
@dafyre said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.
The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.
I'm not seeing anything that offers protection from you using a compromised machine.
You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.
sure, but if the machine is comp'ed, then there is likely a keylogger there capturing that password too.
-
@dashrender said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.
Right, SSH vs. ZT. Basically they have equal security mechanisms. ZT has slightly more risk, because it exposes slightly more. SSH is vastly simpler to implement in most cases, and is built in.
Again, I love ZT, it's just overkill for something basic. It's a common myth that VPNs are super secure and other things are not, SSH is a VPN itself, and all the username/password/key combinations are really just "longer key length" at the end of the day, they don't represent additional security.
-
@pete-s said in Remote management of VMs hosted in colocation:
Don't know anything about it but ZT sounds like one layer of security. Does it at have 2FA?
ZT is a software defined network built using a mix of hub and spoke, and mesh VPN tech. It's very advanced and very good. If you use it purely in a point to point like mode, then you have quite a bit of security, but you still need SSH or whatever at that point. The extra layer, if it is truly completely extra, provides more security by doubling up on the channel, but it also makes things more complex. If you use it for any more points than just the one to one, it becomes a point of exposure above and beyond what was there before.
-
@jaredbusch said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@jaredbusch said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.
@dashrender said in Remote management of VMs hosted in colocation:
I was wondering this exact same thing - why is ZT more trusted in this case than SSH?
@dashrender said in Remote management of VMs hosted in colocation:
but then I ask - how do you securely manage the Saltmaster, where ever it lives?
Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).
Why create all this overhead for no reason, when ZT already handles it?
Answer? Because Salt is SAM's shiny toy.
Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.
Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.
More work than your gimmicky Salt setup? As if.
I'm not talking about Salt, that was to answer a very specific question that would STILL need to be addressed with your ZT setup. But not relevant here.
I was talking about how much simpler SSH was than your gimmicky ZT setup. SSH isn't some pet technology here. And with your ZT, how are you connecting if you aren't using SSH, RDP or something afterwards anyway? It's an extra step.
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
-
@eddiejennings said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.
The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."
If you have SSH or ZT configured to "just connect" on your laptop, and the laptop is compromised, you have a problem. That's a given. Neither forces you to do that. In either case you could make that connection be on demand and make it so that the machine itself never has enough information to connect on its own. So what you mention as a risk is real, but can also be mitigated - it's not a risk implied by the use of either solution.
The risk in either case is also that without your laptop, someone might guess the keys. Not likely, making it a small risk, but both ZT and SSH are vulnerable to someone without your laptop brute forcing the key or finding a vulnerability in the code and breaking through. This is where I feel SSH wins hands down, ZT is great, but OpenSSH is famous for being one of the most solid, reliable, and audited pieces of security code ever written. It's made by the OpenBSD team and their commitment to security is legendary. It's supported by every big name in the industry, including Microsoft (and IBM, Red Hat, Canonical / Ubuntu, Suse, Oracle, Apple, and on and on) and is pretty much impossible to beat when it comes to the code being looked at.
Both approaches are super secure. The two aren't exact rip and replace, there are some assumptions we have to get around... like does ZT assume only exposure of a jump box, or all boxes? Does SSH mean direct to all, or only to a jump box? Does ZT expose a box that is limited to only SSH anyway? Does the box with the first SSH hop have access to anything other than SSH? And so forth.
SSH is simpler than ZT. No third party tools needed. ZT isn't complex, but it's not "just click a button", either. You have to configure the networking and sometimes with the IPv6 it doesn't like to communicate by default. ZT requires you to have ZT installed on any box you want to access from, which isn't extra security as it does nothing to stop bad guys attacking it, but does make it harder for you when you need to access the system and no longer have the device you intended to log in from.
SSH makes things like malware attacks against known vulnerabilities basically not exist. ZT would, if used normally, expose that heavily.
-
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
But it would work normally with people attacking using non-keys, would it not? Or am I missing something about what it would do?
-
@scottalanmiller said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
But it would work normally with people attacking using non-keys, would it not? Or am I missing something about what it would do?
Why would you not require keys? Not making them mandatory defeats the purpose of using them.
-
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
But it would work normally with people attacking using non-keys, would it not? Or am I missing something about what it would do?
Why would you not require keys? Not making them mandatory defeats the purpose of using them.
I think he means - if a hacker is trying to use a password on a system setup to only allow keys - the fail2ban will block those users, or won't it?
-
@dashrender said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
But it would work normally with people attacking using non-keys, would it not? Or am I missing something about what it would do?
Why would you not require keys? Not making them mandatory defeats the purpose of using them.
I think he means - if a hacker is trying to use a password on a system setup to only allow keys - the fail2ban will block those users, or won't it?
No. It's dropped before fail2ban even sees it.
-
@stacksofplates said in Remote management of VMs hosted in colocation:
@dashrender said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@stacksofplates said in Remote management of VMs hosted in colocation:
@scottalanmiller said in Remote management of VMs hosted in colocation:
@eddiejennings said in Remote management of VMs hosted in colocation:
Allowing an SSH connection to the managementVM from the Internet
I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.
Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.
Or add Salt and open/close based on need so it doesn't stay open.
Fail2ban doesn't work with keys.
But it would work normally with people attacking using non-keys, would it not? Or am I missing something about what it would do?
Why would you not require keys? Not making them mandatory defeats the purpose of using them.
I think he means - if a hacker is trying to use a password on a system setup to only allow keys - the fail2ban will block those users, or won't it?
No. It's dropped before fail2ban even sees it.
Oh, makes sense. There is no "attempt" like with a password, it is "already blocked."
