Production KVM server "hardening"?
-
@Pete-S said in Pure KVM server "hardening"?:
I'm thinking about running pure KVM on debian for virtualization hosts. Not Proxmox. There will be no GUI on the servers, no web interface, only ssh for management.
Do I need to do anything special to lock down the security?
I've never used KVM in production, only on my desktop and then I've had virt-manager as well as tools like virtsh. So I don't really know what is required for a pure KVM server to be as "secure" as proxmox, xcp-ng or whatever.
I'd consider this normal rather than special, but force only key-based authentication for SSH.
-
@EddieJennings said in Pure KVM server "hardening"?:
@Pete-S said in Pure KVM server "hardening"?:
I'm thinking about running pure KVM on debian for virtualization hosts. Not Proxmox. There will be no GUI on the servers, no web interface, only ssh for management.
Do I need to do anything special to lock down the security?
I've never used KVM in production, only on my desktop and then I've had virt-manager as well as tools like virtsh. So I don't really know what is required for a pure KVM server to be as "secure" as proxmox, xcp-ng or whatever.
I'd consider this normal rather than special, but force only key-based authentication for SSH.
Good point, thanks. I try to take it a step further and use ssh certificates when I can.
Regarding access, my idea was to replicate how I've setup up virtualization hosts in the past and that is to have management access (ssh in this case) on it's own vlan (or nic). And then the guest VMs have their own vlans. And a firewall that routes and decides what is allowed.
-
I would restrict ssh to very specific hosts. If you want to be flexible on your location, you could just allow a bastion host and/or VPN. Both solutions are very low cost as bastion and VPN server uses very little resources. If you want to implement a solution that's even more proactive you could use a service like Okta that has MFA and short term token access to ssh sessions.
As far as host level, use CIS benchmarks as a good base for hardening template. Removing unnecessary packages can also help and limit potential vulnerabilities on the system. Also, the usual stuff like sending logs to SIEM.
-
@IRJ said in Production KVM server "hardening"?:
I would restrict ssh to very specific hosts. If you want to be flexible on your location, you could just allow a bastion host and/or VPN. Both solutions are very low cost as bastion and VPN server uses very little resources. If you want to implement a solution that's even more proactive you could use a service like Okta that has MFA and short term token access to ssh sessions.
As far as host level, use CIS benchmarks as a good base for hardening template. Removing unnecessary packages can also help and limit potential vulnerabilities on the system. Also, the usual stuff like sending logs to SIEM.
Thanks! My intention was to put the management network behind a VPN - with MFA for human access.
I've used CIS benchmarks as a guide before so I'll reacquaint myself with those again. I'll see if they have something specific for KVM.
-
@Pete-S said in Production KVM server "hardening"?:
Thanks! My intention was to put the management network behind a VPN - with MFA for human access.
Doesn't do anything. SSH is already in a VPN. MFA is great, but just add that to SSH. VPN won't add anything to SSH, but it risks the feeling that you can use it for other things and it tends to become the point of risk.
-
@Pete-S said in Production KVM server "hardening"?:
Thanks! My intention was to put the management network behind a VPN - with MFA for human access.
We go farther and don't enable SSH. We use reach out, rather than reach in. It's more secure, extremely hard to attack.
-
@scottalanmiller said in Production KVM server "hardening"?:
@Pete-S said in Production KVM server "hardening"?:
Thanks! My intention was to put the management network behind a VPN - with MFA for human access.
Doesn't do anything. SSH is already in a VPN. MFA is great, but just add that to SSH. VPN won't add anything to SSH, but it risks the feeling that you can use it for other things and it tends to become the point of risk.
VPN is used to get access to the private network. Servers are not accessible from internet. So there is no way to use ssh without first having access to the private network.
So double encryption was just a by-product.
-
@Pete-S said in Production KVM server "hardening"?:
@scottalanmiller said in Production KVM server "hardening"?:
@Pete-S said in Production KVM server "hardening"?:
Thanks! My intention was to put the management network behind a VPN - with MFA for human access.
Doesn't do anything. SSH is already in a VPN. MFA is great, but just add that to SSH. VPN won't add anything to SSH, but it risks the feeling that you can use it for other things and it tends to become the point of risk.
VPN is used to get access to the private network. Servers are not accessible from internet. So there is no way to use ssh without first having access to the private network.
So double encryption was just a byproduct.
I get that. My point was that you'd get the same security without the private network or the VPN. They only give an illusion of additional security, but cause a lot more effort and more effort often results in work arounds.
If keep SSH from being access to only places that can access the VPN, you get all the benefits that you have with the VPN at a fraction of the effort both up front and in utilization. VLANs and VPNs sound like they are providing additional security but in a case like this, I don't see it actually adding anything. It's just that people typically lock down a VPN in one way and SSH in another, but if you apply the combined rules just to the SSH to get easy to use, with all of the security.
-
@Pete-S said in Production KVM server "hardening"?:
VPN is used to get access to the private network. Servers are not accessible from internet.
This is the part that is weird. Like.... if we look at it from the outside, it's the same...
Step One: Access exposed port
Step Two: Access serverThat behind the scenes it's a private network is irrelevant to someone attempting to access it. In both cases it's a published port that is heavily locked down with encryption and MFA. If you want, you can call the SSL layer of SSH a VPN (it literally is, in every sense) and you can call the inside of that tunnel a private network (it is in any meaningful sense) and voila, the two thigns are the same other than the double encryption which isn't "bad" but isn't beneficial either (typically.)
A bit of pain, no gain. But putting on my security auditor's hat.... it's a bit of "making people annoyed at pointless security" which is, itself, a huge security hole that creates risk. When you make security too onerous, and especially if there is no security justification for it, you typically create both a business need and an emotional desire to circumvent the security.
-
For example, when IT firms are forced to use VPNs to customers, they often circumvent them by making shared machines that have the VPNs open all of the time taking something that "seems" like it would increase security and causing it to be a big opening because they have a job to do and to secure themselves they don't want the expose to themselves of the VPN.
In nearly all cases, over time, I see VPNs being used to circumvent the very security people thought that they were in place to create.
-
@scottalanmiller said in Production KVM server "hardening"?:
I get that. My point was that you'd get the same security without the private network or the VPN. They only give an illusion of additional security, but cause a lot more effort and more effort often results in work arounds.
Yeah, I agree. Anyway, I don't want to go down the rabbit hole of network design specifics at this time. Might circle back to that later though.
My primary concern right now is if there are any special configuration needed to run pure minimal linux KVM virtualization hosts in production in a responsible manner.
-
@Pete-S said in Production KVM server "hardening"?:
My primary concern right now is if there are any special configuration needed to run pure minimal linux KVM virtualization hosts in production in a responsible manner.
Nothing too specific. Very much just general Linux hardening which is just general systems hardening.
-
@scottalanmiller said in Production KVM server "hardening"?:
@Pete-S said in Production KVM server "hardening"?:
VPN is used to get access to the private network. Servers are not accessible from internet.
This is the part that is weird. Like.... if we look at it from the outside, it's the same...
Step One: Access exposed port
Step Two: Access serverThat behind the scenes it's a private network is irrelevant to someone attempting to access it. In both cases it's a published port that is heavily locked down with encryption and MFA. If you want, you can call the SSL layer of SSH a VPN (it literally is, in every sense) and you can call the inside of that tunnel a private network (it is in any meaningful sense) and voila, the two thigns are the same other than the double encryption which isn't "bad" but isn't beneficial either (typically.)
A bit of pain, no gain. But putting on my security auditor's hat.... it's a bit of "making people annoyed at pointless security" which is, itself, a huge security hole that creates risk. When you make security too onerous, and especially if there is no security justification for it, you typically create both a business need and an emotional desire to circumvent the security.
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSH -
@Dashrender said in Production KVM server "hardening"?:
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSHYes, it has MFA. But you can have MFA on just SSH too. So while yes, you are correct that it does that, but not that it adds something special.
-
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSHYes, it has MFA. But you can have MFA on just SSH too. So while yes, you are correct that it does that, but not that it adds something special.
Well - MFA adds a third thing to the list of what I said.
-
@Dashrender said in Production KVM server "hardening"?:
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSHYes, it has MFA. But you can have MFA on just SSH too. So while yes, you are correct that it does that, but not that it adds something special.
Well - MFA adds a third thing to the list of what I said.
No, it's already MFA. MFA is not an additional thing. You literally said "doesn't this allow for two different authentications", that's literally MFA.
-
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSHYes, it has MFA. But you can have MFA on just SSH too. So while yes, you are correct that it does that, but not that it adds something special.
Well - MFA adds a third thing to the list of what I said.
No, it's already MFA. MFA is not an additional thing. You literally said "doesn't this allow for two different authentications", that's literally MFA.
Well then I meant three - the first logon to VPN (which is actually two verifications - creds and MFA) and then a - well I guess - third which is creds to SSH.
-
@Dashrender said in Production KVM server "hardening"?:
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
@scottalanmiller said in Production KVM server "hardening"?:
@Dashrender said in Production KVM server "hardening"?:
Doesn't his setup allow for two different authentications to be required? Assuming I'm right there, wouldn't that be another layer?
i.e. layer one creds (cert likely) at VPN
layer two creds (cert likely) at SSHYes, it has MFA. But you can have MFA on just SSH too. So while yes, you are correct that it does that, but not that it adds something special.
Well - MFA adds a third thing to the list of what I said.
No, it's already MFA. MFA is not an additional thing. You literally said "doesn't this allow for two different authentications", that's literally MFA.
Well then I meant three - the first logon to VPN (which is actually two verifications - creds and MFA) and then a - well I guess - third which is creds to SSH.
But then add three to SSH. Using totally unrelated technologies to layer authentication is valid, but a weird brute force method to get there. If your goal is just MFA, just do MFA in an elegant, efficient, easy way. Want two factor, do a key and passkey. Want a third, add Duo or Google Auth or Authy. Want a forth, text your phone or send an email. Want a fifth, do whatever forth one you didn't do. Want a sixth, have a script check that you are clocked in and at your desk. Want a seventh, do IP locking.
You can get more MFA factors from any one mechanism than you can use. VPN is often used to get MFA without someone realizing that that is what they were trying to accomplish or without realizing that that is where they are seeing the benefit. And because of that, because it's often done without any evaluation of what is really wanted, it rarely fits the need well. Is it MFA? Yes. Is it a good way to get that MFA? Not really. It's okay, but it isn't great. Lots of overhead to do something fundamentally pretty simple. And it makes the MFA location dependent (in most deployments.) You can bypass the MFA by changing your physical location. In most companies that do this, because they don't realize it is MFA that they are trying to do, they make it really easy to bypass the MFA for most people.
-
@Pete-S said in Production KVM server "hardening"?:
I'm thinking about running pure KVM on debian for virtualization hosts. Not Proxmox. There will be no GUI on the servers, no web interface, only ssh for management.
Do I need to do anything special to lock down the security?
I've never used KVM in production, only on my desktop and then I've had virt-manager as well as tools like virtsh. So I don't really know what is required for a pure KVM server to be as "secure" as proxmox, xcp-ng or whatever.
Keep the OS and everything updated. Keep drivers updated. Keep firmware updated. Use only key-based auth for SSH, add only specific devices to authorized_keys file. Ensure firewall configured well. Set up log alerts for access.
