Looking for some insight/input for setting up a "hot site"/fail-over environment
-
I have my company's primary site and we have backups running both local and in cloud storage. I am working on devising a new plan for a local "hot site" at one of our other buildings because I believe that having to restore from backups only is less than desirable since it will take time and effort to manually restore everything.
Here's our general setup:
Primary Site:
-
VMware vSphere Essentials 6.5 Plus Environment:
- Storage Controller: Dell SCv2020 (Compellent/Mirage)
- Compute: 2x Dell PowerEdge Servers (ESXi)
- Networking: 1Gbps for both iSCSI & VM Network
-
Physical Servers: ~5 x Windows 2016 servers
-
Everything sits in a cage with dedicated network switches to feed the rest of the environment
-
Backups are Veeam Backup & Restore Standard for both virtual and physical environment
Options for replicating live systems for a "hot site":
-
Could replicate data at the storage controller level:The SCv2020 does have embedded REPL ports specifically for replicating to another SCv2020 via iSCSI - this requires more additional licensing which is fine
-
I think the SCv2020 is EoL so I'm not sure if I would buy a new one or upgrade/buy double or if the SCv2020 can replicate to anything else
-
Could replicate virtual machines at the VMware level but we will need a new vSphere license, setup may be more complex (also fine)
- Options for restoring backups:
- I am already replicating Veeam backups to secondary site
- the only issue is that there is a less than desirable RTO/RPO vs having a hot site that is ready to go
I am leaning towards replicating machines at the VMware level since it might be the most hardware/environment agnostic. Replicating at the storage controller level may be easier but lead to difficulty down the road. The only other thing I need to worry about is how to replicate/ready the physical servers.
I'm just wondering if anyone here has any knowledge/experience with setting up a hot site and how you might suggest approaching this. I'm trying to figure out the best way to replicate system states in order to have things on "stand-by" in the event of a DR situation. And I do know that DR can be quite extensive rabbit hole but right now I am thinking in terms of if our building were to burn down. We have a few locations that are all relatively close by so I really want to utilize one of those for our hot site. And as mentioned, we do have a cloud service provider but we'd likely only really utilize them if we had no other options for local restore.
-
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
Dell SCv2020
Do you really need dedicated storage instead of local storage?
-
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
Dell SCv2020
Do you really need dedicated storage instead of local storage?
Thanks but I'm not getting into that argument.
-
What type of Windows licensing do you have? Replication can be affected by that.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
Dell SCv2020
Do you really need dedicated storage instead of local storage?
Thanks but I'm not getting into that argument.
If your SCv2020 is EOL, now is a good time to see if you really need a replacement.
-
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
What type of Windows licensing do you have? Replication can be affected by that.
We have Microsoft Volume Licensing and I haven't looked into weather or not we'd have to purchase "double" or not.. I'm not sure but I thought a backup environment doesn't quite count as production.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
What type of Windows licensing do you have? Replication can be affected by that.
We have Microsoft Volume Licensing and I haven't looked into weather or not we'd have to purchase "double" or not.. I'm not sure but I thought a backup environment doesn't quite count as production.
I would say "Hot"=Active.
-
@dave247 BTW, thanks for the downvote lol
-
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
What type of Windows licensing do you have? Replication can be affected by that.
We have Microsoft Volume Licensing and I haven't looked into weather or not we'd have to purchase "double" or not.. I'm not sure but I thought a backup environment doesn't quite count as production.
I would say "Hot"=Active.
hot/warm/standby environment is more what I'm talking about. I do plan to look into the licensing requirements involved. We have Datacenter for the vmware environment.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
We have Datacenter for the vmware environment
You should be good with datacenter if you have it for every server involved in the scenario.
-
I think I might go ahead and use Veeam's VM replication for a warm site setup.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
We have a few locations that are all relatively close by
That is probably the craziest thing about going through all this hassle and being in the same geographic region. I would seriously consider a colo outside of the area. I mean you do all that work and put in all these expenses and you have an area wide disaster and its potentially gone.
Colocation costs are not much more than hosting at your own facility. -
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
Could replicate data at the storage controller level:The SCv2020 does have embedded REPL ports specifically for replicating to another SCv2020 via iSCSI - this requires more additional licensing which is fine
Does this maintain anything other than crash consistency? Unless the SCv2020 is just automating VMware to do the work, this isn't a failover.
-
@IRJ said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
We have a few locations that are all relatively close by
That is probably the craziest thing about going through all this hassle and being in the same geographic region. I would seriously consider a colo outside of the area. I mean you do all that work and put in all these expenses and you have an area wide disaster and its potentially gone.
Colocation costs are not much more than hosting at your own facility.Sometimes, it's even less! Not often, but it can happen.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
I think I might go ahead and use Veeam's VM replication for a warm site setup.
Way safer, and probably cheaper.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
What type of Windows licensing do you have? Replication can be affected by that.
We have Microsoft Volume Licensing and I haven't looked into weather or not we'd have to purchase "double" or not.. I'm not sure but I thought a backup environment doesn't quite count as production.
I would say "Hot"=Active.
hot/warm/standby environment is more what I'm talking about. I do plan to look into the licensing requirements involved. We have Datacenter for the vmware environment.
Just an FYI / PSA... if you use Scale HC3, this kind of replication and failover is all included in the base purchase.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@wrx7m said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
What type of Windows licensing do you have? Replication can be affected by that.
We have Microsoft Volume Licensing and I haven't looked into weather or not we'd have to purchase "double" or not.. I'm not sure but I thought a backup environment doesn't quite count as production.
You need the second site licensed if it is hot, meaning turned on. You do not need anything if it is cold, meaning powered off. Powered off, it's just a backup that's extremely "ready to go".
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
I think the SCv2020 is EoL so I'm not sure if I would buy a new one or upgrade/buy double or if the SCv2020 can replicate to anything else
Typically replication requires same to same, but not absolutely always.
-
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
Could replicate virtual machines at the VMware level but we will need a new vSphere license, setup may be more complex (also fine)
Replication always has to be at this level. You can't safely replicate below it. Storage doesn't have the ability to replicate on its own. If your VMware licensing doesn't allow for replication, that answers all of the questions about the storage lower down the stack. Even if the SC has the ability to replicate, it depends on this feature of VMware to execute it correctly. If you do it without VMware doing the heavy lifting you get inconsistent data. Often looks good in tests, often corrupt in a real life failover.
-
@IRJ said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
@dave247 said in Looking for some insight/input for setting up a "hot site"/fail-over environment:
We have a few locations that are all relatively close by
That is probably the craziest thing about going through all this hassle and being in the same geographic region. I would seriously consider a colo outside of the area. I mean you do all that work and put in all these expenses and you have an area wide disaster and its potentially gone.
Colocation costs are not much more than hosting at your own facility.yeah that would be preferred but we likely won't be able to get our company owners to spend that much money. Setting up a minimal warm site in our other building would be much cheaper/easier.
