Handling DNS in a Single Active Directory Domain Controller Environment

[JaredBusch]

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

3. Jared's solution... point your router to your internal DNS first, then to public DNS. This handles failover and deals with any concerns of Windows DNS stack being flaky. When AD is up, internal DNS works, when it goes down, you transparently fail to public DNS.

Here is how I handle it in EdgeOS

set service dns forwarding cache-size 150
set service dns forwarding listen-on eth1
set service dns forwarding name-server 10.1.1.show 4
set service dns forwarding name-server 1.1.1.1
set service dns forwarding name-server 8.8.8.8
set service dns forwarding options server=/domain.local/10.1.1.4
set service dns forwarding options server=/domain/10.1.1.4
set system domain-name domain.local 
set system name-server 127.0.0.1

And the few absolute core items that I need to 100% resolve get a static host mapping in EdgeOS.

set system static-host-mapping host-name dc02 inet 10.1.1.4
set system static-host-mapping host-name dc02.domain.local inet 10.1.1.4
set system static-host-mapping host-name domain.local inet 10.1.1.4
set system static-host-mapping host-name pbx.domain.com inet 10.1.1.30

[Mike Davis]

@jaredbusch said in Handling DNS in a Single Active Directory Domain Controller Environment:

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

3. Jared's solution... point your router to your internal DNS first, then to public DNS. This handles failover and deals with any concerns of Windows DNS stack being flaky. When AD is up, internal DNS works, when it goes down, you transparently fail to public DNS.

Here is how I handle it in EdgeOS

set service dns forwarding cache-size 150
set service dns forwarding listen-on eth1
set service dns forwarding name-server 10.1.1.show 4
set service dns forwarding name-server 1.1.1.1
set service dns forwarding name-server 8.8.8.8
set service dns forwarding options server=/domain.local/10.1.1.4
set service dns forwarding options server=/domain/10.1.1.4
set system domain-name domain.local 
set system name-server 127.0.0.1

And the few absolute core items that I need to 100% resolve get a static host mapping in EdgeOS.

set system static-host-mapping host-name dc02 inet 10.1.1.4
set system static-host-mapping host-name dc02.domain.local inet 10.1.1.4
set system static-host-mapping host-name domain.local inet 10.1.1.4
set system static-host-mapping host-name pbx.domain.com inet 10.1.1.30

That's smart thinking if you have an on prem PBX. They can still use the phones even if the DC (and primary DNS) is down.

[scottalanmiller]

@mike-davis said in Handling DNS in a Single Active Directory Domain Controller Environment:

That's smart thinking if you have an on prem PBX. They can still use the phones even if the DC (and primary DNS) is down.

Exactly. And that's exactly why we do something similar. We want our PCs and phones online even when other things are down.

And honestly, for things like PBX that is off prem, we never use the in house DNS for the phones at all. Point them straight to CloudFlare or Google and if AD fails, no one knows (from a phone perspective.)

[Donahue]

In our DHCP settings (hosted by the same DC) I currently have 5 DNS servers defined. My two DC's, then the two public DNS from our ISP, then finally 8.8.8.8. But in my experience, if my DC is down, I've got larger issues than just DNS. But when this has happened, I just start up the DHCP server on the router and define 8.8.8.8 as the DNS for everyone while I fix this issue with the DC. I am not sure we have it configured correctly though, because it never seems to actually fail over to the other DC. It could be because its in another subnet at a different location, not sure.

[Mike Davis]

I have a number of clients where they need a server, but Server Essentials on a small server is enough. Veeam for backup and if the box fails, they are down for an hour or two while we restore to something else. The licensing to go to a second AD server would more than double the cost of the project. (and isn't worth it for them)

[scottalanmiller]

@mike-davis said in Handling DNS in a Single Active Directory Domain Controller Environment:

I have a number of clients where they need a server, but Server Essentials on a small server is enough. Veeam for backup and if the box fails, they are down for an hour or two while we restore to something else. The licensing to go to a second AD server would more than double the cost of the project. (and isn't worth it for them)

Great example, that's what we see a lot, even with "larger than Essentials" shops. The downtime is still short, getting new hardware is trivial (a big desktop could do it), a single server restore is simple, and the cost to avoid that pain is huge compared to the cost of the downtime.

[scottalanmiller]

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

In our DHCP settings (hosted by the same DC) I currently have 5 DNS servers defined. My two DC's, then the two public DNS from our ISP, then finally 8.8.8.8. But in my experience, if my DC is down, I've got larger issues than just DNS. But when this has happened, I just start up the DHCP server on the router and define 8.8.8.8 as the DNS for everyone while I fix this issue with the DC. I am not sure we have it configured correctly though, because it never seems to actually fail over to the other DC. It could be because its in another subnet at a different location, not sure.

That might be the cause. Do you have them set up as different sites?

I would avoid using the ISP DNS, that's generally slow and risky. Use Google and CloudFlare for your "other" DNSs on the outside. Fast and secure. And each provides two, so more than enough to replace what you have.

[JaredBusch]

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

In our DHCP settings (hosted by the same DC) I currently have 5 DNS servers defined. My two DC's, then the two public DNS from our ISP, then finally 8.8.8.8. But in my experience, if my DC is down, I've got larger issues than just DNS.

I would never put public DNS entries in anything related to a client system running a Windows Desktop OS. I have had too much direct experience with Windows' network stack flipping to the secondary DNS server and not flipping back to ever want to deal with that crap. It could be better with Windows 10, but I am not going to change now to find out.

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

But when this has happened, I just start up the DHCP server on the router and define 8.8.8.8 as the DNS for everyone while I fix this issue with the DC. I am not sure we have it configured correctly though, because it never seems to actually fail over to the other DC. It could be because its in another subnet at a different location, not sure.

DHCP is a different but related issue.

[Donahue]

I dont think I've ever set it all up correctly. I thought I setup the sites correctly, but that was a few years ago and I really had no idea what I was doing. I've been too busy to really worry about it since then and the DC has been pretty reliable. Squeaky wheel and all that.

[Obsolesce]

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

Given the premise of handling DNS failover scenarios it seems like having two DCs would be better, but I'm open to being convinced.

So there are a few scenarios here. But here are some general approaches to this that I've seen (thanks to @JaredBusch for the big one...)

1. In really small AD environments, there might be zero internal DNS needs and Windows DNS is for AD only. In which case, point to the router's DNS or a public DNS and you are done, easy peasy. Free, simple, universal.
2. In environments with internal DNS dependencies, much of the time the impact of losing that is trivial and you just have workers run in a reduced functionality mode until AD DC is fixed. Impact can be so small as to not measure. An option if you are very small and don't need external web access (super rare.)
3. Jared's solution... point your router to your internal DNS first, then to public DNS. This handles failover and deals with any concerns of Windows DNS stack being flaky. When AD is up, internal DNS works, when it goes down, you transparently fail to public DNS.
4. Use zone transfer to another DNS server. This could just be to your router. Have your router or some other "free or already included" system keep a copy of Windows' DNS and use this as your secondary. Full DNS redundancy (without active updates) while the AD DC is down. You could build a DNS server just for this, but nearly all networks have one already available. But a Raspberry Pi will do this easily if you really have no other hardware.

This number 3 is the big one. In the time it takes to handle 1, 2, or 4, you can have the DC/DNS VM restored.

[scottalanmiller]

@obsolesce said in Handling DNS in a Single Active Directory Domain Controller Environment:

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

Given the premise of handling DNS failover scenarios it seems like having two DCs would be better, but I'm open to being convinced.

So there are a few scenarios here. But here are some general approaches to this that I've seen (thanks to @JaredBusch for the big one...)

1. In really small AD environments, there might be zero internal DNS needs and Windows DNS is for AD only. In which case, point to the router's DNS or a public DNS and you are done, easy peasy. Free, simple, universal.
2. In environments with internal DNS dependencies, much of the time the impact of losing that is trivial and you just have workers run in a reduced functionality mode until AD DC is fixed. Impact can be so small as to not measure. An option if you are very small and don't need external web access (super rare.)
3. Jared's solution... point your router to your internal DNS first, then to public DNS. This handles failover and deals with any concerns of Windows DNS stack being flaky. When AD is up, internal DNS works, when it goes down, you transparently fail to public DNS.
4. Use zone transfer to another DNS server. This could just be to your router. Have your router or some other "free or already included" system keep a copy of Windows' DNS and use this as your secondary. Full DNS redundancy (without active updates) while the AD DC is down. You could build a DNS server just for this, but nearly all networks have one already available. But a Raspberry Pi will do this easily if you really have no other hardware.

This number 3 is the big one. In the time it takes to handle 1, 2, or 4, you can have the DC/DNS VM restored.

Well, but you do them ahead of time

[scottalanmiller]

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

I dont think I've ever set it all up correctly. I thought I setup the sites correctly, but that was a few years ago and I really had no idea what I was doing. I've been too busy to really worry about it since then and the DC has been pretty reliable. Squeaky wheel and all that.

You need someone to come in and take a look at some of that stuff!

See you in a few hours

[Donahue]

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

I dont think I've ever set it all up correctly. I thought I setup the sites correctly, but that was a few years ago and I really had no idea what I was doing. I've been too busy to really worry about it since then and the DC has been pretty reliable. Squeaky wheel and all that.

You need someone to come in and take a look at some of that stuff!

See you in a few hours

That's like number 87 on my priority list though.

[scottalanmiller]

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

@scottalanmiller said in Handling DNS in a Single Active Directory Domain Controller Environment:

@donahue said in Handling DNS in a Single Active Directory Domain Controller Environment:

I dont think I've ever set it all up correctly. I thought I setup the sites correctly, but that was a few years ago and I really had no idea what I was doing. I've been too busy to really worry about it since then and the DC has been pretty reliable. Squeaky wheel and all that.

You need someone to come in and take a look at some of that stuff!

See you in a few hours

That's like number 87 on my priority list though.

More of a Wednesday item

[Kelly]

@scottalanmiller It doesn't seem like you're interested in discussing my comment in good faith. Your progressive inflation of costs in response to my posts is an interesting rhetorical technique, but not a great way of doing a discussion based on the merits. If you have a basis for the costs you're postulating, then let's discuss them.

Scenario: SMB that depends on AD. Has one VM host. Typical SMB configuration would have DNS and DHCP running on the same Windows install as AD. At that size you could go either way for IT support, but for the sake of discussing costs on the lower side we'll say that they have someone in house with sufficient skill to install and manage Windows, AD, DNS, and DHCP. We'll set their wages at $25/hr. We will also assume that they are currently maxed so that any emergency is an additional cost, and not just absorbed by their daily responsibilities. (Feel free to dispute my assumptions with justifications)

Second DC:
Server license: MSRP $699.99 (probably find it for less, but we'll go with this)
Hardware: $1,000 (If all you need is a failover DC then you could run this on old hardware, a desktop, etc. for easily less than $500, but we'll go for an upper bound)

So, the upper end of costs for a second DC in my scenario is $1,700 and then say, 8 hours of labor (probably less), so another $200. Total cost to purchase and implement is $1,900 at the most.

If we assume that this is a typical SMB, sans the @scottalanmiller touch, then DNS and DHCP are also running on the DC VM. If the hardware suffers a catastrophic failure what is the cost to restore it? We won't factor in any replacement costs because those are the same whether we have a second DC or not.

So, backups. Here is where we may differ a bit. If there is a backup solution that is up to the task of backing up AD and restoring it without any additional issues then I would postulate that the business would have room on that hardware to run a second DC running as a Core install. However, if we are wanting to see the most extreme comparison would be, we will assume that there is a good backup of AD to restore, but not the hardware to run a second DC. This keeps our costs for a second DC still at $1,900.

VM host suffers hardware failure taking down AD. From this point it is speculation. If there aren't sufficient resources to spin up a second DC then there isn't anything to restore the back up to either. This means that there is no AD, DNS, or DHCP until something is put into place. Since every business class edge device can handle the latter two, we'll assume that they get moved over by our on site person. The time to do this is extremely variable. Let's grab an hour as the lower bound. So $25 to get DNS and DHCP up and running so that there is external access, plus employee labor impact. DHCP will continue to be fine during that one hour unless a lease comes up. DNS, however, means that most modern devices will stop functioning right away. A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.

At this point, all internal services are down until AD is restored. Another variable that is difficult to account for. The most prevalent one of these would be printers. Impact will vary from business to business. If we say that of those 10 employees, 2 require (whether from felt personal need, or actual professional need it doesn't matter) a printer multiple times per day. How does 5 times sound? There are work arounds. Our enterprising technician goes to each machine to edit their hosts file to allow the users to print. Between getting all the information, figuring out the changes, coordinating with employees, and actually doing the work we'll say it takes an hour, so another $25.

So, our loss of our only DC has cost the company a minimum of $200. These costs can skyrocket depending on the number of employees, their pay rates, and the numbers of internal services. If this is our baseline then almost $2,000 is probably more than a 10 employee company should spend, so I agree with you. If the curves fluctuate much at all (lower priced/repurposed hardware), more employees, more costly employees, more internal service, then the cost/risk ratio approaches 1 very quickly.

Yes, you can defray or reduce almost all of the points of impact, like configuring your DNS scenario the way @JaredBusch suggested, but that is not a common approach. Implementing it and other things to negate potential impacts of down time (scripting system capable of pushing out a new hosts file to clients reducing down time to less than 1 hour, etc.) have their own costs inherent to them, and are also not common in SMB, so I don't think that including them in the comparison of the two approaches.

[scottalanmiller]

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

So, backups. Here is where we may differ a bit. If there is a backup solution that is up to the task of backing up AD and restoring it without any additional issues then I would postulate that the business would have room on that hardware to run a second DC running as a Core install.

Many that I see use consumer or prosumer NAS devices for this, or even USB drives. AD doesn't have any special considerations for backup, it's a normal live/live database. The included Windows backup tools, free Veeam agent, etc. will back up AD no problem.

I don't know what average is here, but in companies that I've seen the majority have backups that will handle AD no problem, but many, but far from most, have redundant hardware to run on. But if they do, they could using just their simple backup mechanisms.

[Obsolesce]

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

Edit: The $1,500 to $2,000 might seem like quite a bit, but those costs are pretty minimal compared to down time and repair costs if you do suffer a catastrophic failure.

A catastrophic DC failure in a single-DC environment means you have to spend 5-10 minutes restoring it from backup. Then you are done. As it's the only DC, you won't run into any of those weird multi-DC issues. That hardly justifies $2k to start for a SMB that has never pointed out a reason to need HA DC services.

[scottalanmiller]

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

If there aren't sufficient resources to spin up a second DC then there isn't anything to restore the back up to either. This means that there is no AD, DNS, or DHCP until something is put into place. Since every business class edge device can handle the latter two, we'll assume that they get moved over by our on site person. The time to do this is extremely variable. Let's grab an hour as the lower bound. So $25 to get DNS and DHCP up and running so that there is external access, plus employee labor impact. DHCP will continue to be fine during that one hour unless a lease comes up. DNS, however, means that most modern devices will stop functioning right away. A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.

Correct, so we reasonably have to assume downtime until "new hardware is procured."

DNS can be pretty easily handled in a number of ways. Assuming you are a single AD shop, this should be done proactively, not reactively. So there should be no downtime or cost here, not even of labour. Whether you need internal DNS and your secondary is the edge device, or you have no internal needs and just go to Google or whatever, that should be "handled". Sure you can't update DNS during this time, but there is no downtime.

DHCP is similar. If you are running in a single DHCP environment, set leases to something long, say 72 hours, and you will always have a day or two to worry about a lease. We assume that you'll have new hardware up in under, say, 36 hours. Maybe that is a bad assumption, but there it is.

So DNS and DHCP should be "handled" with a little forethought. No more forethought than is required to put in a second AD server, so that's a draw IMHO.

I think we can agree... if someone screws up their environment, makes dependencies on things like DNS or DHCP being up continuously to do basic things like print, and let's them go down without any planning that money will be lost. But I think the idea here is that with both of those (DNS specific to this thread) that there are trivially easy, and free ways, to proactively ensure that enough DNS, which might be all of it or just some of it or even none in extreme cases, can be provided during an outage to ensure zero to nominal impact.

In a tiny environment, only ten people, you might even just use IP addresses for things like printers.

[scottalanmiller]

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

At this point, all internal services are down until AD is restored. Another variable that is difficult to account for.

Just to be clear, I work with many single AD environments and one of this happen. Most have some impact, none stop working. Each is different, some have no impact, for others it is pretty annoying. But none have large financial impacts for the first one to two days without AD working.

So while some setups that create a lot of AD dependencies could have this risk, it's not a given. And in all cases, we have to assume that the environment is set up appropriately for the situation chosen.

[scottalanmiller]

@kelly said in Handling DNS in a Single Active Directory Domain Controller Environment:

A business of 10 employees averaging $15/hr means that there is a cost of $150 for that down time.

This is a handy number to use because it is easy to price. But it's important to remember that it's kind of an average. Each business will be unique. One business will lose only $5/hr of productivity because their workers were idle anyway (I've literally had CEOs tell me this.) Another will lose $1,000/hr because while those workers only get paid $15, they earn $115 for the company. That sort of thing.

It's a handy way to discuss likely impacts quickly. Just don't want anyone to think that the per hourly price of an employee directly ties to impact numbers. It is sometimes lower, but often much higher, than their hourly rate, even their loaded cost.

I'd say more of an industry average that a $15/hr hour worker would actually cost a company more like $35/hr should they be forced to go truly idle for an extended period of time (but most can offset a few minutes of idleness by getting coffee or taking lunch early.)