Risk: 3-2-1 Stock Inverted Pyramid Design
-
Following on from Risk: Single Server versus the Smallest Inverted Pyramid Design
Scenario: Three servers, two switches and a single NAS or SAN as shared storage. High Availability solution applied to the servers so that if one server dies the other can immediately spin up the failed VM. (This is a 3-2-1 Inverted Pyramid design.)
Assumptions: We assume the following...
- Server means an enterprise class commodity server, a standard server like the HPE DL380 or Dell R730. Anything in that general category of reliability.
- NAS or SAN is a comparable price range unit which would include unified storage offerings from companies like Synology or ReadyNAS as well as dedicated SAN from Dell and HPE.
- The logic in considering the 3-2-1 IPOD design is because the "high availability" checkmark was required.
- That the baseline for standard availability is defined as the level of availability associated with a standard enterprise server.
The 3-2-1 design is a salesman dream as it creates high margins and plays on business fears and emotional responses. It's the easiest sale with the highest profit margins for resellers so has become the stock model to push regardless of its total waste of money and dangerous design.
The name 3-2-1 (and the term inverted pyramid) are derived from this design which entails three or more servers on "top" facing the business in a "high availability layer", two storage switches in a second "high availability layer" and on the bottom of our stack is a storage device, typically a SAN but potentially NAS. Each "layer" is an independent failure domain or "link" in a dependency chain. All three layers are in a chain ... which means that all three are critical. If any link in the chain fails the entire chain fails.
What is most critical is that we realize that while the risk of any given layer in our design is low, the risk is additive. The risk of each layer is added together to form the total risk profile. It is also important to note that the interfaces between layers creates a point of risk.
It is impossible to determine the exact risk of each layer individually because we do not know the level to which the failure risk is mitigated. If X is the failure rate of a single component then we know that the top layer, our three servers, has a failure rate that is going to be very low, much less than X, but far from zero as well. (As this layer grows to more and more servers the risk increases, rather than decreases. In theory, with enough servers the risk would be X again, but that would be an unreasonably large number and the risk at this layer changes behaviour and is workload dependent.)
In the switching layer our switches are rarely X but are often much more fragile, closer to 2X. Switches fail far more often than enterprise servers due to a number of factors. Very high end switches will mitigate this, but this increase cost and cost is a form of risk. While each switch alone is a >X risk, when put together into an HA solution the overall layer is much lower than X in risk, but greater than zero and greater than the risk from the layer above in many cases.
The bottom layer is the most critical one. The storage layer is the most difficult to protect and the most critical as all other layers represent loss of availability risk and the storage layer represents this as well as loss of data risk. A SAN here, or NAS, is just another server. The risk of this layer, being unmitigated (dual controllers is not a risk mitigation measure) remains X. For many reasons, it is actually likely to be somewhat greater than X, but for the point of the discussion we will give it the benefit of the doubt and simply call it X to be generous.
So if X is the standard risk of a component and Y is a risk that is greater than zero but far less than X we can represent each of the two upper layers as having a Y risk as we cannot accurately determine their relationship to one another in a generic example. We can now express our total risk as being X + 2Y. That is risk as a "chance of failure." So our risk is much higher than just X (a single server on its own) and a little higher (roughly Y higher) than the X + Y risk that we showed for the 2+1 IPOD model using a DAS.
The cost of a 3-2-1 is hard to say for sure as there are so many factors, but roughly we can assume that each server costs X, that the two switches together are around X and that the SAN is likely 2X or more (3X or more is very common.) Even if the SAN is only X, we have a total system cost of 5X or 500% the cost of a single server. Because of the assumption of N+1 design for the top layer, we assume that if we have three servers in this design that two servers are needed to handle capacity. So the minimum expense that we can have is 2X or 200% the cost of a single server without redesigning the server layer.
So the best case cost scenario for a 3-2-1 is roughly 250% higher than doing two individual servers. And likely far more than that as 500% is the lowest reasonable cost and 800% is more likely in the real world. That is a massive cost increase for something that is more risky than using a single server alone!
To add to this, using two individual servers, each holding half of the workloads means that any failure impacts only 50% of systems. This dramatically offsets risk in most cases through no action other than simplifying the environment.
Adding from the previous article as this information is often missed:
Many people like to think or claim and vendors will happily feed into this belief, that storage devices, especially SANs, are "magic" and not subject to the same risks as normal servers even though they are just servers themselves. A typical unified storage device is built on SuperMicro servers or similar which puts the risk profile nearly identical to that of a standard enterprise server. This is, indeed, a standard server in every way.
It is very common to look to dedicated SANs that are slightly more expensive as also being infallible sometimes simply because of the name SAN (which is only a reference to its network protocols) and sometimes because of the assumption that dual controllers will break the rules of risk and make the single device effectively riskless. Dual controllers are not used in standard enterprise servers for a reason - they generally add no value and often create additional risk. Indeed in non-active/active controllers (the only ones that can be remotely considered in this price range and scale) dual controllers are shown to routinely create disasters far more often than standard servers fail.
Most storage devices in this range also lack the support options that enterprise servers have. This is not a problem built into the solution type but into common choices in the approach to this layer so should not generally be calculated as a rule. But it is very common to see people say that they must avoid five minutes of downtime at the server layer but will select a storage device with a two week SLA on repairs - and will not be returned with the data intact!
Because of the combinations of lower production rates, less testing, dual controller induced failure, special case software and more the storage layer is actually normally quite a bit more risky than the server layer even with comparable hardware. So we are actually overly generous to the IPOD solution approach by calling this risk X, when in fact it is generally quite a bit higher, possibly 2X or more!
It should be noted that it is possible to move up to very high end, very expensive storage devices that will mitigate a large portion, but never all, of this risk. But in a 2+1 design the cost would normally double or more the entire product cost and are effectively unthinkable as far better risk mitigation strategies can be done that are both less risky and far less costly in other ways.
-
@scottalanmiller said:
(dual controllers is not a risk mitigation measure)
Great explication.
Why dual controller are not a risk mitigation?
What's best alternative to this example ?
-
@iroal said:
Why dual controller are not a risk mitigation?
Here is an expert drawing to illustrate the problem.
How can dual controllers go wrong.
-
@Breffni-Potter said:
@iroal said:
Why dual controller are not a risk mitigation?
Here is an expert drawing to illustrate the problem.
How can dual controllers go wrong.
Motherboard is not included in the controller?
-
Dual controllers don't mean anything when you lose the chassis though
-
@iroal said:
@Breffni-Potter said:
@iroal said:
Why dual controller are not a risk mitigation?
Here is an expert drawing to illustrate the problem.
How can dual controllers go wrong.
Motherboard is not included in the controller?
They do, the risk is from close coupling, not from lack of redundancy. I'll get a post together.
-
@iroal said:
@scottalanmiller said:
(dual controllers is not a risk mitigation measure)
Great explication.
Why dual controller are not a risk mitigation?
What's best alternative to this example ?
http://mangolassi.it/topic/8822/why-dual-controllers-is-not-a-risk-mitigation-strategy-alone
-
Most storage devices in this range also lack the support options that enterprise servers do.
This sentence in the third italicized block of text seems odd.
-
@Dashrender said:
Most storage devices in this range also lack the support options that enterprise servers do.
This sentence is the third italicized block of text seems odd.
Fixed, thanks.
