@scottalanmiller said:
@donaldlandru said:
I agree we do lack true HA in the production side as there is a single weak link (one storage array), the solution here depends on our move to Office 365 as that would take most of the operations load off of the network and change the requirements completely.
Good deal. We use O365, it is mostly great.
If I can sell them on Office 365 this time around (third times a charm), but that is for a different thread
@scottalanmiller said:
@donaldlandru said:
@scottalanmiller said:
@donaldlandru said:
Which I can add for as cheap as $5k with RED drives or $10k with Seagate SAS drives.
WD makes RE and Red drives. Don't call them RED, it is hard to tell if you are meaning to say RE or Red. The Red Pro and SE drives fall between the Red and the RE drives in the lineup. Red and RE drives are not related. RE comes in SAS, Red is SATA only.
It's all in a name. When I say REDs I am referring to WD Red 1TB NAS Hard Drive 2.5" WD10JFCX. When I say seagate I am referring to Seagate Savvio 10K.5 900 GB 10000 RPM SAS 6-Gb/S ST9900805SS
Edit: I don't always use WD NAS (RED) drives, but when I do I use the WDIDLE tool to fix that problem
Boy those have gotten cheap!
http://www.newegg.com/Product/Product.aspx?Item=N82E16822236600
But they will be terrible slow. Those are 5400 RPM SATA drives.
This is why I made my comment about not using the "RED" drives earlier, they don't have the PRO in 2.5" form factor; however, the savvios are twice the speed at 4x the price.
@scottalanmiller said:
@donaldlandru said:
Which I can add for as cheap as $5k with RED drives or $10k with Seagate SAS drives.
WD makes RE and Red drives. Don't call them RED, it is hard to tell if you are meaning to say RE or Red. The Red Pro and SE drives fall between the Red and the RE drives in the lineup. Red and RE drives are not related. RE comes in SAS, Red is SATA only.
It's all in a name. When I say REDs I am referring to WD Red 1TB NAS Hard Drive 2.5" WD10JFCX. When I say seagate I am referring to Seagate Savvio 10K.5 900 GB 10000 RPM SAS 6-Gb/S ST9900805SS
Edit: I don't always use WD NAS (RED) drives, but when I do I use the WDIDLE tool to fix that problem
@scottalanmiller said:
@donaldlandru said:
Ahh -- there is the detail I missed. Just re-read my post and that doesn't make this clear. Yes, the discussion was supposed to pertain to the non-production side. My apologies.
LOL, a rather sizeable detail
I think we've been focused almost entirely on the operations cluster in our discussion and/or putting the two together to assess needs as a whole - which is worth considering, is there actually a good reason that they are independent to this level?
LOL -- it's all in the details is there a :sheepish: emoji??? Nope.
As to them being separate this why a design consideration outside of my control being hired in mid process. I believe the thought was to have a separate pane of glass. I would much rather have a three node cluster in this case holding both roles but what I have is what I have.
If I bring up the operations nodes only have 1CPU each and only 64GB of memory I just cringe and this goes a third direction.
@scottalanmiller said:
The cost of external storage for the compute nodes is a huge percentage of the cost of just replacing the whole thing, right? If you could spend $14K on an MSA for them, you should be able to spend around $16K, I'm guessing, to get a single node with more CPU and more RAM than you have between the two nodes currently while getting a storage system that is bigger and likely orders of magnitude faster.
HP DL360p Gen 8 with 2 Intel E5-2640 and 384GB ram cost us roughly $13k each -- this is without local drives. On our current large compute node I am only 20% utilized on CPU and 50% utilized on RAM (at peak). I am however, out of storage. Which I can add for as cheap as $5k with RED drives or $10k with Seagate SAS drives.
The $13k does not include VMWare licensing, which is obviously much debated if I even need it; however, send I am decommissioning 4 CPUs when we upgrade I still have available licenses.
@scottalanmiller said:
@donaldlandru said:
Back to the original requirements list. HA and FT are not listed as needed for the development environment. This conversation went sideways when we started digging into the operations side (where there should be HA) and I have a weak point, the storage.
Okay, so we are looking exclusively at the non-production side?
But production completely lacks HA today, it should be a different thread, but your "actions" say you dont need HA in production even if you feel that you do. Either what you have today isn't good enough and has to be replaced there, or HA isn't needed since you've happily been without it for so long. This can't be overlooked - you are stuck with either falling short of a need or not being clear on the needs for production.
Ahh -- there is the detail I missed. Just re-read my post and that doesn't make this clear. Yes, the discussion was supposed to pertain to the non-production side. My apologies.
I agree we do lack true HA in the production side as there is a single weak link (one storage array), the solution here depends on our move to Office 365 as that would take most of the operations load off of the network and change the requirements completely.
We have qasi-HA with the current solution, but now based on new enlightenment I would agree it is not fully HA.
@donaldlandru said:
Here is what the business cares about the solution: Reliable solution that provides necessary resources for the development environments to operate effectively (read: we do not do performance testing in-house as by the very nature, it is much a your mileage may vary depending on your deployment situation).
In addition to the business requirements, I have added my own requirements that my boss agrees with and blesses.
- Operations and Development must be on separate storage devices
- Storage systems must be built of business class hardware (no RED drives -- although I would allow this in a future Veeam backup storage target)
- Must be expandable to accommodate future growth
Requirements for development storage
- 9+ Tib of usable storage
- Support a minimum of 1100 random iops (what our current system is peaking at)
- disks must be in some kind of array (zfs, raid, mdadm, etc)
Back to the original requirements list. HA and FT are not listed as needed for the development environment. This conversation went sideways when we started digging into the operations side (where there should be HA) and I have a weak point, the storage.
@Dashrender said:
@scottalanmiller said:
@Dashrender said:
Because of the volatility of your dev environment, I wonder if using a SAM-SD for central storage would be best. What happens if the entire storage array is down? Can you live for a day or two without it on the dev environment? What are you planning for backups on it? What is your RTO and RPO?
His proposed ZFS-based storage option is a SAM-SD, just in case anyone missed that.
You're right it is, but for the dev environment it might be all that he needs with a good backup solution. He's currently hamstrung by his old servers - two of which are slated to be replaced in the next year or so.
Perhaps he should do nothing until it's time to replace those boxes.
I can't do nothing, I do not have enough storage to host a new client that starts soon. I have to do something there. I am not opposed to overall architecture changes in a refresh cycle, but in the meantime -- I have a budget and need disk.
@Dashrender said:
Because of the volatility of your dev environment, I wonder if using a SAM-SD for central storage would be best. What happens if the entire storage array is down? Can you live for a day or two without it on the dev environment? What are you planning for backups on it? What is your RTO and RPO?
Your operations systems - I like the two node sync'ed approach, if you even really need that, but you already have the two servers.
That is pretty much where this all started, do I need to fork out the money to HP or is the other way good enough.
In operations the RTO/RPO is 24 hours. We carry our HP care pack on the MSA. Everything is backed up by Veeam several hours throughout the day and replicated offsite. We have physical access to the offsite location in case of datacenter failure for faster recovery.
For the development environments up to six months ago there was no backup of the development environments as the thought was this could be rebuilt from scratch. This was until I outlined the effort it would take to bring everything back. -- roughly 6 months.
Now the RPO is one week with a RTO of 72 hours.
@scottalanmiller said:
@donaldlandru said:
I disagree with this, the controllers fail independently of each other. We have experienced a controller failure in the MSA and while it was degraded performance wise, zero downtime was experienced. HP sent a technician out with replacement controller, hot-swapped and checked configuration 10 minute process again with zero downtime.
No they can fail independently of one another. That is not the same thing. Under certain types of hardware failure they are redundant, under the most common forms of firmware failure, they are not. This makes them work great in demos as you can reliably yank out a controller and it keeps working but search SW and you'll see the MSAs dying with both controllers going out at once with one killing the other as it goes. They are tightly coupled, you aren't in the range of independent controllers here.
They also die far more frequently than standard RAID controllers. A normal RAID controller is expected to have a multi-decade average life. One of the most reliable components in your servers (I've see this over 80,000 servers over a decade of monitoring.) Your failure rate from that one controller dying in your environment puts the failure rate at hundreds of times higher than I've measured in servers. It's purely anecdotal on your end, but something to consider. How many server controllers have died in the same time period even though you have many more servers?
Yes they can and have failed independently of each other, outside of a demo environment (as I just outlined above). Firmware update risks are everywhere, shared and local storage both so one way or the other doesn't mitigate that risk.
Out of the hardware in our datacenter I have had the one MSA controller fail, a P420 in the HP DL360p G8 and a perc in the dell 2950, all inside the four years I have been here. To me this shows no better level of reliability than the other. Both of the controller failures in the blades caused downtime to the organization, the failure in the MSA did not.
The biggest challenge with local storage is getting your extra capacity where you need it. This is something that matters in development, I can't have an overworked compute node with a boatload of extra storage, and an underworked one with no storage. This is what shared storage (at the dev level) solves for us. And keep in mind, those storage location needs can change at anytime.
@scottalanmiller said:
@donaldlandru said:
Real life I am not sure if it works, on paper it does. It is a false sense of security but the MSA does have active/active controllers built in (10GB iSCSI), redundant power supplies, and of course the disks are in a RAID. The risks that are not mitigated by the single chassis are:
Not active/active. It has codependent controllers that fail together. It's the opposite of what people expect when they say "redundant". It's the two straw houses next door in a fire, scenario. Having two houses is redundant, but if they are both made of straw and there is a fire, the redundant house will provide zero protection while very likely making a fire that much more likely to happen or to spread. Active/Active controllers from HP start in the 3PAR line, not the MSAs.
All that other redundant stuff is a red herring. EVERY enterprise server has all of that redundancy but without the cripplingly dangerous dual controllers. Making any normal server MORE reliable than the MSA, not less. If anyone talks to you about the "redundant" parts in an MSA you are getting a sales pitch from someone trying very hard to trick you unless they point out that every server has those things so this is "just another server".
I disagree with this, the controllers fail independently of each other. We have experienced a controller failure in the MSA and while it was degraded performance wise, zero downtime was experienced. HP sent a technician out with replacement controller, hot-swapped and checked configuration 10 minute process again with zero downtime.
Now, if we are worried about the neighbors house on fire, sure we have that issue as everything (operations) is housed in a single data center. We accept the risk that our operations is highly available (might incur downtime during failover) but is not fault-tolerant (services are not running in an active/active format).
Software bugs on the other hand have bit us in the past we makes us very cautious when we do upgrades, scheduled maintenance windows, extra backups on hand, etc.
@dafyre said:
In your Dev environment, you have 3 servers... with 288GB of Ram, 64GB of RAM, and 16 GB of RAM... Assume RAM compatibility... What happens if you balance out those three servers and get them at least close to having the same amount of RAM?
Does that help you at all? If that is a good idea, then why not look at converting them to XenServer and switching to Local Storage? You could then replicate the VMs to each of the three hosts, or you could set up HA-Lizard.
The two smaller servers pre-date my time with the company and were likely back of truck specials. Both of these are slated to be replaced next year with a single server with similar specs to the big server. The smallest one is already maxed out and the other one doesn't make sense to upgrade just to retire.
I also don't need HA on these (I don't have HA today on these) so I think this is an opportunity to move to different platform.
@scottalanmiller said:
@donaldlandru said:
- Add a second SAN that replicates with the first (HP MSA easy to do, not so nice price tag)
I've never seen someone do this successfully. That doesn't suggest that it doesn't work, but are you sure that the MSA series will do SAN mirroring with fault tolerance? I'm not confident that that is a feature (but certainly not confident that it isn't.) Double check that to be sure as I talk to MSA users daily and no one has ever led me to believe that this was even an option.
I know that Dell's MD series cannot do this, only the EQL series.
Real life I am not sure if it works, on paper it does. It is a false sense of security but the MSA does have active/active controllers built in (10GB iSCSI), redundant power supplies, and of course the disks are in a RAID. The risks that are not mitigated by the single chassis are:
All in all I think the operations is pretty well protected, minus the three risks listed above. It is two nodes that can absorb either node failing, it is on redundant 10gig top of rack switches and redundant 1gig switches. Also, backups are done and tested as well with Veeam. Am I missing something here?
Unless I am mistaken, and Scott please correct me if I am, it is the three node development cluster that is in sorry shape.
@Dashrender said:
@scottalanmiller said:
Going to VSAN, Starwind, DRBD, etc. would be an "orders of magnitude leap" that is not warranted. It just can't make sense. What you have today and what you are talking about moving to are insanely "low availability." Crazy low. And no one had any worries or concerns about that, right?
That's just it - the company probably thinks they have that super high level of availability and the fact that they've never had a failure feeds that fire of belief.
This has always been the issue I've had when I try to redesign/spend more money on a new solution. I get the push back - "well, we did it that cheap way before and it worked for 8 years, why do I suddenly now need to use this other way, clearly the old cheap way works."
This -- all day long! It worked for the 7 years before you got here, it will keep working long after. I fight this fight every day
@scottalanmiller said:
@donaldlandru said:
Ok if we split this into two separate topics the only unmitigated failure point in operations in the single SAN. Two options to mitigate the risk are:
Not currently, you had said that your nodes do not have the tools or the overhead to absorb the load from a failed node, correct? That makes the risk of those nodes failing unmitigated as well. You only have enough nodes to handle your capacity not enough to use them for failure mitigation.
In Operations, the two node cluster,I said they do have necessary resources to absorb the other node failing. It is the development "cluster that isn't a cluster" that cannot absorb.
My next biggest concern, like any technology, is how do I get there from here. I have enough budget for a storage node, and we are going to run out of space within the next 60 days. I do not have, and will not receive additional funding this year for new servers. So some form of "in-place" style of upgrade has to occur. Obviously, this is a server down, convert vm bring it back up type of process that has an unknown LoE.
Trying to not paint a picture of a rock and a hard place, but realistically where else am I at right now?
@scottalanmiller said:
It's important to recognize that it is a SPOF. But being a SPOF is not the core issue, believe it or not, just the one that causes the biggest emotional reaction. If you were to buy a super high end active/active EMC or HDS device for this (mainframe class storage, start around $50K for the smallest possible units) the fact that it was a SPOF would be heavily mitigated. The whole mainframe concept is built around making a SPOF that is unlikely to fail.
But your issues are bigger. Here are the big issues that you are left with in both of your scenarios:
- Single point of failure on which everything rests (the thing most likely to fail causes EVERYTHING to fail.)
- No risk mitigation for the other layers in the dependency chain. This isn't a 3-2-1 as traditionally described but actually a (1/1/1-1) meaning ANY server failure results in unmitigated (literally) failure AND any storage failure results in total failure. You have a dramatic increase in failure risk with this design, not just a small or moderate increase like most people see (because most people are confused and heavily mitigate risk at one or two but not all three layers.) So it is very important to realize that this is at least one full order of magnitude more risky than a traditional inverted pyramid of doom.
- The single point of failure that you have is actually a pretty fragile one. Probably more fragile than the servers themselves. So not only is the risk of failure doubled by having two completely places for things to fail, but the single point of failure that impacts everything is the most fragile piece of all.
- This has the highest cost both today AND going into the future.
Ok if we split this into two separate topics the only unmitigated failure point in operations in the single SAN. Two options to mitigate the risk are:
When dealing with the Microsoft licensing to create the redundancy to obtain the reliability the business wants I think we are coming in at around the same price. Going with local storage here would reduce the complexity and if I can convince the organization to go with Office 365 we actually have a lot lower risk here and wouldn't need to create a bunch of highly available services.
The second topic (scope) is the development environments and you are 100% correct, even if we have active/active SAN clusters the failure will always be at the server level. The lack of vmotion in this "cluster" and the lack of available resources to do a failover, make the compute layer the biggest problem. If we lose a compute node those servers are offline until replaced. The business accepts that risk as long as we have a fast way of spinning down VMs and bringing up the VMs the team is working on. This is much easier with shared storage than local, in my opinion.
So I do have multiple problems to solve, with different sets of requirements.
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
By dropping VMware vSphere Essentials you are looking at a roughly $1200 savings right away. Both HyperV and XenServer will do what you need absolutely free.
Did the price of Essentials double? I thought it was $600 for three nodes for Essentials? and something like $5000 for Essentials Plus.
Those are the rough numbers. He has five nodes so that means either buying all licenses twice (so $1200 and $10,000) or being disqualified from Essentials pricing altogether and needing to move to Standard licensing options.
This is very much the case here.
To get this all into a single cluster (and hopefully using something like VSAN) would require us to upgrade to standard or higher, we would be able to use acceleration kits to get us there but is no small investment.
@scottalanmiller said:
Currently you have an inverted pyramid of doom, not the best design as you know.
This is true, in all scenarios we are playing out we are left with this giant SPOF. This is why I really like the alternatives shared solutions, ZFS, openindiana, etc. because it will allow me to build a second storage node and do replication for failover.
The business is also screaming for reliability and 110% uptime, but falls short when it comes time to writing the check for what they want.
Do the dev environments need to be highly available -- IMO no, but the business sees that as it's bread and butter, they are aware that we still have not fulfilled this requirements.