Posts made by StorageNinja

@scottalanmiller You say free storage migrations?

http://www.vladan.fr/free-tool-extrasphere/

@donaldlandru said in ZFS Based Storage for Medium VMWare Workload:

My next biggest concern, like any technology, is how do I get there from here. I have enough budget for a storage node, and we are going to run out of space within the next 60 days. I do not have, and will not receive additional funding this year for new servers. So some form of "in-place" style of upgrade has to occur. Obviously, this is a server down, convert vm bring it back up type of process that has an unknown LoE.

So when someone says "You will not get more funding" you need to reply "You will not get more IT for no spend". Apply quota's to the file servers, implement aggressive archiving cut back vCPU and memory allocations so your cluster can fail over, quota mailboxes on Exchange and in general "enforce" their "no new budget policy". Turn on FSRM reporting and post reports that show who's using the most space (Dump similar reports from Exchange). Don't let things hit a wall and crash, start putting the breaks on growth. IT is not a magyver episode where your expected to conjure IOPS and capacity and RAM from thin air.

Trying to not paint a picture of a rock and a hard place, but realistically where else am I at right now?

Your not between a rock and a hard place they are. Follow the rules above, and they will either find you capital to invest in so they can continue to use more storage and compute, or they will agree that its not worth the spend. THIS IS NOT YOUR PROBLEM. Your problem is to quantify what it will cost to deliver x and deploy if funded. It is not your job to be given an arbitrary budget and produce y. I remember when I realized this, and my job became a lot more Zen like. "Do what you say you will do" was my old office motto. Because of that people said no a hell of a lot.

@scottalanmiller said in ZFS Based Storage for Medium VMWare Workload:

WD makes RE and Red drives. Don't call them RED, it is hard to tell if you are meaning to say RE or Red. The Red Pro and SE drives fall between the Red and the RE drives in the lineup. Red and RE drives are not related. RE comes in SAS, Red is SATA only.

They just re-branded the consumer side (IronWolf, brought back Barracuda with a FireCuda cache drive). I was looking at them and realized they have like a .05 DWPD rating which actually makes their write endurance worse than the cheapest of Enterprise TLC drives.

@donaldlandru Cuts licensing for VSAN in half (single CPU)

@scottalanmiller

VSAN is more than high availability.

1. Dedupe and Compression, combined with distributed erasure code's makes all flash cheaper than hybrid in most cases.

2. Distributed RAID/RAIN So you don't have to mirror mirrors (doubles capacity efficiency vs. P4xxx type design)

3. On the fly ability to change availability. Have a test/dev workload you don't care about? FTT=0 and that sucker is RAID 0. Its moving to Production? change to FTT=1 and without disruption it will now be protected with either RAID 1, or distributed RAID 5 (All Flash). Its become really damn important, Change to FTT=2 and get triple mirror or RAID 6. Can even adjust stripe width etc on the fly. Moving ephemeral workloads to FTT=0 saves a lot of space.

4. Non-disruptive expansion (No RAID expansion limits, just add drives or hosts as you need).

5. Self healing. When you have 4 nodes, and 1 node fails, it will use free capacity on the other hosts to re-protect mirrored data.

It may be overkill for what he's doing, but the product has some opex beenfits (no managing LUNs, you get 24/7 support with it etc).

Also I saw a comment about 2 hosts. You can technically deploy VSAN in a 2 node configuration (It just needs a witness VM to run elsewhere).

One thing I will say is for a shop with 100+ VM's and a dev heavy enviroment your bound to be hemorrhaging money from wasted labor on people waiting on things because of how IOPS starved this enviroment is. To Show the value of "Fast" storage Grab A good flash drive, put it in one of the hosts (behind a proper smart array or HBA, not some b model garbage) and put a few VM's on it and see if people notice the difference. They'll be fighting each other to get you budget for more flash...

@scottalanmiller said in ZFS Based Storage for Medium VMWare Workload:

That $1200 number was based off of Essentials. Just saw that you have Essentials Plus. What is that for? Eliminating that will save you many thousands of dollars! This just went from a "little win" to a major one!

Essentials Plus SnS includes 24/7 flat rate support. Getting that On other platforms is a hell of a lot more expensive. Its ~$1200 a year (Essentials is only like $600 and it's software updates only renewal is like $100)

@Dashrender said in ZFS Based Storage for Medium VMWare Workload:

@dafyre said:

@donaldlandru said:

The politics are likely to be harder to play as we just renewed our SnS for both Essentials and Essentials plus in January for three years.
<snip>
Another important piece of information with the local storage is that everything is based on 2.5" disks -- and all but two servers only have two bays each, getting any really kind of local storage without going external direct attached (non-shared) is going to be a challenge.

He brings a good point about the 2 bays and 2.5" drives... Do they even make 4 / 6 TB drives in 2.5" form yet?

If not, would it be worth getting an external DAS shelf for each of the servers?

It's been 15 years, but I've seen DAS shelves that can be split between two hosts. Assuming those are still made, and there is enough needed disk slots, that would save a small amount.

They make flash drives in that size. Actual large capacity small format magnetic drives? No.
The bigger issue is with this level of VM density you run out of IOPS before you run out of capacity.

Shared DAS shelfs though don't handle RAID they are designed for some type of controller to manage them and disk locking etc.

BTW, you can limit IOPS on a VM in vSphere if you just have a noisy neighbor problem.
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1038241

@Dashrender IF you put enterprise grade in a REAL datacenter its scary how long it will run without failure. Now if this stuff is going in some Nolo Telco Dataslum, or his closet yah, stuff dies all the time.

@donaldlandru said in ZFS Based Storage for Medium VMWare Workload:

Ok, so a little background. the storage situation at my organization is our weakest link in our network. Currently we have a single HP MSA P2000 with 12 spindles (7200 rpm) serving two separate ESXi clusters.

This is not a lot of IOPS I have easily 100x more IOPS in the laptop I'm typing this on, than this disk configuration.

It is not uncommon for us to max out the disk i/o on 12 spindles sharing the load of almost 150 virtual machines and everyone is on board that something needs to be changed.

Yep, Go all Flash.

Here is what the business cares about the solution: Reliable solution that provides necessary resources for the development environments to operate effectively (read: we do not do performance testing in-house as by the very nature, it is much a your mileage may vary depending on your deployment situation).

If your a VMware shop doing QA testing, there's some workflows you can do with Linked Clones, and Instant Clones (No IO overhead, 400ms to clone a VM with zero memory or disk as it runs ajournal log for both) to reduce disk load, speed up process's and in general make everyone's life easier.

In addition to the business requirements, I have added my own requirements that my boss agrees with and blesses.

1. Operations and Development must be on separate storage devices

This isn't necessary at 150 VM's. Just get something with all flash and if it is that big of a deal that people are running IO burner something that has QoS as an option to smack them down.

2. Storage systems must be built of business class hardware (no RED drives -- although I would allow this in a future Veeam backup storage target)

Think strongly about using reds with Veeam. Reverse Incremental or roll ups use random IO and your backup windows will hate you.

Requirements for development storage

• 9+ Tib of usable storage
• Support a minimum of 1100 random iops (what our current system is peaking at)
• disks must be in some kind of array (zfs, raid, mdadm, etc)

Proposed solutions:

#1 a.k.a the safe option
HP StoreVirtual 4530 with 12 TB (7.2k) spindles in RAID6 -- this is our vendor recommendation. This is an HP renew quote with 3 years 5x9 support next-day on-site for ~$15,000

Wait is this a single node storevirtual? ALso the IOPS on this will be awful. (7.2K drives are slow).

Less performance than solution #2 out of the box
More expensive to upgrade later (additional shelves and drives at HP prices)
All used hardware
Its worse than that, as you have to not just buy HP parts but licensing.

#2 ZFS Solution ~$10,000
24 spindle 900Gb (7.2k SAS) in 12 mirrored vdevs

To my knowledge no one makes 900GB 7.2K SAS drives.

Based on Supermicro SC216E16 chassis
X9SRH-7F Motherboard
Intel E5-1620v2 CPU
64 GB of RAM
No L2ARC or ZIL planned

Then why the hell would you use ZFS?

Dual 10gig NICs

Pros
Better performance out of the box (twice the spindle count)
Non-vendor specific parts means upgrades require less investment

Cons
Alright, tear me apart tell me I am wrong or provide any other useful feedback. The biggest concerns I have exist in both platforms (drives fail, controllers fail, data goes bad, etc) and have to be mitigated either way. That is what we have backups for -- in my opinion the HP gets me the following things:

1. The "ability" to purchase a support contract
2. Next-day on-site of a tech or parts if needed

Be careful with NBD Parts, as a failure on Thursday afternoon really means Monday afternoon is within SLA as its based on when it was isolated.

With the $4000 saved from not buying the HP support contract I can buy a duplicate Supermicro system, and a couple extra hard drives, and have the same level of protection.

Note: this is my first time posting an actual give me feedback topic, I tried to include all information I felt was relevant. If more is needed I can provide.

Your a VMware shop, curious if you looked at Using VSAN? You could go all flash, and get inline Dedupe and Compression which makes all flash cheaper than 10K drives at that point.

@StrongBad

At my last job we had some sales guys who used it, but they really used it 100% of the time so they didn't end up with some documents on one laptop, some on the VDI, some on another. I used VDI, but primarily as a jump host to pop from our data center, or if I was traveling with just my iPad, That's another thing, is you REALLY want client designed for iPads with custom gesture support etc. Citrix actually has a bluetooth mouse option even. They wanted to be able to use different devices to access their data and we were using some applications that did not have native mobile, or web apps.

Now the office would loose power, cooling, water etc, and the most effective thing for DR was we all had laptops and would just wander off and find internet somewhere else.

At my new job all of our applications are web based and or have mobile apps. We use a SSO broker system to access those applications (do a 2FA once, and then I can access anything in a single click. I can get into CRM, 365, to payroll or 401K as well as internal web applications. We have VDI but its really only something I use to get behind the firewall and access my labs, and as a company I would argue as an industry we are "moving past" VDI. VDI is not the ultimate form of elegant end user computing management, its a stop towards MAM,MDM, Identify Brokering, etc.

That makes more sense. (I remember being maxed at 3 hours a day on a school day or something).

@Minion-Queen This isn't legal is it?

Under the Fair Labor Standards Act (FLSA) there are quite a few restrictions on youth labor.

So outside of a few niche use case (family owned business or farms) 13yr olds can't work for a company. Last time I Checked, unpaid internships were only available for education credit type systems (Typically college under DoE regulations if I'm not mistaken) Adding in restrictions (18 hours a school week) the 20 hours a week wouldn't be supportable into the school year.

@StrongBad Then what? VDI is something I wouldn't run without robust support. When EVERY desktop in your environment crashes and burns you kinda need 24/7/365 support

@John-Nicholson virtual bridges (the guys behind it) got bought by nimbox that went out of business.

@scottalanmiller IBM tried to push it and ultimately gave up.

@dafyre VNC kinda sucked IMHO over the wan.

The other thing is VDI environments are complicated beasts and rich monitoring tools (everything from protocal level network into, to troubleshooting infrastructure issues) are something you need at the scale where VDI normally makes sense.

@dafyre there's a lot of stuff that doesn't work cleanly over folder redirection (OSTs being one). Stuff like persona also kept login storms under control by doing just in time profile loading and stubbing everything else. Stuff like immedIo/UEM can even allow templates central control of app settings I'm profiles. At large scale changing things not in GPO (A niche setting in say Microsoft Lync or a custom app, or tying certain apps to a printer, or fixing default window sizes) this stuff is handy.

2012R2 moved towards VHds for profiles which improved some things (performance) but lost others (can't put them on a DFS share)

@scottalanmiller I though spice was weak over the WAN. It was rich but was bandwidth hungry.

The problem with VDI is there is always "1 more feature" that you'll run into that the cheaper can't do that makes the use case fall apart (profile virutalization, print redirection, WAN optimized protocals, 3D graphics and video card virtuailzation, session broker, FIPS support for smart cards) which is why most people end up running one of the major suites. (Citrix or View).

Custom forwarders have advantages (super fast source filtering, compression and TLS support, custom meta tags, lower CPU) was my Experiance with LI.

Note, outside of maybe sumologic everyone with custom agents allows you to use legacy syslog.