So to define cluster a little better in our environment.

For the ops servers cluster is the likely the proper term. We have two nodes, ensure that each node has available resources to run the entire workload of both servers if needed and use VMware's HA to manage to this.

For the dev servers, it is simply a single pane of glass, which really is all the essentials kit provides you and the access to the backup APIs.

The politics are likely to be harder to play as we just renewed our SnS for both Essentials and Essentials plus in January for three years.

Coupled with this, our offshore datacenter also has a 3 node development "cluster" which pushes us even further from truly having a single pane of glass (three so far if you are keeping count) which is also based on an essentials kit.

Another important piece of information with the local storage is that everything is based on 2.5" disks -- and all but two servers only have two bays each, getting any really kind of local storage without going external direct attached (non-shared) is going to be a challenge.

@scottalanmiller said:

@donaldlandru said:

24 spindle 900Gb (7.2k SAS) in 12 mirrored vdevs

That's RAID 01, you never want that. You want 12 mirrors in a stripe for RAID 10.

Understanding RAID 10 and RAID 01.

This was modeled after the way TrueNAS (commercial version of FreeNAS) quoted us.

Ok.. your feedback is actually showing something I have been afraid of, I have severe tunnel vision is servicing the current solution.
Doing a quick inventory as to why I am trying to do that:

1. We have the investment into this. Like another recent thread here discussed once an SMB gets heavily invested one way it is hard to switch. To be honest, I am not sure how I could convince them too at this point. This actually seems like an opportunity for a great learning experience
2. Training of supporting resources -- I have a counterpart in our off-shore office that is just getting up to speed on how VMware works -- to be this will be even harder to change
3. I have been using Vmware for 4 years at the office and at home, so I am comfortable with it. This reason should also make the list as to why I should change it.

One limiting factor I see right now is our current chassis are 1U with 2-4 drive bays which would hamper a local storage deployment.

Edit -- Stepping back and thinking, the lack of drive bays are not a valid limiting factor as I could easily add SAS and do DAS storage on these nodes.

@scottalanmiller said:

That $1200 number was based off of Essentials. Just saw that you have Essentials Plus. What is that for? Eliminating that will save you many thousands of dollars! This just went from a "little win" to a major one!

Essentials plus is to allow us to use VMotion on operations cluster, where is would likely be cheaper in the long-run to acquire MS Server datacenter licensing and building redundant services, this was the approved solution to move VM's back and forth for node maintenance / upgrades.

The ops layout is
2x AD DC (one hosts DHCP server)
1x SQL server for SharePoint
1x SharePoint foundation
1x Exchange server
1x File Server (hosts a bunch of other services because of no additional server licenses)
handful of other CentOS servers for monitoring, help desk, internal web server

The ops cluster could likely be decommissioned and what little remaining services could be collocated on the dev environments if I could only convince the owners to go with Office 365

@scottalanmiller said:

@donaldlandru said:

We have a 2 node cluster for our operations (Exchange, AD, SharePoint Foundation, and other miscellaneous applications) and a 3 node cluster for development machines.

So a two node cluster and a three node cluster. This seems straightforward.... no external storage at all. The rule of thumb of external storage is that it should not be considered until you are above four nodes in a single cluster and even then, not normally until much larger. What is the purpose of having external storage at all?

This setup was implemented when I first started four years ago, we used a third-party consultant and they designed this at the solution for the operations cluster. There were initial plans to do something different for the development cluster, but due to cost of the SAN (which may or may not have been needed) it was then value-engineered by the people leading the project, and little regard to my input, as I was the new guy.

My initial plan was to build a four-node cluster with shared storage without the ops/dev silos. The ops (2node) cluster is licensed with VMWare Essentials Plus and the dev cluster is licensed with VMware essentials. I do rely on vmotion and drs in the ops cluster for better utilizing resources and doing maintenance.

VMotion is of little use to me in the dev cluster as these machines (RAM: 288GB, 64GB, 16GB) don't have enough resources to host everything should a node drop so it is mainly licensed for the backup API access

@scottalanmiller said:

Before I dive into it, what is the need around ZFS? It sounds like you are leading with the solution, rather than the goal, which will not lead us in the direction of a best answer. We should step back and think at the goal level and determine what it is that we want to accomplish. Maybe ZFS will be the answer, but what it if isn't? Leading with the answer and looking for the question isn't the best way to design a solution.

In a sense I am, only due to outside of the MSA and Windows based storage this is what I am most familiar with. Seeing as if we don't go with a vendor supported solution, this would require the minimal effort to support. Doesn't make it the right answer, just the one I am most comfortable with putting my name next too.

Ok, so a little background. the storage situation at my organization is our weakest link in our network. Currently we have a single HP MSA P2000 with 12 spindles (7200 rpm) serving two separate ESXi clusters. We have a 2 node cluster for our operations (Exchange, AD, SharePoint Foundation, and other miscellaneous applications) and a 3 node cluster for development machines. Development is our core business, in simple terms we do SI work for Oracle Retail applications which includes custom development. Some in the organization argue this data may be even more important than the aforementioned operations systems, thankfully IMO my boss (the CEO) disagrees with that opinion. Also, when presenting this same information (rolled up better to speak CEO), my bosses response was whatever I think is the better solution. The company really does stand behind me in what I suggest, I just don't want to add additional risk.

It is not uncommon for us to max out the disk i/o on 12 spindles sharing the load of almost 150 virtual machines and everyone is on board that something needs to be changed.

Here is what the business cares about the solution: Reliable solution that provides necessary resources for the development environments to operate effectively (read: we do not do performance testing in-house as by the very nature, it is much a your mileage may vary depending on your deployment situation).

In addition to the business requirements, I have added my own requirements that my boss agrees with and blesses.

1. Operations and Development must be on separate storage devices
2. Storage systems must be built of business class hardware (no RED drives -- although I would allow this in a future Veeam backup storage target)
3. Must be expandable to accommodate future growth

Requirements for development storage

• 9+ Tib of usable storage
• Support a minimum of 1100 random iops (what our current system is peaking at)
• disks must be in some kind of array (zfs, raid, mdadm, etc)

Proposed solutions:

#1 a.k.a the safe option
HP StoreVirtual 4530 with 12 TB (7.2k) spindles in RAID6 -- this is our vendor recommendation. This is an HP renew quote with 3 years 5x9 support next-day on-site for ~$15,000

Pros
Can purchase support
Single-vendor -- "one throat to choke"
Integrated solution
Cons
Less performance than solution #2 out of the box
More expensive to upgrade later (additional shelves and drives at HP prices)
All used hardware

#2 ZFS Solution ~$10,000
24 spindle 900Gb (7.2k SAS) in 12 mirrored vdevs
Based on Supermicro SC216E16 chassis
X9SRH-7F Motherboard
Intel E5-1620v2 CPU
64 GB of RAM
No L2ARC or ZIL planned
Dual 10gig NICs

Pros
Better performance out of the box (twice the spindle count)
Non-vendor specific parts means upgrades require less investment

Cons
Self-supported
I am the support contract
Multiple vendors and suppliers to acquire parts
Combination of new and used hardware (the chassis) to get this price point

Alright, tear me apart tell me I am wrong or provide any other useful feedback. The biggest concerns I have exist in both platforms (drives fail, controllers fail, data goes bad, etc) and have to be mitigated either way. That is what we have backups for -- in my opinion the HP gets me the following things:

1. The "ability" to purchase a support contract
2. Next-day on-site of a tech or parts if needed

With the $4000 saved from not buying the HP support contract I can buy a duplicate Supermicro system, and a couple extra hard drives, and have the same level of protection.

Note: this is my first time posting an actual give me feedback topic, I tried to include all information I felt was relevant. If more is needed I can provide.

Good timing on this (I might create a separate topic on this) we are contemplating buying HP StoreVirtual (lefthand) vs rolling out own zfs based solution.

The goal is for large, moderate performing, VMware storage for our development environments. I personally have deployed over 100TiB on FreeNAS but concerned about introducing that into our business.

@JaredBusch said:

@donaldlandru said:

USH is closer to MSP than SGN. Typically is 29 hours of travel time from MSP to SGN (MSP-SFO-NRT-SGN).

There are tons of direct flights form MSP to NRT. Why the heck are you stopping in SFO first?

Limited to flying United. I've also done Msp-sfo(orORD)-hkg-sgn

My last time back I had to do nrt-iad-msp.

Some of our people fly Delta straight to nrt but only because they have status

I know it is extremely early in the process, does anyone have a proposed date(s) for said event?

@scottalanmiller said:

One thing that is a big deal is whether your job is one of productivity or one of availability. If your job is engineering or decision making making you a productivity worker, then going to 20-30 hours a week is found to be ideal for getting the best work out of you.

If, on the other hand, you are an availability worker like a system admin, helpdesk tech or similar then cutting hours reduces your value to the company by making you less available.

So it really depends. When I was at CitiGroup they offset this by having me be always on call as a system admin but only in the office for 20 hours a week. There are ways to make availability workers valuable without making them "work" too much, but you have to be more creative than just reducing hours.

How do I get this????!! I am in the office 40-50 hours and then on-call for the other 150 or so hours

@scottalanmiller said:

@donaldlandru said:

@scottalanmiller said:

@donaldlandru said:

@RojoLoco said:

Tierra del Fuego

Ouch! $1500 plane ticket from Minneapolis

The cost is nothing, look at the flight time!!

I'm getting there by boat out of Buenos Aires for my birthday next year.

I can deal with flight time, I do Minneapolis to Ho Chi Minh city twice a year, those tickets are only $700. Now by boat would really stink!

Isn't that a bit closer? And... twice a year?

USH is closer to MSP than SGN. Typically is 29 hours of travel time from MSP to SGN (MSP-SFO-NRT-SGN).

I go twice a year as 2/3 of our organization is located in HCMC, it is a fun trip every time!

@scottalanmiller said:

@donaldlandru said:

@RojoLoco said:

Tierra del Fuego

Ouch! $1500 plane ticket from Minneapolis

The cost is nothing, look at the flight time!!

I'm getting there by boat out of Buenos Aires for my birthday next year.

I can deal with flight time, I do Minneapolis to Ho Chi Minh city twice a year, those tickets are only $700. Now by boat would really stink!

@RojoLoco said:

Tierra del Fuego

Ouch! $1500 plane ticket from Minneapolis

I work 60+ hours a week every week. I am currently in a "I am all you get" role for anything that runs on electricity. Both myself and the company I work for acknowledge this is a risk and we are trying to fix it; however, it is a slow process.

Note: I do NOT wear this as a badge of honor.

@Dashrender said:

@donaldlandru said:

Couldn't live without my Outlook. It has everything I need bundled together in one spot. Love the OneNote integration for meetings.

On my mobile I use Touchdown for Android -- to me this is the best featured mobile app; although it does lack the integrations of the Outlook client, I do mostly only use my phone for send/receive.

Have you tried the new Outlook client for Android?

I tried it when it first launched and wasn't all that impressed. Mainly the way attachments and folders work didn't fit me well. Need to be able to attach items from my device, not cloud storage links.

Couldn't live without my Outlook. It has everything I need bundled together in one spot. Love the OneNote integration for meetings.

On my mobile I use Touchdown for Android -- to me this is the best featured mobile app; although it does lack the integrations of the Outlook client, I do mostly only use my phone for send/receive.

@scottalanmiller said:

@Dashrender said:

@scottalanmiller said:

@Dashrender said:

@scottalanmiller said:

@dafyre said:

Also, where would be a good place to go to for doing your own Index funds... a traditional IRA account or something more along the lines of TDAmeritrade & ilk?

Almost anywhere but I highly recommend Vanguard. Both famously the best Index Fund supplier on the market and just an awesome investment firm to work with.

wanna give a shout out to a broker there?

I don't use a broker, no need for one since I use Index Funds. Investing gets very difficult to make money if you need a broker as the broker fees are what cripple you. Index Funds are all computerized keeping the overhead way down allowing the investments to make money more easily.

So you just visit the website and open an account, never talking to a person? OK I guess that works.

Yup, pretty easy

This could be a session at MangoCon!

If this is the secret club you are talking about in the conference dichotomy thread count me out

And the reply all hell has started on the Horizon DataSys email thread