@scottalanmiller said in Domain Controller Down (VM):

@Dashrender said in Domain Controller Down (VM):

If HA is fully thought out and is felt is needed (don't forget about the power situation, and cooling, etc, etc, etc, - remember HA isn't a product, it's a process) then they should fully realize it. I'm guessing by the fact that the switches were 100 Mb that it really wasn't fully thought out, instead someone in the place of authority thought it sounded good and they tossed what they have in today in.

It's as simple as "there was no HA and no attempt made at it."

It would take me about 5 minutes to explain to a 3rd grader why the system he has isn't redundant is bad. The fact that it continues to exist shows that either...

1. Management has the intellectual capacity below a 3rd grader (possible)
2. No one in non-jargon english explained how bad this configuration was. (more likely).

@Dashrender said in Domain Controller Down (VM):

As for the rest, I generally agree with you. It shows the real costs of DOING IT RIGHT - but as most of us know - few SMBs are really willing to do what's right in IT.
Hell, just look at all of the threads in SW talking about print shops that couldn't upgrade their XP machines because their 10K+ printers didn't support anything newer. it's a never ending problem of knowing the real costs of doing something right.

The real cost of doing IT right is cheaper. Simply not having an onsite FTE, and having a MSP manage this stuff is likely cheaper (FTE's are expensive!). This outage might have been embarrassing enough for them to loose a patient or two (or worse someone die, and they get hit with a million wrongful death dollar lawsuit that spikes their premiums). Doing IT RIGHT includes understanding the capex and opex costs, and associated risks and external costs of doing IT right or wrong.

Doing IT Wrong means wasting tons of money and getting an output that causes other costs. IT budgets do NOT exist in a vacuum to the rest of the operations and their output (Especially in 2016!).

@scottalanmiller said in VMWare OSx:

It's really that startups like to flaunt their ability to waste money. It's why I'd never invest in a company going into the Valley. The IT talent pool is pathetic and IT controls don't exist. Companies starting there fail like nowhere else. You definitely get the best developer talent, but at a cost that makes it just silly. Give me Spain or Romania or Ukraine or Russia any day.

Agree I wouldn't start there now today. Our primary campus is there, but IT operations, GSS, actual datacenter are all elsewhere.

@IRJ said in VMWare OSx:

@JaredBusch said in VMWare OSx:

@IRJ said in VMWare OSx:

I am curious why you would want to run OSX in a VM anyway?

What advantages does OSX give you over Windows or Linux Distros other than the obvious video or music editing which probably isn't ideal in a VM anyway?

He said a couple posts up that he hopes it was good for learning. So maybe self education.

In the IT field, that would be the last OS I would be interested in learning. Especially if your company doesn't have any so you have to download a poorly made image.

Here is how you install a OS X VM.


I like how VMunderground does the opening acts pannels.

https://blog.vmunderground.com/tag/opening-acts/

Disclaimer, I work for VMware SABU, who make software for HCI

There is no reason that a new hyperconverged cluster cannot run side by side with the pre-existing infrastructure.

The Challenge comes that a lot of HCI vendors/platforms don't "play nice" with existing storage assets. By this I mean they can't mount external storage arrays, re-use existing assets that still have support-depreciation. This lock out can be support/technical (I don't think Scale Computing supports you using external storage with their product), This can be hardware limited (You can't get FC cards for a Simplivity/Nutanix box) or it can be administrative. An example of this EVERY HCI Product other than UCP-HC and VxRAIL that has a management system that outright is hostile/doesn't support extending it's ease of management to external products. Beyond the example of SPBM allowing management of HCI storage (as well as extending it's support to external arrays with VVOLs) it should be noted that HCI systems tend to make their own storage easier (While actively trying to make it harder to add external assets). Why does this matter? Expansion costs and support fee's. HCI vendors are increasingly adopting the legacy storage vendor model of discounting hugely up front, then charging significantly higher support renewals (and removing discounts once you get past your starter pack). Signs of this are when you go to add node 4 or 5 discovering that it costs as much as nodes 1-2 and 3. Another sign is a support renewal of 6-12K for an otherwise ordinary server. While I'm not opposed to an increasing shift to opex for IT (It's necessary especially to keep companies from running gear into the ground foolishly), doing so in way that you don't realize until year 2 or 4 support renewals is something everyone should watch out for and be aware of. Price != Costs, and while HCI does reduce a LOT of hidden opex costs, you need to be aware of the real Total cost of ownership of what your buying. A great read on this topic is HDS's "34 costs of storage" white paper.

@scottalanmiller said in Why is VMWare considered so often:

Ability to restore individual files is huge.

Ability to Recover Individual Emails, GPO's, Schema's on databases is a good way to not have to waste 1/2 a day on a recovery of something small.

Ability to test backups.
Ability to orchestrate DR (IE handle IP changes etc). Can chain DR copies from backups.
Use of proper CBT API's for Hyper-V and VMware meaning 95% faster backup windows vs. agent based backups on XS.
WAN efficient replication (Built in WAN accelerator, compression etc).
Dedupe and compression for backups for space efficiency. Tape, and cloud repository support.
SureBackup lets me automate my backup tests so I actually know the recovery will work

@coliver said:

@Breffni-Potter said in Why is VMWare considered so often:

Well this is new.

http://blogs.vmware.com/vsphere/2016/05/goodbye-vsphere-client-for-windows-c-hello-html5.html

They've been talking about this for awhile now. We ran into it in the past year where they told us they wouldn't be updating the desktop client to work with TLS1.1 or 1.2.

You've got a ton of options for management of hosts.

HTML5 native, SSH, PowerCLI (PowerShell), RESTFUL API's with SDK's for Python, C, Java and a bunch of other languages, SOAP API. If you want a thick client just use Fusion/WorkStation (What I use at my house for quick VM/console access). There's other interfaces you can use too (vRA, VCD have their own).

@olivier said in Why is VMWare considered so often:

@John-Nicholson said in Why is VMWare considered so often:

Years ago, I had a DRDB cluster go split brain on me

This, totally agree. Before adopting a product in a real prod env, you have 2 choices:

1. pay for a turnkey solution (packaged with support)
2. install it by yourself ONLY after having enough knowledge on how it works.

As you said earlier, you can't master something in 4h. You need to practice, validate, crash it, restore it etc.

See we can all get along. One thing that Xen/XenServer has to go up against is the massive amount of training and operational experience that is in the field. It reminds me of the linux desktop. Linux makes a decent desktop, but without 3rd party vendors seeing value (and porting apps) and existing staff seeing value in re-training) it's hard to get larger adoption. You'll have to not only reach feature parity (and cost) but actually have "killer app" incentives to switch (of which drives VMware R&D to adapt). This is the arms race of our industry (and honestly we all benefit from this, vendors, customers, and end users...).

@olivier said in Why is VMWare considered so often:

Anyway, my point was:

• Free software is great and powerful, you just trade this against time to understand how it works (or cross your fingers, but it's not acceptable in production). Note that you could mitigate the risk in different ways but you should understand most of your infrastructure.

• Support/Service on proprietary software can be useful if you have money and you don't care about what's happening here (ie not your core business)

• Support/Service on Open Source software is a kind of best of both worlds.

But that's my opinion

One of the benefit of open source is that there are multiple parties contributing to it that you are not having to pay. (The challenge is their needs may not align with yours, although this happens with commercial software also). The SMB is perpetually in an awkward drafting of larger enterprises hoping for lower cost solutions that solve their problems to fall off the truck. Sometimes this works, sometimes it doesn't. When I worked for a SMB I'd always grown when I saw new versions come out that supported 10K VM's instead of 5000VMs and so forth...

The VMware VCSA uses Postgres, Photon Linux (you can find on GitHub). The major proprietary "Secret sauce" is the 150MB worth of proprietary VIB's on a host and the ESXi kernel itself. If you include GPL drivers, BusyBox, Linux, Tomcat/Apache, I think your typical vSphere deployment actually has more "Free" code on a per MB basis than not...

@olivier said in Why is VMWare considered so often:

@John-Nicholson I'm not here to attack the product at all (I don't even know what half of the acronyms meant). I'm not building an hypervisor.

I'm just here to try to survive with the crumbs left from server virt market, without leaving my philosophy (making Free software).

There's a lot of crumbs (maybe even spare slices of bread) in the hypervisor market (especially Xen) for easier to use management stacks and tool chains. Look at what Scale Computing is doing in going after the SMB market. There's a lot of people with 2-12VM's 1-2 IT guys, and they are not going to the cloud.

@IRJ said in Why IT Builds a House of Cards:

The moral of the story here is you get what you pay for. Every once in awhile you can get a rockstar IT person at $40K a year, but it doesn't really happen very often. Sure, many IT people go start out a $20k and work their way up ( I did), but they are going to view that $40K a year salary as a stepping stone with limited growth. Give them limited support (money and verbal) and the chances of having a dank network get even lower.

$10 an hour to hire IT?!? You can make more waiting tables, or at a gas station. 40K isn't much better (Bartenders make more than this).

@DustinB3403 Personally I prefer robocopy. Hell of a lot faster, and you can make sure it will copy exactly how you want....

Robocopy <Source> <Destination> /copyall /xo /w:0 /r:0 /log:Logfile.txt

Tail the log file (Wintail is handy for this) as having it log to the console slows down copy's.

@Dashrender said:

@scottalanmiller said in Cost Study: 3 Node Scale vs. 3 Node VMware VSAN:

VMware VSAN Support Cost: $25,440 ($1060 per CPU for each year after the first)

Wow - why even sell VSAN at that point - why don't they just do subscription and get over with it - wow that seems expensive. Is this in line with support contracts for other SAN products? I know it's hard to judge that because this is based on CPU (luckily not cores) where I'm assuming typical SAN support is more based upon capacity.

Because that's not what vSAN costs for a 3 node cluster. The capital cost is 15K List for a 3 node cluster. I'm guessing he's bundling the first 3 years of support in or something and putting zero discounting on the cost.

This cost study is also using 4TB SATA drives which vSAN doesn't certify. Also the only 1.9TB drive I"m familiar with that Dell sells (this could have changed) is a PM863 that gets awful write latency consistency and is only certified for capacity usage not write cache. Beyond that you would be better served by 2 smaller write intensive SSD's. This cost study ignores the HCL, the design and sizing guide.

@scottalanmiller said in Vendor Mistake - VMware Infrastructure Decisions:

@NetworkNerd said in Vendor Mistake - VMware Infrastructure Decisions:

@scottalanmiller said in Vendor Mistake - VMware Infrastructure Decisions:

@John-Nicholson said in Vendor Mistake - VMware Infrastructure Decisions:

@scottalanmiller If your doing a 3 node vSAN for a low cost deployment you should go single socket and get more core's per proc. Leaves you room to scale later and costs the vSAN cost in half.

They are likely stuck here with whatever was already bought. But good info for a greenfield deployment. Or if they manage to return these for three R730 for example.

I'm not entirely certain we'll be stuck with what we bought. My boss and I were on a conference call with folks from Dell yesterday afternoon. They were talking about different options in SAN devices that would meet our requirements (whether it was Compellent, EMC, etc.), but the biggest issue was that these options were so expensive. Again, not one of them mentioned the potential for a VSAN deployment, so we brought it up (using either VMware VSAN or Starwind). The Dell team has to go back and redesign a quote for gear that would better support a VSAN deployment. In their words, they would likely have to return the servers and the PowerVault we have right now (not sure about the other gear - PowerConnect switches, TrippLite devices, APC PDUs, AppAssure appliance, and ip KVM switch).

I'll be curious to see what comes back when they re-quote.

Why do they have to design a quote? You just tell them what you want, they give you a price. Other than "looking up the price", what are they doing?

Verifying the HCL (for vSphere, and vSAN for the storage devices). If they are 13Gen servers though they should be adaptable, it's just a batter of getting a supported HBA (Hint, you want the HBA 330) and getting supported drives. Other thing I'll comment in general (Not related to Dell or vSAN) is avoid Intel NIC's and go Broadcom. LSO/TSO seems to not be stable on large frames (This can be mitigated by disabling offload at the cost of a few % of CPU if you need). After years of hating broadcom NIC's this feels weird and Intel SHOULD be fixing it at some point this quarter, but after 2 years of putting up with this on large frames I'm not that hopeful.

@KOOLER said in Vendor Mistake - VMware Infrastructure Decisions:

@NetworkNerd said in Vendor Mistake - VMware Infrastructure Decisions:

Before I started here a couple of months ago, my boss purchased a couple of Dell R630s and a PowerVault MD3820i (20 drive bays) to be our new infrastructure at HQ. We have dual 10Gb PowerConnect switches and two UPS devices, each connected to a different circuit. The plan is to rebuild the infrastructure on vSphere Standard (licenses already purchased) and have a similar setup in a datacenter somewhere (replicate the SANs, etc.). We're using AppAssure for backups (again, already purchased).

The PowerVault has 16 SAS drives that are 1.8 TB 7200 RPM SED drives and 4 SAS drives that are 400 GB SSD for caching. Well, we made disk groups and virtual disks using the SEDs (letting the SAN manage the keys), but it turns out we cannot use the SSDs they sent us for caching. In fact, they don't have SED SSDs for this model SAN.

At the time the sale was made, Dell ensured my boss everything would work as he requested (being able to use the SSDs for caching with the 7200 RPM SED drives). Now that we know this isn't going to be the case, we have some options.

First, they recommended we trade in the PowerVault for a Compellent and Equalogic. The boss did not want that because he was saying you are forced to do RAID 6 on those devices and cannot go with RAID 10 in your disk groups. As another option, Dell recommended we put the SSDs in our two hosts and use Infinio so we can do caching with the drives we have. In this case we would make Dell pay for the Infinio licenses and possibly more RAM since they made the mistake.

But I'm wondering if perhaps there is another option. Each server has 6 drive bays. So we have 20 drives total. Couldn't we have Dell take the SAN back, give us another R630, and pay for licenses of VMware vSAN for all 3 hosts? Each server has four 10 Gb NICs and two 1 Gb NICs. That might require we get additional NICs. But in this case, I'm not sure drive encryption is an option or if we can utilize the SEDs at all.

I've not double-checked the vSAN HCL or anything for the gear in our servers as this is just me spit balling. Is there some other option we have not considered? We're looking to get the 14 TB or so of usable space that RAID 10 will provide, but the self-encrypting drives were deemed a necessity by the boss. And without some type of caching, we will not hit our IOPs requirements.

Any advice is much appreciated.

Keep R630s, refund PowerVault, refund AppAss. Get VMware VSAN and Veeam (accordingly).

I've got (a non-trivial amount) of R630's in my lab running vSAN. You'll want the HBA 330 ideally (you can settle for the PERC H730 if you already have it) but otherwise the server works fine. Only limit over the R730/R730XD is fewer drive bays, and no GPU support.

@scottalanmiller said in Vendor Mistake - VMware Infrastructure Decisions:

@NetworkNerd said in Vendor Mistake - VMware Infrastructure Decisions:

@scottalanmiller said in Vendor Mistake - VMware Infrastructure Decisions:

@NetworkNerd said in Vendor Mistake - VMware Infrastructure Decisions:

@scottalanmiller said in Vendor Mistake - VMware Infrastructure Decisions:

@NetworkNerd said in Vendor Mistake - VMware Infrastructure Decisions:

I'm also assuming you are turning RAID off on each host so Starwind can provide RAIN for you (thus creating the storage pool).

No, you leave RAID on on the hosts and Starwind provides Network RAID. There is no RAIN here.

So you'd leave RAID on and then make a small local VMFS datastore for the Starwind VM to run on so that Starwind can use the rest of the unformatted storage on the host for its network RAID?

You just follow the Starwind install guide. But yes, that is what is going on.

After reading each of these, I finally understand how it works:
http://www.vladan.fr/starwind-virtual-san-product-review/
http://www.vladan.fr/starwind-virtual-san-deployment-methods-in-vmware-vsphere-environment/
https://www.starwindsoftware.com/technical_papers/HA-Storage-for-a-vSphere.pdf

So, in a nutshell, you do use RAID on the host as you normally would and even provision VMware datastores as you normally would. It's the VMDKs you present to the Starwind VM that get used as your virtual iSCSI target. And you can add in the cache size of your choice from the SSD datastores on your ESXi host.

So if I'm patching servers like I should, I'd have to patch the VMs running Starwind as well. Oh man would I hate to install a patch from MS that bombs my storage. I guess theoretically that isn't too different from installing some firmware on a physical SAN that has certain bugs in it. If one Starwind VM gets rebooted, you still have your replication partner presenting storage to the hosts and are ok.

Right. And Hyper-V alone has very tiny, solid patches. Nothing like patching the OS.

Hyper-V with a console is just as big as windows server from a patching perspective, and even Core Install's see patches with regular (IE monthly quite often) frequency. The install requirements for The ~150MB VMKernel are tiny vs the 10GB+ for Hyper-V Core installs. ESXi regularly goes ~6 months without needing a patch. Most of the patch surface is in upper stack things.

@stacksofplates said in Virtual Machines vs Containers:

I know KVM can do dynamic resource allocation. You have to set a max number beforehand, but you can change RAM and CPU on the fly as long as its the same or under your max.
Not sure about other hypervisors.

ESXi supports this (HotAdd is the term used). Requires BaseOS support it.

@BBigford

Couple things...

1. I've load balanced Horizon View with a Netscaler (It works). Even the free version works.

2. Netscaler isn't a "native" per say as something that Citrix Acquired and likes to bundle sell. What was native was running CSG's with RR DNS (Note you can use RR DNS for View Security and connection servers also, never been the biggest fan but it has always worked.

3. VMware has NSX that can do basic L4 LB's for View and is integrated from a management and hypervisor standpoint for some of it's services. Note most people going down the NSX path will do it for advanced stuff like micro segmentation and network introspection offload (I've seen it also used for Citrix for these reasons).

4. F5 has a nice "license per user option" for Horizon View and they can completely replace the edge security server function if you go down this path..

5. KEMP's are stupid cheap to use for LB services when you just need basic stuff. I think that most of the Netscalers functionality (layer 7 stuff, host specific balancing) is mitigated by Horizon including DRS and moving VM's no the back end to balance load (which mostly happens AFTER initial connection anyways). The reason why Citrix historically needed this stuff is you proxied connections to bare metal and you couldn't reshuffle heavy users after connection.

I've always felt GSLB's are "dumb" and would rather just pay a 3rd party DNS provider to manage failover for me. Very few people need their functionality.

Example NSX with View Config.
https://elgwhoppo.com/2015/09/25/load-balancing-horizon-view-with-nsx/

The key to understanding View is understanding what all talks to what

http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/vmware-horizon-7-end-user-computing-network-ports-diagram.pdf