@dustinb3403 said in The argument for official support vs third party support:

I think it's a critically important conversation to have. Support is support, it's primary benefit shouldn't be to fix bugs in the software you paid for.
It is there to make sure that if your system goes belly up, that you have the support you need to be rectified and functional within the time frame that you planned for.

I'd argue where we are going should be for people to predict that an environment is going to go boom and fix it beforehand. MSP's tried to prevent this by doing various things (we did SNMP monitoring, WMI, log analytics and pattern matching for previous issues we had found), but the scale that some of these guys can do. https://www.nimblestorage.com/technology-products/infosight/ 86% fixing problem before impact is pretty damn impressive, as well as the ability to RCA things that are not exactly "their bag/fault" is where stuff is going. Just pointing a finger at the network isn't a solution. Mapping what's wrong (and then assisting with fixing it) is the future. Vendors are starting to bleed into the MSP's capabilities here (and do it better given the size of the data they can collect).

@scottalanmiller said in The argument for official support vs third party support:

I have to agree, if the premise is that people keep choosing software that doesn't work under many conditions and the vendor doesn't fix when a bug is found... sounds like software to generally avoid.

This isn't unique to Hypervisors, this is an issue on switches, routers, and other devices that run proprietary binaries. We had at the MSP tons of bugs we got fixed by Brocade, HPE and Cisco (I'm convinced the RSTP standards committee meetings required everyone drink a bottle of vodka before they got started).

One trend I've noticed is a lot of bugs are now ecosystem bugs. Issues with API's called by backup vendors (who may be doing stupid things that they need to fix). Hardware driver and firmware bugs (Intel basically ignored massive problems with multicast retransmits and LSO/TSO on their 540 NIC for the better part of 4 years). HPE's Switch to Adaptec controllers on Gen 10 came with some.... interesting stability issues. Storage bugs have become more pronounced as people push the limits of SATA SSD's (SATA tunneling protocol was a bad idea in general) and technologies that may have been stable at one time (SAS buffering) just scaled poorly. Your OS vendor having an HCL program (RedHat, VMware do this, Microsoft's 5 pack of Heineken and running a script requirement tends to be a bit weaker) that mandates the other engineers will work with them to get fixes on things. Now you could argue this is the hardware vendors problem (it is) but many times it's the OS/Hypervisor who ends up being the one who forces the issue or puts resources into the RCA. Given the massive monoculture for NIC's, HBA's, and drive controllers this is becoming more "fun". In some ways I"m hopeful things will tame down a bit (the death of SATA and SCSI will help) but in other ways, I'm a bit worried (RCoE could be a mess).

My experience came from working with hundreds of customers, some of whom push the extremes of things, but also SMB's who just lost the hardware/driver/firmware lottery, or discovered (especially with XenApp) that they were the first people to try to use a feature (vGPU offload used to be fun!).

While it's true most vendors will eventually fix critical bugs, if you're not a paying customer they still might take a few weeks. A few weeks of crashes to a company generally isn't cool and is a great way as an MSP to get fired.

The other interesting side of vendor support is the new classes of proactive support (Things like Infosite by Nimble, and Skyline and Humbug at Vmware) where you have telemetry systems written into the software that phone home performance, logs and config data and allow for machine learning systems on the vendor to "Predict" issues across multiple customers. MSP's can't aggregate 500K customers to identify corner cases like this. Cloud providers can, but it's interesting to see traditional infrastructure companies adopt the same model of correlation and continuous improvement.

If you're going to complain about support practices of commercial companies I'd argue the one that bothers me the most is vendors that hide their KB systems and admin guides from people who are not paying. WHAT do they have to hide!

@olivier said in Xenserver and Storage:

Real life usage
So we decided to take a look with some benchmarks, and despite choosing in priority something safe/flexible, we had pretty nice performances, as you can see in our multiple benchmarks.

Your benchmarks leave a lot to be desired. I don't see working set size. Testing the performance of local DRAM (What gluster does). This isn't very real world....

@scottalanmiller said in Xenserver and Storage:

GlusterFS is still RLS, the advice is not really to use a VSAN, but to use RLS. People used to be sloppy and use VSA to refer to RLS, now they use VSAN. Neither is correct as RLS is more than any one connection technology.

Gluster can also be deployed as an external storage system as part of a classic 3 tier design...

@olivier said in Xenserver and Storage:

@matteo-nunziati This is why we have an extra arbiter VM in 2 nodes setup. I node got 2 VMs (1x normal and 1x arbiter), and the other one just a normal VM.

Just to be clear, that arbiter isn't on a node or else if that node that has 2 votes goes down the entire cluster goes boom....

To quote Christos this is trying to cheat physics....

@olivier said in Xenserver and Storage:

Gluster client is installed in Dom0 (the client to access data). But Gluster server are in VMs, so you got more flexibility.

This architecture has a few limitations vs. something running against bare metal on a hypervisor, or a 3 tier storage.

1. You are adding latency to the back-end disk path unless you are running SR-IOV pass thru of the HBA/RAID controller.

2. You are adding TCP overhead (CPU, and 10us of latency) to the front end EVEN if/when the data is local. If you are using NFS to present gluster to the hosts (the supported/tested method).

3. Unless you've invented a native client for Xen, you destroy the primary thing I liked about gluster (local DRAM on the client side being used for ultra-low latency reads) as you are adding 10us and TCP overhead (Well I guess you could do NFS RDMA, but that's even more non-standard/unstable than pNFS)

4. The above hairpins (BACK and front end) burn a lot of extra compute. As you scale (especially on the network transport side) this gets ugly on wasting CPU cores. If you have any applications licensed per core or socket this becomes a nasty "VSA TAX" on your environment vs. a traditional 3 tier storage array deployment or something more efficient.

I do agree with you that 2 node multi-master DRDB is hilarious dangerous. I've personally had to fix split brains multiple times from people doing this and the stateful system (like what gluster uses) is 1000x safer to use. The challenge with DRDB is that the people smart enough to deploy it correctly gennerally are smart enough to do something else instead....

@dbeato said in Xenserver and Storage:

@olivier I would not do HA Lizard, it is problematic with XenServer. You can ask @StorageNinja . I have gone through many SW posts having issues with this. I did recommend it once but it was not worth it. XOSAN will be much better
https://xen-orchestra.com/blog/xenserver-hyperconverged-with-xosan/
or if you can afford two more host with WIndows Server and StarWind VSAN then it would be good too.

Note, XOSAN is just Gluster under the hood. You do NOT WANT TO RUN GLUSTSER WITH 2 nodes. IT IS NOT SUPPORTED. (you can run a 3rd metadata only node, but you need SOMETHING out there to provide quorum).

It requires a proper stateful quorum of a 3rd node. Also for maintenance, you really likely want 4 nodes at a minimum so you can do patching and still take a failure. You'll also need to consider having enough free capacity on the cluster to maintain health slack on the Bricks, (20-30%) AND take a failure, so do that math into your overhead. Also for reasons, I'll get into in a moment you REALLY want to run local raid on Gluster nodes.

Also note, Gluster's local drive failure handling is very... binary... RedHat (who owns Gluster) refuses to issue a general support statement for JBOD mode with their HCI product, and directs you to use RAID 6 for 7.2K drives (no RAID 10). Given the unpredictable latency issues with SSD's (Garbage collection triggering failure detection etc) their deployment guide completely skips SSDs (as I would expect until they can fix the failure detection code to be more dynamic, or they can build a HCL). JBOD because of these risks is a "Contact your Red Hat representative for details." (Code for we think this is a bad idea, but might do a narrowly tested RPQ type process).

@nerdydad said in I am going to start an ISP:

even with oversubscribing by 4:1)

That sounds low for oversubscription.....

@scottalanmiller said in FIPS 140-2 compliance and Ubiquiti VPN:

OpenVPN is FIPS compliant.

It's a cryptographic module that can be inside solutions so the term would be "FIPS 140-2 Inside" technically (As the implementation hasn't been validated).

@tim_g said in Free VM Backup:

@wrx7m said in Free VM Backup:

So is this just an exercise in proving to some exec that you should be paying for a backup and recovery solution?

It's bigger than that and much more to it, but I won't deny that isn't part of it.

Why don't you just post your ACTUAL situation and business problem and we talk about that? Maybe your existing backup target sucks at dedupe and we could recommend a better target storage that would cover that use case?

@jaredbusch said in Free VM Backup:

That many VM's and it is not worth the money to buy a solution?
Just WTF?

You don't want free backup software. Because when it's 3AM and you're trying to do a restore and having an issue you need support. Free means no one verifies that a new patch in windows will not break it.

https://cormachogan.com/2016/01/18/where-are-they-now/

Good read of the boneyard of storage companies.

Ahhhh He's claiming Channel Stuffing"

Ubiquiti's financials don't reconcile with its distributors

Anyone who's done corporate ethics training at a vendor will recognize what he's making allegations of (to be fair, EMC did this with VCE by shipping VMAX's across the street at end of quarter to recognize the revenue before they made it to a real customer as VCE was a customer of EMC).

@scottalanmiller said in Notorious Short-seller labels Ubiquiti Networks $UBNT as FRAUD:

@momurda said in Notorious Short-seller labels Ubiquiti Networks $UBNT as FRAUD:

Short sellers typically don't own the stock they are selling.

In finance, short selling (also known as shorting or going short) is the practice of selling securities or other financial instruments that are not currently owned (usually borrowed), and subsequently repurchasing them ("covering"). In the event of an interim price decline, the short seller profits, since the cost of (re)purchase is less than the proceeds received upon the initial (short) sale. Conversely, the short position closes out at a loss if the price of a shorted instrument rises prior to repurchase.

Yes, but you can't remove that part of the market. All investing is investing "against" something. You can't isolate certain types and limit it. Forcing people to only invest in growth and never in shrinkage is a huge problem.

I took out options against Nutanix after IPO because I thought they were overvalued. I made money. Inversely someone thought it was going up and took the opposite position. What exactly is unethical about what I did? It was a business contract between me and the other party in the option.

https://forums.theregister.co.uk/forum/1/2017/09/08/tintri_second_quarter_revenues_dwarfed_by_losses/

80mil cash, 70mil debt. GAAP Net Loss of $51.7m for the quarter. They are running out of money fast.

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

@anthonyh said in Malicious Logins To Zimbra Mail Server:

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

@anthonyh said in Malicious Logins To Zimbra Mail Server:

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

@storageninja said in Malicious Logins To Zimbra Mail Server:

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

Correct, MTA is always on 25 unless you have an agreement with someone. Then it could be anything.

I'm a bigger fan of having an external service or device (that can mailbag) do your filtering, and then you only accept SMTP with TLS from that service (So your firewall rules don't allow port 25 from the world to the actually mail back end).

Yup, agreed. You never really want to be accept email directly yourself (on your email server, at least.)

What about doing a Zimbra multi-server install and installing the MTA on one VM and the rest of the services on another VM?

Not a bad idea, but doesn't provide you with enterprise mailbagging. It would in no way eliminate the best practice of having an HA hosted mailbagging system.

Right. After I replied I realized what you meant by not accepting mail directly yourself....ha.

I have been considering diving into a multi-server deployment at some point. I've been considering putting the mailbox service on it's own hosts for performance reasons, but maybe instead I can organize services by publicly facing/not publicly facing and do two VMs that way.

In no way does this help in the scenario of the OP, though.

Just got to a larger VM in most cases. Separating them rarely will speed it up until you are going to lots of separate hardware.

I've seen a single VM handle 5000 users just fine (With Exchange). For Zimbra I can't imagine what the point of separating them out is unless it has functionality similar to DAG.

Also to be blunt, why on earth are you manually reading the logs for this stuff? This is a colossal waste of manpower. For security auditing, you should...

1. Outsource this. There are a lot of great SOC/IDS systems.
2. Have an IDS layer 7 devices and reverse proxy manage a lot of this or you. (You shouldn't need to be tweaking brute force detection on different systems).
3. If you care about security stop running your own email server. Pay someone who has dedicated SOC teams, patch management teams, massive spends on layer 7 inspection devices etc.
4. If you work for a F500 you might have a internal SOC, but if you do this you basically are dedicated to this.
5. Invest in internal security (MicroSegmentation and security inspection). Most of your DC traffic (~70%) is east wast and focusing on the external means your likely missing the real attacks as the control channel will be encrypted and tough to find on the stuff coming in north south.

When I worked in consulting, people who were wasting time chasing down hits on their firewall were generally the people looking for a new job a bit later.

@scottalanmiller said in Malicious Logins To Zimbra Mail Server:

Correct, MTA is always on 25 unless you have an agreement with someone. Then it could be anything.

I'm a bigger fan of having an external service or device (that can mailbag) do your filtering, and then you only accept SMTP with TLS from that service (So your firewall rules don't allow port 25 from the world to the actually mail back end).

@anthonyh said in Spiceworld Trip Cancelled Due to CA Legislation - Alternate Conference Suggestions?:

So, I work for a government agency in California. Legislation was passed where the state will not reimburse underlying agencies for travel expenses to states that have law(s) on the book that "have the effect of voiding or repealing existing state or local protections against discrimination..."

Isn't this a breach of the interstate commerce clause?

Also note: ESXi has 2 boot banks so if one corrupts or doesn't install cleanly (to where it can't boot) it will roll back to the alternative one.

At large scale people tend to use host profiles or some sort of state management so they can just restore configurations.