Burned by Eschewing Best Practices
-
@JaredBusch said:
@Dashrender by me of course.
Because the way it was phrased was click baity and inferring more than it was.
I never said it wasn't an outage. I only said it wasn't an Exchange outage. it was an account outage. Semantics. But important to be clear on exactly where the outage occurred.
Something like that can easily be called an Exchange outage initially. but once things are known to be an account outage, then it needs specified.
On the other hand I have also argued with vendors that say they do not have an outage because their upstream provider has an outage. To me the user, it is my vendor's outage. I am not a client of the upstream provider.
This is different because it is unrelated to my account in any way. Unlike the instance @scottalanmiller was referring to.
Also, I want my cake and will eat it too.
I mostly agree here, but the issue, to us, was that while there was an account outage, that triggered an account-localized Exchange and Azure outage. Exchange and Azure services were unavailable to a group of customers (we know of several others affected too, not just our account). So by any normal reasoning yes, there was an account outage, but that's a little like the vendor saying that their ISP failed. To us, as a customer, Exchange and Azure had failed and it something technical on the vendor's side (account is not technical, that they could not fix the account was technical) that they took days in one instance to fix and are still on months trying to fix in the other.
If we use the criteria of "are the services no longer available to the customer(s)" then the answer was yes. The cause was "technical glitches in the account management system" rather than "vendor's ISP failed" , but that's after the point that Exchange and Azure had outages. The end result was Exchange and Azure being down to customer(s).
-
I had a conversation on Wednesday even with someone that just was so crazy it locked up my mind and I could not create a coherent argument. The beer count prior to this conversation also had something to do with it I am sure.
I was out drinking with an old high school buddy that runs a local PC shop in my hometown. He does not do commercial work in general. Basically he spends all day everyday removing viruses and recovering basic stuff for home users. Great guy, knows the limits of what he wants to learn and do, etc.
Anyway, some local guy that is recently into doing some basic consulting work sees us (my friend mostly) and comes and joins us. Basic conversation on various IT stuff ensues, then this guy suddenly starts digging into my buddy wondering if he has any KVM experience (he does not) and if he was wanting to maybe get in on a deal with him to put a rack in his office and set up severs to for people.
A little questioning from me and he is like MS sucks. VMWare is crap and no way would I use Citrix. All that stuff is too expensive. I setup everything myself in Linux and use KVM.
I just locked up... I mean WTF and you have actual clients? Apparently he does. Currently all hosted out of his house. I am fairly certain that violates the ToS on his internet service.
He seriously had no answer for why he was not using something with any kind of support. I mean, I have nothing against KVM, but damn if I am gong to pay for my shit to be someplace, I want it someplace with something behind it.
I told the guy, if you want KVM, buy Scale gear. If you want free, use XenServer or Hyper-V in that order. Either way do something that approaches some kind of industry standard that you can get supported.
-
@JaredBusch said:
I had a conversation on Wednesday even with someone that just was so crazy it locked up my mind and I could not create a coherent argument. The beer count prior to this conversation also had something to do with it I am sure.
I was out drinking with an old high school buddy that runs a local PC shop in my hometown. He does not do commercial work in general. Basically he spends all day everyday removing viruses and recovering basic stuff for home users. Great guy, knows the limits of what he wants to learn and do, etc.
Anyway, some local guy that is recently into doing some basic consulting work sees us (my friend mostly) and comes and joins us. Basic conversation on various IT stuff ensues, then this guy suddenly starts digging into my buddy wondering if he has any KVM experience (he does not) and if he was wanting to maybe get in on a deal with him to put a rack in his office and set up severs to for people.
A little questioning from me and he is like MS sucks. VMWare is crap and no way would I use Citrix. All that stuff is too expensive. I setup everything myself in Linux and use KVM.
I just locked up... I mean WTF and you have actual clients? Apparently he does. Currently all hosted out of his house. I am fairly certain that violates the ToS on his internet service.
He seriously had no answer for why he was not using something with any kind of support. I mean, I have nothing against KVM, but damn if I am gong to pay for my shit to be someplace, I want it someplace with something behind it.
I told the guy, if you want KVM, buy Scale gear. If you want free, use XenServer or Hyper-V in that order. Either way do something that approaches some kind of industry standard that you can get supported.
LOL - that situation is extremely common. Most people never ask, or care about the back end until it's to late.
-
So even after we've told the OP here, that by allowing Modify access to the Shared Desktop is bad practice, he ignores our advice and does it anyways....
He's not yet burned, but likely will be in the future....
The Shared Desktop on Windows is designed to be controlled by the System Administrator, not every user to delete (and add) what they want / don't want on their desktops.
If a few users complain about a "messy" desktop, go and delete the shared icons for them, rather then circumventing the system and granting everyone access to modify what is there...
-
Not exactly a best practice, but a general rule is that if you have a good, enterprise hardware solution you don't replace it with software RAID without a really, really good reason (this started to become a thing because of the Cult of ZFS and FreeNAS community issues a few years ago.)
This guy didn't research how the most standard controller on the market works, made a ridiculous assumption that "all" businesses lack super basic functionality and lept to crazy conclusions resulting in him throwing away a $600 controller, buying a $200 FakeRAID device, using it like a $100 SAS controller and resulting in him being unable to read performance results from the resulting system and losing big features like blind swap. All around a misconception that if he had asked a question about, people would have corrected him instantly. Instead he decided that HP was crazy and that their and Adaptec's super expensive, top end controllers did not work and that the entire IT world was crazy and tried to reinvent the wheel.
-
Sometimes you have to step back and think "is every enterprise AND this really, really high end enterprise vendor really that crazy?" Sometimes it just isn't reasonable to assume such ridiculous things and we need to say "maybe I'm missing something."
I remember when someone, a Microsoft MCSE+I actually, tried to tell me that he could not correctly set the timezone on his Windows server because Microsoft didn't get timezones right and they just didn't work. I said "That's not a reasonable thing to assume. I don't have any proof that this is wrong, but it is so unreasonable to assume that after thirty years Microsoft hasn't gotten the time right and that every business in the world is running Windows servers with the wrong time that it isn't true, it simply cannot be true."
He actually argued that all businesses suffer from having the time wrong, off by an hour half the year, and just deal with it. I said "no, that's just insanity and you are making that up."
We dug more and yes, it turned out he simply didn't know his timezones and had imagined the entire problem. Lots of people mess up time zones, that's not a big deal. What's a big deal was that in his mind he jumped from "the time is off" to "every business in the world has its time off because of Microsoft" without any reason to believe so other than that he couldn't figure out how to configure it properly.
Sometimes if we just assume that we are wrong and haven't figured out the answer yet, we will get to a good conclusion far, far faster than if we assume that giant vendors are screwing up really basic things.
-
So this isn't by any means a Burned by Eschewing Best practices topic, but it definately leads down that path (FYI this is a private conversation from SW and posted here, I've removed the other person's name for their privacy as I have no idea how they might feel about me posting the conversation here)
Now to gain context to the conversation you'll want to reference the topics from SW, but the OP was attempting to find a way to backup his Snapshots as a way of creating backups. This he wanted to script himself, which is weird, when there are many well documented solutions to properly building your VM Backups.
DustinB3403 Jan 9, 2016 at 9:14 AM I saw your post at http://community.spiceworks.com/topic/post/5373821 You might want to read some of my topics on on mangolassi.it you'll find them to be very useful. http://mangolassi.it/topic/7349/xen-orchestra-on-ubuntu-15-10-complete-installation-instructions http://mangolassi.it/topic/7474/xen-orchestra-delta-restore http://mangolassi.it/topic/7467/xenserver-disk-or-array-performance-monitoring http://mangolassi.it/topic/7457/xenserver-usb-pass-through http://mangolassi.it/topic/7476/hypervisor-hypervisor-who-s-got-the-best-hypervisor Jan 9, 2016 at 5:47 PM thanks, i will look at these, thanks for your help, appreciated! DustinB3403 Jan 10, 2016 at 3:34 PM I'm giving you a solid warning, don't think backing up snapshots is a way to protect the vm. It isnt. Use the tools I recommended in the topic, either NAUBACKUP or XenServer Orchestra Jan 10, 2016 at 6:56 PM But the free version of XO doesnt let you make backups clones etc its either the 70 dollars or 200 dollars? DustinB3403 Jan 10, 2016 at 7:04 PM XO free allows you to do everything that the paid version does but for free. What do you mean backup clones? A second copy of a vm incase something goes wrong? Jan 10, 2016 at 7:06 PM Yes exactly if my original/primary vm fails/corrupt i can fire up the clone/backup so there os no down time and no lose of data depending how long you leave it Jan 10, 2016 at 7:08 PM Really the free alows ypu to do this straight of the bat, what do the paid versions do then that are different or better DustinB3403 Jan 10, 2016 at 7:08 PM That is why you create a proper backup. Snapshots won't work if you have a broken vdisk for a vm. You really need to read up on Xen Orchestra or even NAUBACKUP DustinB3403 Jan 10, 2016 at 7:09 PM The paid version comes with support from the developer Jan 10, 2016 at 7:11 PM Ok great i will install XO on a vm on my host and start playing with it and do scheduled backups or clones DustinB3403 Jan 10, 2016 at 7:12 PM Delta backups are your friend. They save a huge amount of disk space Jan 10, 2016 at 7:15 PM I will look up delta, thanks, i will get back to you if run into plems, thanks mate DustinB3403 Jan 10, 2016 at 7:17 PM Delta are incremental backups or differential if that makes it easier to understand Jan 10, 2016 at 7:22 PM Awesome sounds fantastoc, even better, so you can create scheduled backups and once the first time its backed up from then on you can do incremental schedule backups? And tbis is all using the FREE version of XO DustinB3403 Jan 10, 2016 at 7:25 PM Yep. All for free. You can even talk with him on mangolassi.it Jan 10, 2016 at 7:27 PM Thanks, you have saved me alot of batch scripting DustinB3403 Jan 10, 2016 at 7:33 PM I just put you on a good parh, others saved you and I am ton of scripting time. Jan 10, 2016 at 7:37 PM But what was so wrong about the way i was doing it ie Create a snapshot and then make that snapshot into a vm I tried it and it worked good, even deleted the original vm to see if it was using that vm in anyway but it wasnt so i know it was independent DustinB3403 Jan 10, 2016 at 7:41 PM Well it's weird for one. And you'd never want to delete your vm if you can avoid it. Snapshots also don't keep all of the information to be reliable as a recovery method. Would you want to rebuild your vm from something you've seen work once, or with something you can research as a solution and know it works? Jan 10, 2016 at 7:45 PM When you say snapshots also dont keep all information to be reliable So why make a convert snapshot to vm tool in xen center DustinB3403 Jan 10, 2016 at 7:49 PM For testing purposes or to have a template of that vm. Jan 10, 2016 at 7:52 PM But to have a template of that vm is just as good isnt it as the name says a template from the original vm so its all there ie in the new template DustinB3403 Jan 10, 2016 at 7:54 PM The uuid and Mac address changes, so no it's not all there. Templates and snapshots are not backups. They just don't work. Scottalanmiller can explain better than i. It's just not designed to be used to recover a dead vm from Jan 10, 2016 at 8:00 PM one last question is it best to install XO on a vm of the host or on a physical server DustinB3403 Jan 10, 2016 at 8:02 PM I set it up as a vm on my host. My backup target (nfs server) is a vm on a separate Xen server host. Jan 10, 2016 at 8:06 PM mmm i was going to suggest having two SRs on the host one is live vm disks and the other is backup vm disks DustinB3403 Jan 10, 2016 at 8:08 PM And what if your server catches fire, or a leak in the roof, or a failed raid array, or a URE on the array, or a fried powersupply Jan 10, 2016 at 8:10 PM no i mean one sr on the host and the other is an nfs or iscsi synology not in thevsame building DustinB3403 Jan 10, 2016 at 8:13 PM The saying is "virtualize everything you can, including your backups". Separating them in different buildings is good, but you still want to virtualize them. Once you have a virtual copy you can back them up to a synology or usb or tape. Jan 10, 2016 at 8:17 PM ok so put them on another xenserver with a same size sr on the pool and then make a physical backup to a nas or tape what would you say is better for physical backups, iscsi or nfs or cifs DustinB3403 Jan 10, 2016 at 8:18 PM Here is a really great explanation of the 3-2-1 rule. https://www.veeam.com/blog/how-to-follow-the-3-2-1-backup-rule-with-veeam-backup-replication.html DustinB3403 Jan 10, 2016 at 8:19 PM Nfs is just so simple, and native to xen... iscsi you have to deal with lun #s etc. Jan 10, 2016 at 8:20 PM true ie creating volumes luns and targets just an extra headache -
@scottalanmiller said:
Not exactly a best practice, but a general rule is that if you have a good, enterprise hardware solution you don't replace it with software RAID without a really, really good reason ......
This one keeps going. As expected, once there is a lack of best practices or "being weird" happens, chances are there are layers and layers of it. So far we've found physical installs and now an IPOD built on disposable hardware. Haven't gotten to the point of looking at the business goals yet, but it's a good discovery process. OP is good about accepting the advice and isn't reacting badly. Probably just out of his depth. He got caught by the familiar bad advice from the FreeNAS flunkies that catch so many storage newbies.
-
http://community.spiceworks.com/topic/1388338-looking-to-add-raid-to-a-freepbx-server
- No RAID
- Looking at using a $5 "RAID" card
- Physical install
- Less than desktop class hardware
- Posts a storage question in a VoIP forum because he thinks that the application running on top of the OS is what matters and not that he is asking about how to implement RAID? Um... who cares what application is running up there when you are asking what RAID card is compatible with the OS?
-
This just went in the IPOD thread as it is that too. But this guy also got burnt by having a salesman make his IT decisions. he did the "thinking he could get free IT work" thing and got screwed, but time. To the tune of tens of thousands of dollars and has a setup not nearly as good as something much cheaper. Napkin estimate says he overspent by $40K on a project that should not have cost more than $20K total!
http://community.spiceworks.com/topic/1392732-w00t-excited-for-this-project
-
@scottalanmiller said:
This just went in the IPOD thread as it is that too. But this guy also got burnt by having a salesman make his IT decisions. he did the "thinking he could get free IT work" thing and got screwed, but time. To the tune of tens of thousands of dollars and has a setup not nearly as good as something much cheaper. Napkin estimate says he overspent by $40K on a project that should not have cost more than $20K total!
http://community.spiceworks.com/topic/1392732-w00t-excited-for-this-project
I'm assuming you realize that probably something like 90% of IT is actually done this way (maybe even more in the SMB only market)?
That asked, how do we change it? Is it even possible to change it? I think you're trying to overcome human nature with this problem.
-
@Dashrender said:
That asked, how do we change it? Is it even possible to change it? I think you're trying to overcome human nature with this problem.
Demand accountability. Hire better people. Take IT serious. Invoke oversight. Treat IT like any other department. Audit for incompetence. Use forums to verify BEFORE purchasing instead of using them for congratulations when no work was done.
What aspect do you feel is human nature, incompetence?
-
@Dashrender said:
I'm assuming you realize that probably something like 90% of IT is actually done this way (maybe even more in the SMB only market)?
57% of adults buy lotto tickets too. We still try to fix that when we can. That most people don't bother being even slightly good at their jobs or caring at all is a problem, one that needs to be fixed from the business side by hiring fewer, better people rather than hiring low cost in bulk.
-
@scottalanmiller said:
What aspect do you feel is human nature, incompetence?
yes.
pushing your work off on someone else - human nature. -
@Dashrender said:
@scottalanmiller said:
What aspect do you feel is human nature, incompetence?
yes.
pushing your work off on someone else - human nature.That's actually running a scam. That's not incompetence, that's actually a con job. That's the same as saying most people are unethical. Probably true. There is a simple solution. You fire them. In some cases, you sue them as well.
-
@scottalanmiller said:
@Dashrender said:
@scottalanmiller said:
What aspect do you feel is human nature, incompetence?
yes.
pushing your work off on someone else - human nature.That's actually running a scam. That's not incompetence, that's actually a con job. That's the same as saying most people are unethical. Probably true. There is a simple solution. You fire them. In some cases, you sue them as well.
interesting, though I would say a con job would require fore thought of wrong doing (they knew they were doing the wrong thing). Most of those people (my friend who did exactly the same as your most recent post - sales person sold them a 1 server VMWare solution with a SAN - yes guys and gals you read that right.... A ONE server solution with a SAN. sigh!
But my friend didn't know any better. He's just always heard through conversations that when you virtualize you also use SAN. He's never been a forum user, so he didn't ask for any help.
He was also a helpdesk person who took over the reigns as the IT admin of a small school district only 3 months earlier... so it was a perfect storm of crap.
-
He deleted his post.
-
-
-
I just considered another problem here - in the works toward fixing the problem...
The businesses themselves (mostly SMB) don't want to spend money on consulting. I definitely have the problem here. The BOD has this broken idea that I work for them, that should be all they ever need. They shouldn't need to hire/pay for consultants because, hey, that's your (mine) job.
So those admins who find themselves fairly swamped taking care of the day to day cruft don't have the time to educate themselves, so they turn to the only resource they do have, the sales people.... who of course are more than willing to put together a quote for something they sell no matter how good or bad it is for you.
