The Inverted Pyramid of Doom Challenge
-
Picked up a client.
2 epic dumb-assery things done by their "MSP"
Single server environment, 10-15 user office, very small. The original provider left RDP open to the internet with no AD security, no account lock outs, nothing.
Bad guys brute forced in, dropped ransomware on the server, took them down for a day and a half.
Same client, the outside company decided that UAC was causing issues, so they disabled it on every machine. So, I'm working remotely, nothing is responding to me when I give instructions whilst logged in as standard user, Can't log out....because....I can't elevate the session t'o run as admin....because I can't elevate anything to run as admin.
-
@Veet said in The Inverted Pyramid of Doom Challenge:
@DustinB3403 said in The Inverted Pyramid of Doom Challenge:
@Veet said in The Inverted Pyramid of Doom Challenge:
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
This is my quote from the original challenge: "We all (I hope by now) know that SANs have their place and a super obvious one that explains why enterprises use them almost universally and know why that usage has no applicability to normal SMBs - scale."
I agree with why lots of shops might deploy systems like you are describing, even if I generally don't agree with that decision, but I'm pretty confident that the use cases that you are describing @John-Nicholson are tied, nearly universally, to a scale that would already prompt a SAN-based infrastructure (or similar.)
Have you seen these in small environments where the scale did not exist to warrant a SAN otherwise?
Just a couple of months ago - I was contacted by a prospective client , who was looking to get his website designed ... So, I went over to his office one day, for a general face-to-face, and we got talking, and quite proudly mentioned about recently acquiring a Synology DS2015 box ... which was all pretty alright, until he mentioned why .. It turned out that their vendor recommended that they migrate their one Windows 2012 server to a VM, and that, if they WANTED RELIABILITY, SCALABILITY & PERFORMANCE, they would HAVE TO, move from a local storage to a NAS .. btw, their current total data size is a little less than 1TB ... They have around 40 users ... Now, for the cherry on the cake .. The vendor took-out the 2x2TB HDDs from the server, and reused them in the new NAS box. Apart from that, they installed another 2 TB HDD in the NAS box for "Backups" (Can you believe it, I could not ), and then installed a 128 GB HDD on the server, to install Hyper-V 2012. This , the vendor said would "further increase performance,, and that they did not have to buy new HDDs, which would save money" The VMs and data were on the NAS box ...
Upon, pointing-out and explaining the rather obvious flaws in this design, the client was left rather gobsmacked ... Anyway, I designed their website, and will be taking-over the support & maintenance of their IT, once the annual contract with the existing vendor runs it course.... I recommended, that they reattach the HDDs to the server, and run everything locally, and return or try to sell-off the DS2015 box, and get a smaller one, just for back-ups (VEEAM)... I hear, that the existing vendor, recently agreed to take back the DS2015, and compensate them by installing a lower-end 4 bay box, and by extending their service contract (I'm not sure if my client is going to agree to this) ....
Shocking, no ?
This is the same practice many SMB's experience every day. The IT Vendor clearly doesn't have an expert in house, just someone who gets paid to sell hardware with enough experience to setup some basic hardware.
I'm not shocked, and glad you were able to point out the issues. I didn't see what server they have that was scaled back to just a compute node though. . .
I don't think it's about lack of knowledge or experience ... I feel, it's just about unscrupulous business practice, of up-selling something ...
Stupidity, and there's multiple people to blame.
-
Small business's are not blameless, they should seek good advice and consulting. When they hire people who they pay less per hour than geek squad this is what you get. By refusing to pay real consulting rates, this is what they end up getting...
-
Small consulting shops that center around Hyper-V these days seem to be in love with building clusters on prosumer grade QNAP/synology etc. As there is no deal registration its not actually something they can "mark up" much. It does add a lot of labor, but you have to look at these shops training commitment (or often lack there of). They tend to be fed with cert mill grade MCSE's who learned out to make a Hyper-V cluster and Microsoft's storage curriculum emphasis this without ever discussing quality of storage. (Meanwhile a VCP 5.5 or newer will cover scale out local storage as there are quite a few VSAN questions on that test).
-
There is a growing trend where the self taught IT guy in the SMB's knoledge is drifting farther and farther from the enterprise. The tools and best practices are making the "Cargo Cult of the Enterprise" even more dangerous.
As far as a shop with only a single server instance I'm starting to ask why even bother? Why not host the application, get as many of your apps delivered by SaaS, and leverage MDM/MAM/SSO tools and move away from the need for GPO or local domain for management.
Does doing this cost a little more? Sure. It does however give you a much more transparent cost to IT (Your not assuming risks because the SLA's are fairly well known and far more absolute from a SaaS provider these days than a server in a closet).
I think our real boogyman in the (S in SMB) is not the guy with the Synology but anyone advocating physically local servers at all. Servers with some exceptions for SME's or niche industries increasingly belong in datacenters.
-
-
@John-Nicholson
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
@Veet said in The Inverted Pyramid of Doom Challenge:
@DustinB3403 said in The Inverted Pyramid of Doom Challenge:
@Veet said in The Inverted Pyramid of Doom Challenge:
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
This is my quote from the original challenge: "We all (I hope by now) know that SANs have their place and a super obvious one that explains why enterprises use them almost universally and know why that usage has no applicability to normal SMBs - scale."
I agree with why lots of shops might deploy systems like you are describing, even if I generally don't agree with that decision, but I'm pretty confident that the use cases that you are describing @John-Nicholson are tied, nearly universally, to a scale that would already prompt a SAN-based infrastructure (or similar.)
Have you seen these in small environments where the scale did not exist to warrant a SAN otherwise?
Just a couple of months ago - I was contacted by a prospective client , who was looking to get his website designed ... So, I went over to his office one day, for a general face-to-face, and we got talking, and quite proudly mentioned about recently acquiring a Synology DS2015 box ... which was all pretty alright, until he mentioned why .. It turned out that their vendor recommended that they migrate their one Windows 2012 server to a VM, and that, if they WANTED RELIABILITY, SCALABILITY & PERFORMANCE, they would HAVE TO, move from a local storage to a NAS .. btw, their current total data size is a little less than 1TB ... They have around 40 users ... Now, for the cherry on the cake .. The vendor took-out the 2x2TB HDDs from the server, and reused them in the new NAS box. Apart from that, they installed another 2 TB HDD in the NAS box for "Backups" (Can you believe it, I could not ), and then installed a 128 GB HDD on the server, to install Hyper-V 2012. This , the vendor said would "further increase performance,, and that they did not have to buy new HDDs, which would save money" The VMs and data were on the NAS box ...
Upon, pointing-out and explaining the rather obvious flaws in this design, the client was left rather gobsmacked ... Anyway, I designed their website, and will be taking-over the support & maintenance of their IT, once the annual contract with the existing vendor runs it course.... I recommended, that they reattach the HDDs to the server, and run everything locally, and return or try to sell-off the DS2015 box, and get a smaller one, just for back-ups (VEEAM)... I hear, that the existing vendor, recently agreed to take back the DS2015, and compensate them by installing a lower-end 4 bay box, and by extending their service contract (I'm not sure if my client is going to agree to this) ....
Shocking, no ?
This is the same practice many SMB's experience every day. The IT Vendor clearly doesn't have an expert in house, just someone who gets paid to sell hardware with enough experience to setup some basic hardware.
I'm not shocked, and glad you were able to point out the issues. I didn't see what server they have that was scaled back to just a compute node though. . .
I don't think it's about lack of knowledge or experience ... I feel, it's just about unscrupulous business practice, of up-selling something ...
Stupidity, and there's multiple people to blame.
-
Small business's are not blameless, they should seek good advice and consulting. When they hire people who they pay less per hour than geek squad this is what you get. By refusing to pay real consulting rates, this is what they end up getting...
-
Small consulting shops that center around Hyper-V these days seem to be in love with building clusters on prosumer grade QNAP/synology etc. As there is no deal registration its not actually something they can "mark up" much. It does add a lot of labor, but you have to look at these shops training commitment (or often lack there of). They tend to be fed with cert mill grade MCSE's who learned out to make a Hyper-V cluster and Microsoft's storage curriculum emphasis this without ever discussing quality of storage. (Meanwhile a VCP 5.5 or newer will cover scale out local storage as there are quite a few VSAN questions on that test).
-
There is a growing trend where the self taught IT guy in the SMB's knoledge is drifting farther and farther from the enterprise. The tools and best practices are making the "Cargo Cult of the Enterprise" even more dangerous.
As far as a shop with only a single server instance I'm starting to ask why even bother? Why not host the application, get as many of your apps delivered by SaaS, and leverage MDM/MAM/SSO tools and move away from the need for GPO or local domain for management.
Does doing this cost a little more? Sure. It does however give you a much more transparent cost to IT (Your not assuming risks because the SLA's are fairly well known and far more absolute from a SaaS provider these days than a server in a closet).
I think our real boogyman in the (S in SMB) is not the guy with the Synology but anyone advocating physically local servers at all. Servers with some exceptions for SME's or niche industries increasingly belong in datacenters.
Yep... I agree - Small Businesses are not blameless .. . For some, it's as if, it's a matter of pride, that they have a NAS or SAN ... Most small businesses will opt for vendors who come cheap ...This is what I say to them "You throw peanuts, you'll attract only monkeys"
Where, I don't agree with you - Shift to a hosted SaaS based platform ...been there .. done that .. failed ... does not work, where I am ... Primarily due to poor bandwidth quality... I have a few clients, with factories in areas, where no ISP even wants to provide service ..
Hosting with local data centers is an expensive affair over here (India) ... If we were to host in US/Europe data centers (which work-out to be cheaper), then the issue of latency arises ..
-
-
Give us 3-4 months and i'll put you in contact with our MSP as they will be trying to sell us some hardware and SAN with a refresh, even after i've suggested better like Scale
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
-
I think one thing that is important to mention is that the term SAN as used in this topic refers to a single storage node upon which all of your data rests.
Back 7 or 8 years ago when I first started learning about SAN systems, I was shocked that people would only build or buy based on a single unit, without understanding that if that single unit died, they were dead in the water.
-
@hobbit666 said in The Inverted Pyramid of Doom Challenge:
Give us 3-4 months and i'll put you in contact with our MSP as they will be trying to sell us some hardware and SAN with a refresh, even after i've suggested better like Scale
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
Do you have a write up of common purchasing mistakes to hand to management before hand?
-
@travisdh1 said in The Inverted Pyramid of Doom Challenge:
@hobbit666 said in The Inverted Pyramid of Doom Challenge:
Give us 3-4 months and i'll put you in contact with our MSP as they will be trying to sell us some hardware and SAN with a refresh, even after i've suggested better like Scale
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
Do you have a write up of common purchasing mistakes to hand to management before hand?
I think @scottalanmiller could contribute a few points to that one as well!
-
@dafyre said in The Inverted Pyramid of Doom Challenge:
@travisdh1 said in The Inverted Pyramid of Doom Challenge:
@hobbit666 said in The Inverted Pyramid of Doom Challenge:
Give us 3-4 months and i'll put you in contact with our MSP as they will be trying to sell us some hardware and SAN with a refresh, even after i've suggested better like Scale
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
Do you have a write up of common purchasing mistakes to hand to management before hand?
I think @scottalanmiller could contribute a few points to that one as well!
I'll just print off his blog site that should do
-
@hobbit666 said in The Inverted Pyramid of Doom Challenge:
Give us 3-4 months and i'll put you in contact with our MSP as they will be trying to sell us some hardware and SAN with a refresh, even after i've suggested better like Scale
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
There are other service providers...hint hint, nudge nudge....
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
One last thought...
IF the reason that Xen has 2% market share is because there is NO LOGICAL REASON for vSphere or paid Hyper-V (with VMM to manage) then that means 98% of IT people are idiots. If 98% are idiots, wouldn't that mean they should be outsourcing their IT as much as possible to their vendors or others? (and therefore not deploy Xen).
Catch-22
Xen has 2% of the market share because it never came out of the niche. Most admins are lazy (it's natural, all people are so deeply inside and lazy = OK) so if you need to perform 1 simple activity with ESXi and it "just works" but you need to run many whistles with Xen... Naturally you'll go ESXi next time! Just because you want to spend these +2 hours drinking Coors and watching Vikings losing another game. Being too professional != being good. IMHO of course.
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
There was thread on SW recently where someone said "NIMBLE SUCKS I DON"T GET THE IOPS I PROMISED". The next post was his Nimble sales rep posting "So I see your at 20% load, your IO latency is .5 ms currently and while your 220C model is one of our smaller ones we have far larger ones. If your having any problems please call us and we will help you" I laughed, but it made me realize the damage that incompetent IT do to the name of a product or application. We are at the point that a sales rep would rather piss off a customer and call them out as an idiot (he was nice about it) than risk their companies name being drug through the mud.
That's not incompetence, though. That's just someone lying. there is a difference.
There's no line in sand about that. Whatever you're going to do 10 people are going to love you but there will be one who'll either hate you or he'll not care. Nimble is more or less safe - they don't do software, but with software it's very easy to a) misconfigure and b) break something working and done by other guys. Who's one to blame? Of course storage vendor! He has SLAs!
-
@KOOLER said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
One last thought...
IF the reason that Xen has 2% market share is because there is NO LOGICAL REASON for vSphere or paid Hyper-V (with VMM to manage) then that means 98% of IT people are idiots. If 98% are idiots, wouldn't that mean they should be outsourcing their IT as much as possible to their vendors or others? (and therefore not deploy Xen).
Catch-22
Xen has 2% of the market share because it never came out of the niche. Most admins are lazy (it's natural, all people are so deeply inside and lazy = OK) so if you need to perform 1 simple activity with ESXi and it "just works" but you need to run many whistles with Xen... Naturally you'll go ESXi next time! Just because you want to spend these +2 hours drinking Coors and watching Vikings losing another game. Being too professional != being good. IMHO of course.
I'd argue that Xen is the market leader by far, whereas XenServer you're right on track with. Picking at straws I know.
-
@travisdh1 said in The Inverted Pyramid of Doom Challenge:
@KOOLER said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
One last thought...
IF the reason that Xen has 2% market share is because there is NO LOGICAL REASON for vSphere or paid Hyper-V (with VMM to manage) then that means 98% of IT people are idiots. If 98% are idiots, wouldn't that mean they should be outsourcing their IT as much as possible to their vendors or others? (and therefore not deploy Xen).
Catch-22
Xen has 2% of the market share because it never came out of the niche. Most admins are lazy (it's natural, all people are so deeply inside and lazy = OK) so if you need to perform 1 simple activity with ESXi and it "just works" but you need to run many whistles with Xen... Naturally you'll go ESXi next time! Just because you want to spend these +2 hours drinking Coors and watching Vikings losing another game. Being too professional != being good. IMHO of course.
I'd argue that Xen is the market leader by far, whereas XenServer you're right on track with. Picking at straws I know.
At very least in the public, high end, enterprise cloud space.
-
@dafyre said in The Inverted Pyramid of Doom Challenge:
I think one thing that is important to mention is that the term SAN as used in this topic refers to a single storage node upon which all of your data rests.
Correct, to be an IPOD it has to be a single storage node, but not necessarily SAN. It could be SAN, NAS, DAS, a USB drive, a shared JBOD enclosure, whatever. Many things resting on the reliability of one.
-
@hobbit666 said in The Inverted Pyramid of Doom Challenge:
Might be suggesting a 3rd party review of the solution so look out for an E-mail NTG
@Minion-Queen waits with baited breath.
