The Inverted Pyramid of Doom Challenge
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
SAP HANA is a incredible pain in the ass to tune and setup, and depending on the underlying disk engine may have different block sizing or other best practices. This is one of the times where things like NUMA affinity can actually make or break an experience. Getting their support to understand a platform enough to help customers tune this, their PSO assist in deployments, and their support identify known problems with the partner ecosystem means they are incredibly restrictive (Hence they ares still defining HCI requirements).
Right, so at this point you are talking about outsourcing your IT department as this isn't application work, this is infrastructure work. So this is turning into a completely different discussion. Now you are hiring an external IT consulting group that doesn't know the platform(s) that you might be running. That's a totally different discussion.
But what we are talking about here is needing an application vendor to do underlying IT work for them. It's a different animal. It does happen and there is nothing wrong with outsourcing IT work, obviously, I'm a huge proponent of that. But there is no need to get it from the application vendor, that some do might make sense in some cases, that application teams demand that they also be your IT department is a problem unless you are committed to them delivering their platform as an appliance and it should be treated that way in that case. Nothing wrong with that per se and a lot of places do just that.
This is largely SAP's support model today. They have validated appliance or tightly controlled RA's. Honestly making in memory database work is such a niche skill set I don't blame them for demanding to take that on for two reasons.
- Its weird work most in house IT will not know.
- They can charge a ton of money.
- No one complains because if your using HANA for what its intended the 2 million you spend on it is a joke vs the benefit it brings to your org.
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Backups - (Some Hypervisors have changed block tracking so a backup takes minutes, others don't meaning a full can take hours). BTW I hear Hyper-V is getting this in 2016 (Veeam had to write their own for their platform).
Sure... but what does the application layer care? Either the application takes care of its own backups and doesn't care what the hypervisor does, or it relies on IT to handle backups and it isn't any of their concern either.
Again, this is an application vendor or programmer trying to get involved in IT decisions, processes and designs. Do you let the company that makes your sofa determine how big your fireplace has to be because "they want to ensure that you are cozy?"
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
This is largely SAP's support model today. They have validated appliance or tightly controlled RA's. Honestly making in memory database work is such a niche skill set I don't blame them for demanding to take that on for two reasons.
- Its weird work most in house IT will not know.
- They can charge a ton of money.
- No one complains because if your using HANA for what its intended the 2 million you spend on it is a joke vs the benefit it brings to your org.
And it helps to explain why SAP failure rates are through the roof. They are the standard example used in nearly all studies for where projects just totally fall flat. SAP is famous for not delivering working systems. When they do pull it off, they are very good. But very often, they don't. Dealing with one right now, actually, and SAP has been failing big time, even in a very small deployment.
But it's also very important to note that when we are talking about systems getting this big, how often does it have any value to merging this onto other infrastructure, anyway? If I was getting SAP and spending millions to acquire it, why bother with trying to make all other systems fit onto the "leftovers" of the SAP one?
The argument for "why someone like SAP needs to determine all kinds of things under the hood" equally causes problems that other vendors might say "well SAP messed this up so we won't support it." So it very strongly says that if you are going to accept a vendor that needs this kind of control to try to make money and avoid blame, that you need to isolate your different vendor's systems so that they can all make these independent choices.
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Because performance and availability problems come from the bottom up not the top down. SQL has storage as a dependency, storage doesn't have SQL as a dependency, and everything rolls downhill...
That doesn't make sense, though. Applications care that they have enough CPU, memory, IOPS, bandwidth, etc. That's it. They don't care how it is delivered, only that it is available when needed. This would be, again, a failing of any application team and any IT team if they look to the application for issues involving not providing enough resources for performance.
If your point here is that incompetent IT departments tend to buy unsupportable, crappy software.. sure. No one is denying that people don't do their jobs well. But that doesn't mean we should recommend doing things poorly just because lots of people aren't good at their jobs.
Most IT departments (Even enterprises) are not skilled (or skilled well) at troubleshooting infrastructure (Especially beats like ERP that might have a dozen interdependent systems) without assistance. Most ERP vendors know this and so rather than let the customers deploy a database for 20K users on a Hyper-V host with a 3 Disk RAID 5 (and then the project be written off as a failure and their name be damaged) take this choice away.
For the 5 years I consulted "why is this slow" was one of the most common engagements. 9/10 of the time I was chasing some crazy application issue it had nothing to do with the application. Generally it was staring people in the face, had a giant RED alarm, and was fairly obvious (Disk latency isn't supposed to be 1200ms, and NL-SAS drives shouldn't be used for DB's in 5 billion dollar companies Yo?). Assuming that vendors are crossing a line by assuming internal IT doesn't understand what it will take to deliver their applications is CRITICAL to being a successful application vendor. I've seen users, IT and C-suite trash applications that worked fine, but the infrastructure was all wrong....
This is part of a huge reason for many vendors pushing for SaaS offerings, or OPEX offerings. If you don't bundle high levels of support that can extend beyond the application your risking your revenue. Much like why Scale (and other highly successful HCI vendors) try to own support of the ENTIRE stack. If they didn't own support of the hypervisor people would do awful, awful things then blame them.
Its not fair, but its the reality we live in...
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Most IT departments (Even enterprises) are not skilled (or skilled well) at troubleshooting infrastructure (Especially beats like ERP that might have a dozen interdependent systems) without assistance. Most ERP vendors know this and so rather than let the customers deploy a database for 20K users on a Hyper-V host with a 3 Disk RAID 5 (and then the project be written off as a failure and their name be damaged) take this choice away.
Most ERP vendors are not skilled at this either, though. SAP continuously fails at this. Their competitors are often worse. Some actually do the opposite and specifically require three disk RAID 5 rather than do this to avoid it.
Sure, most IT departments are bad. But again we are going down the "assumption of bad decisions" bad. The one that says "we should make bad decisions, because we make bad decisions, so we start recommending bad things." It's not good logic to say "people are often dumb, so we assume you are and make products based on that." That might be good logic for making money (and why vendors always recommend it) but it's not a good idea to do business with those vendors.
Basically how I read this is "in these cases, your vendor is building their products based around you not being competent and that you will make bad decisions." That's great, that just, to me, repeats by original point that IT should rule out those vendors.
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Backups - (Some Hypervisors have changed block tracking so a backup takes minutes, others don't meaning a full can take hours). BTW I hear Hyper-V is getting this in 2016 (Veeam had to write their own for their platform).
Sure... but what does the application layer care? Either the application takes care of its own backups and doesn't care what the hypervisor does, or it relies on IT to handle backups and it isn't any of their concern either.
Again, this is an application vendor or programmer trying to get involved in IT decisions, processes and designs. Do you let the company that makes your sofa determine how big your fireplace has to be because "they want to ensure that you are cozy?"
Application owners have RPO/RTO's and they expect the infrastructure people to often take care of that. (When I have a 5TB OLTP database, in guest options generally fail to deliver somehow).
If I buy a couch or desk that's massive for a tiny apartment I could see the sales guy asking how big my door way is to make sure they can deliver it. Otherwise I'll be saying "GALLERY FURNITURE SUCKS THEY SELL COUCHES THAT DON"T WORK". This is what users, application owners, and infrastructure people do today. Vendors MUST protect their name. I'm not saying these whiners make any sense, but people do this.
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
For the 5 years I consulted "why is this slow" was one of the most common engagements. 9/10 of the time I was chasing some crazy application issue it had nothing to do with the application. Generally it was staring people in the face, had a giant RED alarm, and was fairly obvious (Disk latency isn't supposed to be 1200ms, and NL-SAS drives shouldn't be used for DB's in 5 billion dollar companies Yo?). Assuming that vendors are crossing a line by assuming internal IT doesn't understand what it will take to deliver their applications is CRITICAL to being a successful application vendor. I've seen users, IT and C-suite trash applications that worked fine, but the infrastructure was all wrong....
And that's why external IT consulting was brought in. Not a random application vendor.
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Application owners have RPO/RTO's and they expect the infrastructure people to often take care of that. (When I have a 5TB OLTP database, in guest options generally fail to deliver somehow).
Yup, and if they outside IT to the application vendor, that SLA isn't owned by the actual IT department but by someone who came in, put in something new and ran away. If the application owners need a reliable RPO/RTO, they need to work with IT, not work against them.
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Application owners have RPO/RTO's and they expect the infrastructure people to often take care of that. (When I have a 5TB OLTP database, in guest options generally fail to deliver somehow).
Yup, and if they outside IT to the application vendor, that SLA isn't owned by the actual IT department but by someone who came in, put in something new and ran away. If the application owners need a reliable RPO/RTO, they need to work with IT, not work against them.
Its the reality in most companies. Software vendors requirements are not rooted in how IT SHOULD be run, but how it does. I agree with you in principal (it shouldn't matter) I've just seen hundreds of counter examples that would have destroyed these companies names.
There was thread on SW recently where someone said "NIMBLE SUCKS I DON"T GET THE IOPS I PROMISED". The next post was his Nimble sales rep posting "So I see your at 20% load, your IO latency is .5 ms currently and while your 220C model is one of our smaller ones we have far larger ones. If your having any problems please call us and we will help you" I laughed, but it made me realize the damage that incompetent IT do to the name of a product or application. We are at the point that a sales rep would rather piss off a customer and call them out as an idiot (he was nice about it) than risk their companies name being drug through the mud.
The "IT guy is always" right attitude in IT bothers me. Part of why I always enjoyed arguing with you (and others internally) as its the only way to challenge my idea's and learn and thing of new things. Part of the reason I enjoyed consulting (although I did learn a lot of tact of how to carefully make people think it was their idea, or gently expose why what they were doing was hilariously a bad idea).
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
If I buy a couch or desk that's massive for a tiny apartment I could see the sales guy asking how big my door way is to make sure they can deliver it.
Would you honestly still do business with a furniture store that didn't state the size (that's demanding certain performance, the results not the means) but rather demanded that you buy a certain make or style of door regardless of the fact that the one that you had would have been big enough? because that's the comparison.
Making sure that the SIZE is right I always agreed to. Demanding only doors from certain vendors be used is where the insanity happens. Or forcing you to install a new door because they don't trust your measurements.
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
If I buy a couch or desk that's massive for a tiny apartment I could see the sales guy asking how big my door way is to make sure they can deliver it.
Would you honestly still do business with a furniture store that didn't state the size (that's demanding certain performance, the results not the means) but rather demanded that you buy a certain make or style of door regardless of the fact that the one that you had would have been big enough? because that's the comparison.
Making sure that the SIZE is right I always agreed to. Demanding only doors from certain vendors be used is where the insanity happens. Or forcing you to install a new door because they don't trust your measurements.
I think we've reached stasis here. I've provided examples where the platform matters.
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
Its the reality in most companies. Software vendors requirements are not rooted in how IT SHOULD be run, but how it does.
It's mandated shadow IT. It's a subversive approach. One nice thing for internal IT is that any failing of the system you get to run the vendor through the ringer. But I want products for my customers that are based around them being successful, not assuming their failure. I have very different goals (I want the company to succeed) than the software vendors (they could care less if it works, only that they don't get blamed.)
I don't blame vendors for taking advantage of bad business processes, suckers deserve to be suckered they have no one to blame but themselves, but my point is that good IT would be working on protecting their businesses from these processes and good management would be tasking them to do so.
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
There was thread on SW recently where someone said "NIMBLE SUCKS I DON"T GET THE IOPS I PROMISED". The next post was his Nimble sales rep posting "So I see your at 20% load, your IO latency is .5 ms currently and while your 220C model is one of our smaller ones we have far larger ones. If your having any problems please call us and we will help you" I laughed, but it made me realize the damage that incompetent IT do to the name of a product or application. We are at the point that a sales rep would rather piss off a customer and call them out as an idiot (he was nice about it) than risk their companies name being drug through the mud.
That's not incompetence, though. That's just someone lying. there is a difference.
-
One last thought...
IF the reason that Xen has 2% market share is because there is NO LOGICAL REASON for vSphere or paid Hyper-V (with VMM to manage) then that means 98% of IT people are idiots. If 98% are idiots, wouldn't that mean they should be outsourcing their IT as much as possible to their vendors or others? (and therefore not deploy Xen).
Catch-22
-
@scottalanmiller said in The Inverted Pyramid of Doom Challenge:
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
There was thread on SW recently where someone said "NIMBLE SUCKS I DON"T GET THE IOPS I PROMISED". The next post was his Nimble sales rep posting "So I see your at 20% load, your IO latency is .5 ms currently and while your 220C model is one of our smaller ones we have far larger ones. If your having any problems please call us and we will help you" I laughed, but it made me realize the damage that incompetent IT do to the name of a product or application. We are at the point that a sales rep would rather piss off a customer and call them out as an idiot (he was nice about it) than risk their companies name being drug through the mud.
That's not incompetence, though. That's just someone lying. there is a difference.
I learned years ago to never prescribe malice to what you can attribute to ignorance in this industry. He likely was unhappy the number in his dashboard didn't say 100K!
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
I think we've reached stasis here. I've provided examples where the platform matters.
Okay, I'll buy that. The platform matters when internal IT has failed and you outsourced to an external IT department who has an interest in selling you something that you don't need to make extra money on probably the sale and definitely the consulting. Yes, I agree, but I don't agree that that doesn't match my original point. It's not in the interst of the customer, but there is a reason why they feel that they have to do it based on other decisions made in the same way.
Do you feel, however, that since this discussion is based on scale for the context of the original question, that there is ever a realistic time that this happens at three or fewer compute nodes? We are talking about three nodes for an entire business here. What business, anywhere, is that small and deploying systems where vendors interact with them in this manner? I'm not saying that theoretically it isn't possible, but this thread is asking for an example where this has ever happened.
Outside of pure theory, and even there I feel that it is hard to theorize, who has products that need these kinds of things while being so small as to not have benefits of the IPOD due to scale?
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
I learned years ago to never prescribe malice to what you can attribute to ignorance in this industry. He likely was unhappy the number in his dashboard didn't say 100K!
And likewise, my rule of thumb is that willful ignorance is one of the worst forms of malice
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
One last thought...
IF the reason that Xen has 2% market share is because there is NO LOGICAL REASON for vSphere or paid Hyper-V (with VMM to manage) then that means 98% of IT people are idiots. If 98% are idiots, wouldn't that mean they should be outsourcing their IT as much as possible to their vendors or others? (and therefore not deploy Xen).
Catch-22
Agreed. That makes total sense. And I agree 100%. Almost all (and I totally mean that, something like 98%) of IT should be outsourced. The industry should shrink dramatically, the smaller pool of people who remain should be consolidated into fewer shops and those shops should demand far higher levels of excellence and continued training and raise salaries significantly as the industry tends to lose people who are really valuable because they can often make more money elsewhere and choose to.
However, this is what I'm talking about that assuming bad decisions will be made we then make bad recommendations based on that. It's not valuable to make recommendations and there is no point in doing so, to the 98%. They neither look for nor listen to good advice. Good advice always exists solely for those that will take it. It's the same discussion as "is college valuable for you (in IT.)" If every single person ever listened to that advice, it would be self defeating in weird ways. But they don't, advice around it is for the .1% who might listend and enact change. Good advice remains good advice, that there are reasons why bad decisions are made doesn't make them bad decisions.
I'm never sure if I can explain what I mean here well, but I see it a lot in IT - people give bad advice (could be in any arena, I see it constantly, though) based on the idea that "well they won't listen to my actual good advice." Sure, we know that they won't, but should we make bad recommendations because of that?
-
@John-Nicholson said in The Inverted Pyramid of Doom Challenge:
The "IT guy is always" right attitude in IT bothers me.
Me too, that's why I rarely feel that IT should be allowed to decide things. But IT is the sole critical information holder for a lot of things in the business, performance, cost and security being key ones. No other department has a view into these factors. It is the job of business management to ensure that they have skilled IT that knows its role in the business, to pass decisions through them and to listen to them. And it is IT's job to understand that it is part of the context of the business and only knows certain factors.
That's why an IT veto is important because IT can identify scams, incompetence, lack of industry standards, bad practices and such that other departments would likely not even understand (what do you MEAN VB6 isn't a common thing any longer!!) - because they can filter out options that have no place being considered. Whereas the business needs to make most decisions because they know what is valuable to the business. Neither can act alone. The idea with much of this, though, is that IT would be bypassed and much of the most important IT decisions be made by people who are not IT.
Imagine doing this to HR or accounting. Would other departments demand that HR violate good practices around compensation or reporting? Or that accounting not correctly record expenditures or not pay all taxes? (These do happen, and it's normally very bad.)
-
This is my quote from the original challenge: "We all (I hope by now) know that SANs have their place and a super obvious one that explains why enterprises use them almost universally and know why that usage has no applicability to normal SMBs - scale."
I agree with why lots of shops might deploy systems like you are describing, even if I generally don't agree with that decision, but I'm pretty confident that the use cases that you are describing @John-Nicholson are tied, nearly universally, to a scale that would already prompt a SAN-based infrastructure (or similar.)
Have you seen these in small environments where the scale did not exist to warrant a SAN otherwise?
