Burned by Eschewing Best Practices
-
@dashrender said in Burned by Eschewing Best Practices:
@grey said in Burned by Eschewing Best Practices:
@jaredbusch said in Burned by Eschewing Best Practices:
No, you went all south on that thread.
I don't understand the purpose of this thread. Is it just to document how people are shooting themselves in the foot for some kind of exercise later where you can point to the collated material and say, "look, here, these are all cases of fake IT pros, or salespeople, failing to work in the best interests of [company]. They have all built an inverted pyramid of doom and are paying the price."
900+ pages of people who are wrong on the Internet?
Pretty much.
Is it needed?
What, you don't like making fun of people?
Really the main benefit to this thread in my eyes is that new comers can just browse a few of these pages and see many examples of how doing the wrong thing really costs these people and their businesses.
The fact that this page keeps growing so fast just shows how much of an epidemic it really is.
I understand that it is a fine line, when does documenting why things are best practices become a problem. But let's think about it another way. How often do people use the lack of threads like this as "proof" that SANs don't fail or that IPODs aren't costly or that patches aren't needed. People use the lack of anecdotal evidence as "proof" that statistics and logic aren't real. This thread is a testament to the fact that things we've learned to be bad patterns are really bad patterns and that best practices exist for a reason and that in the real world, skipping them will burn you over and over again, often in ways you might not have predicted.
-
@dashrender said in Burned by Eschewing Best Practices:
@grey said in Burned by Eschewing Best Practices:
@jaredbusch said in Burned by Eschewing Best Practices:
No, you went all south on that thread.
I don't understand the purpose of this thread. Is it just to document how people are shooting themselves in the foot for some kind of exercise later where you can point to the collated material and say, "look, here, these are all cases of fake IT pros, or salespeople, failing to work in the best interests of [company]. They have all built an inverted pyramid of doom and are paying the price."
900+ pages of people who are wrong on the Internet?
Pretty much.
Is it needed?
What, you don't like making fun of people?
Really the main benefit to this thread in my eyes is that new comers can just browse a few of these pages and see many examples of how doing the wrong thing really costs these people and their businesses.
The fact that this page keeps growing so fast just shows how much of an epidemic it really is.
I just don't think it's necessary to make fun of people who may be very sincere in trying to do their best. Even the documentation of the poor decisions or runs in to territory where oit may be less about a learning experience for others and more about collecting things to laugh and troll over.
-
@grey said in Burned by Eschewing Best Practices:
@dashrender said in Burned by Eschewing Best Practices:
@grey said in Burned by Eschewing Best Practices:
@jaredbusch said in Burned by Eschewing Best Practices:
No, you went all south on that thread.
I don't understand the purpose of this thread. Is it just to document how people are shooting themselves in the foot for some kind of exercise later where you can point to the collated material and say, "look, here, these are all cases of fake IT pros, or salespeople, failing to work in the best interests of [company]. They have all built an inverted pyramid of doom and are paying the price."
900+ pages of people who are wrong on the Internet?
Pretty much.
Is it needed?
What, you don't like making fun of people?
Really the main benefit to this thread in my eyes is that new comers can just browse a few of these pages and see many examples of how doing the wrong thing really costs these people and their businesses.
The fact that this page keeps growing so fast just shows how much of an epidemic it really is.
I just don't think it's necessary to make fun of people who may be very sincere in trying to do their best.
Well we aren't making fun. And the point here is that violating best practices is not doing their best. At some point a baseline of professional responsibility is needed. That's why best practices exist. This is not a thread of "things that went wrong" or "poor decisions". This is blatant violations of things that are clearly not okay.
And until people do not use the lack of this thread as "proof" that best practices are false, it's critical that things like this exist.
-
@grey said in Burned by Eschewing Best Practices:
Even the documentation of the poor decisions or runs in to territory where oit may be less about a learning experience for others and more about collecting things to laugh and troll over.
By that logic, posting questions is the same thing. But that's not what it is. We work in an industry where vendors use a lack of evidence to bully customers into bad decisions that favour the vendors. Customers who have made bad decisions do the same thing to try to validate their own bad decisions. People routinely try to do their jobs with zero thought, research or verification. These things need to be called out. Both because we can't bury this stuff, and also because not calling it out is directly used as an excuse to bully others into thinking violating best practices is okay.
-
https://www.itnews.com.au/news/petya-damage-to-tnt-express-systems-is-likely-permanent-468600
International courier TNT Express has warned that it may have permanently lost access to some critical business data and systems following the damaging Petya malware attack.
Its parent company FedEx also today revealed the business had similarly fallen victim to the WannaCry malware just one month earlier.
...
The Petya attack was a heavy blow to a company that had spent the past month grappling with the fallout of the WannaCry ransomware attack. -
@nadnerb said in Burned by Eschewing Best Practices:
https://www.itnews.com.au/news/petya-damage-to-tnt-express-systems-is-likely-permanent-468600
International courier TNT Express has warned that it may have permanently lost access to some critical business data and systems following the damaging Petya malware attack.
Its parent company FedEx also today revealed the business had similarly fallen victim to the WannaCry malware just one month earlier.
...
The Petya attack was a heavy blow to a company that had spent the past month grappling with the fallout of the WannaCry ransomware attack.http://i0.kym-cdn.com/entries/icons/original/000/000/554/facepalm.jpg
-
@nadnerb said in Burned by Eschewing Best Practices:
https://www.itnews.com.au/news/petya-damage-to-tnt-express-systems-is-likely-permanent-468600
International courier TNT Express has warned that it may have permanently lost access to some critical business data and systems following the damaging Petya malware attack.
Its parent company FedEx also today revealed the business had similarly fallen victim to the WannaCry malware just one month earlier.
...
The Petya attack was a heavy blow to a company that had spent the past month grappling with the fallout of the WannaCry ransomware attack.Holy crap!
-
@scottalanmiller @nadnerB @travisdh1 We ship lots of stuff. Occasionally in the past the shipping person here would get fake TNT Express emails notifications telling them to click this link for delivery notifications(just like the fake UPS/Fedex ones), etc. User didnt bite. I checked them out on mxtoolbox after that, no spf. Not surprising their computers werent updated either.
-
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
-
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
-
@dustinb3403 said in Burned by Eschewing Best Practices:
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
Hypervisor on an SD card isn't an option as these aren't virtualized servers. That battle that on my list to wage.
As far as the SSDs, when I can wage and win the virtualization battle, that will likely be the path I go, as I can put the HDDs to use elsewhere. At the time, doing what I did was the best option I had.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
Hypervisor on an SD card isn't an option as these aren't virtualized servers. That battle that on my list to wage.
As far as the SSDs, when I can wage and win the virtualization battle, that will likely be the path I go, as I can put the HDDs to use elsewhere. At the time, doing what I did was the best option I had.
@EddieJennings decision makers:
https://i.imgflip.com/1d7bh7.jpg -
@travisdh1 I'm not 100% without blame, as I could've probably pushed hard for virtualization at the time the RAID 5 was dying. In retrospect I should've, but I chose the path where I knew I could mitigate the problem and not have pushback.
As far as the other server with a split array (Two SSDs in RAID 1 and Four in RAID 10), that decision was made before I both had the power to actually influence decisions and gained some wisdom about how storage is supposed to be designed.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@travisdh1 I'm not 100% without blame, as I could've probably pushed hard for virtualization at the time the RAID 5 was dying. In retrospect I should've, but I chose the path where I knew I could mitigate the problem and not have pushback.
As far as the other server with a split array (Two SSDs in RAID 1 and Four in RAID 10), that decision was made before I both had the power to actually influence decisions and gained some wisdom about how storage is supposed to be designed.
And the difference between management and you, is that you know better now and will fix the issues when the time comes to replace equipment. Question is, will management get stuck in and prevent you from doing things right?
-
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
-
@dafyre I've heard the same from others. The things that are virtualized here use Hyper-V, and likely that's what I'll use when the aforementioned servers are virtualized.
-
@dafyre said in Burned by Eschewing Best Practices:
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
Agreed
-
@dafyre said in Burned by Eschewing Best Practices:
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
The only reason the other hypervisors don't do this any more is because they've failed at explaining how to protect the boot environment for the hypervisor.
Hyper-V once recommended installing to the cheapest storage, so long as you had a good working backup.
You can do the same with XS, but you need to customize the installation so none of the logging goes to the boot device. Which adds a lot of complexity. -
@dustinb3403 said in Burned by Eschewing Best Practices:
Which adds a lot of complexity.
Which is what we are trying to avoid.
