Burned by Eschewing Best Practices
-
@nadnerb said in Burned by Eschewing Best Practices:
https://www.itnews.com.au/news/petya-damage-to-tnt-express-systems-is-likely-permanent-468600
International courier TNT Express has warned that it may have permanently lost access to some critical business data and systems following the damaging Petya malware attack.
Ā
Its parent company FedEx also today revealed the business had similarly fallen victim to the WannaCry malware just one month earlier.
...
The Petya attack was a heavy blow to a company that had spent the past month grappling with the fallout of the WannaCry ransomware attack.Holy crap!
-
@scottalanmiller @nadnerB @travisdh1 We ship lots of stuff. Occasionally in the past the shipping person here would get fake TNT Express emails notifications telling them to click this link for delivery notifications(just like the fake UPS/Fedex ones), etc. User didnt bite. I checked them out on mxtoolbox after that, no spf. Not surprising their computers werent updated either.
-
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
-
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
-
@dustinb3403 said in Burned by Eschewing Best Practices:
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
Hypervisor on an SD card isn't an option as these aren't virtualized servers. That battle that on my list to wage.
As far as the SSDs, when I can wage and win the virtualization battle, that will likely be the path I go, as I can put the HDDs to use elsewhere. At the time, doing what I did was the best option I had.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
@eddiejennings said in Burned by Eschewing Best Practices:
@dustinb3403 said in Burned by Eschewing Best Practices:
Appears to be ignoring all best practice by splitting arrays, and using RAID5 (unknown if its on spinning rust or ssd)
Not installing ESXi onto an SD card, but instead using a RAID1.
Alas, two of our production boxes have split arrays
One of the arrays was my doing.There was no way to convince those who needed convincing to purchase enough SSD storage to have OBR10 on the machine that was [formerly] using a Raid 5 on spinning disks attached via iSCSI for storage. Thus, I opted for the lesser of two evils. Two SSDs in Raid 1 (original config). Four HDDs in RAID 10.
Um.... you should have set it up as OBR5 if you have SSDs and installed the hypervisor to an SD card.
Hypervisor on an SD card isn't an option as these aren't virtualized servers. That battle that on my list to wage.
As far as the SSDs, when I can wage and win the virtualization battle, that will likely be the path I go, as I can put the HDDs to use elsewhere. At the time, doing what I did was the best option I had.
@EddieJennings decision makers:
https://i.imgflip.com/1d7bh7.jpg -
@travisdh1 I'm not 100% without blame, as I could've probably pushed hard for virtualization at the time the RAID 5 was dying. In retrospect I should've, but I chose the path where I knew I could mitigate the problem and not have pushback.
As far as the other server with a split array (Two SSDs in RAID 1 and Four in RAID 10), that decision was made before I both had the power to actually influence decisions and gained some wisdom about how storage is supposed to be designed.
-
@eddiejennings said in Burned by Eschewing Best Practices:
@travisdh1 I'm not 100% without blame, as I could've probably pushed hard for virtualization at the time the RAID 5 was dying. In retrospect I should've, but I chose the path where I knew I could mitigate the problem and not have pushback.
As far as the other server with a split array (Two SSDs in RAID 1 and Four in RAID 10), that decision was made before I both had the power to actually influence decisions and gained some wisdom about how storage is supposed to be designed.
And the difference between management and you, is that you know better now and will fix the issues when the time comes to replace equipment. Question is, will management get stuck in and prevent you from doing things right?
-
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
-
@dafyre I've heard the same from others. The things that are virtualized here use Hyper-V, and likely that's what I'll use when the aforementioned servers are virtualized.
-
@dafyre said in Burned by Eschewing Best Practices:
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
Agreed
-
@dafyre said in Burned by Eschewing Best Practices:
The only Hypervisor I trust enough to install to SD cards right now is ESXi. If you're using anything else, just save yourself the headaches and install it in OBR.
The only reason the other hypervisors don't do this any more is because they've failed at explaining how to protect the boot environment for the hypervisor.
Hyper-V once recommended installing to the cheapest storage, so long as you had a good working backup.
You can do the same with XS, but you need to customize the installation so none of the logging goes to the boot device. Which adds a lot of complexity. -
@dustinb3403 said in Burned by Eschewing Best Practices:
Which adds a lot of complexity.
Which is what we are trying to avoid.
-
Someone needs to seriously automate this.
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Someone needs to seriously automate this.
Don't you already have a SAM-bot? Hopefully you're doing better than Microsoft's last 2 iterations.
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Someone needs to seriously automate this.
It amazes me that XS hasn't solve this issue already!!!!!!
-
@dashrender said in Burned by Eschewing Best Practices:
@scottalanmiller said in Burned by Eschewing Best Practices:
Someone needs to seriously automate this.
It amazes me that XS hasn't solve this issue already!!!!!!
XS doesn't care, it is all focused on VDI. XCP is where any innovation like this might happen, but isn't going to.
It's more likely that Suse would solve this in the Xen space.
-
@scottalanmiller said in Burned by Eschewing Best Practices:
Someone needs to seriously automate this.
Automate what?
-
@dustinb3403 said in Burned by Eschewing Best Practices:
@scottalanmiller said in Burned by Eschewing Best Practices:
Someone needs to seriously automate this.
Automate what?
The live setup process to make other hypervisors trivial to install in a live way.
