Burned by Eschewing Best Practices
-
I felt that we should grab all of that as the OP was threatening to delete it as it exposed his boss the CTO and his relationships with resellers; but it was such a good example of how one senior manager trying to get away without doing his job can sabotage a company.
-
We all totally understand that YOU are not the one at fault here. You have a boss that doesn't know what he is doing (or worse, that's the really scary part) and you are new to the company and IT in general. But you are in a position now to do something about it. Is it worth making waves? It is worth exposing what we assume must be a form of corruption (pretending to be a CTO level and not even being a viable entry level tech and getting paid to screw the company is a form of corruption here and appears to be the best case scenario?) Only you can know that. But I've worked in financials where, if I was in your position, I would be required by my social contract with the company to take this information to the CEO. That does NOT mean that you make claims of corruption, theft, ineptitude or impropriety. That's not how it works. YOU take this thread, print it out and say "look Mr. CEO, I had this conversation and a casual oversight of our organization left a large number of IT pros looking at us as having some very visible problems ranging from technical incompetence to outright fraud and inappropriate / unethical reliance on sales people at vendors to do the work that we are tasked with doing internally to protect the company. I'm new to IT and I'm new here, but they thought that I should pass this on to you, unfiltered, so that you could decide if they were right, wrong or if you even care either way. My job is only to bring it to your attention, not to pass judgment on it. So here it is, do with it what you will."That's all that YOU need to do. Then, whatever. but use this as a learning experience for yourself. You need to KNOW that your CTO isn't qualified to work for me as an intern and if I was a CEO (and I am) he's be fired and I'd seriously consider legal charges based off of the things you've said here. From a purely business, non-technical standpoint, he should be fired for the relationships he's build on impropriety. On a technical level, he should be fired for having allowed such obvious abuses of the company that anyone even moderately technical should have seen. No matter what his technical OR business levels, he should have protected the firm from what has happened thus far. And this is likely just the tip of the iceberg. Every little thing you've mentioned here is atrocious. Imagine what less obvious things are going on if this is what they were willing to do with full visibility!
-
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
-
I'm still trying to determine if this is a physical installation or not, but he is being burned because he doesn't provide relevant information to resolve the issue.
I'm also guessing at the size, he doesn't have a second domain controller.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
WHAT? Sure it's good to have 3 copies of your data, but having only 2 is acceptable.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
WHAT? Sure it's good to have 3 copies of your data, but having only 2 is acceptable.
No, 3-2-1
Production system, onsite backup, offsite backup.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
10/100 here
-
The bigger take away here. This was a backup system that was on a physical server not on a hypervisor. So he couldn't take a snap of the system prior to upgrading it. Always virtualize your infrastructure.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
This - so much this!
Is he doing the best option available - no, but not everyone can.
The fact that his Backup Server was not virtualized doesn't even play into his problem, most likely. He updated the hardware and was bricked. That's just as likely to happen to a hypervisor as it is a normal OS installed directly on the hardware.
Having the offsite copy of the data wouldn't really change his situation in any way, other than at this moment, if his main system dies, well he has no backup.
So I'm still asking where he was burned by not following best practice? His main system is still up and running, right? So he hasn't been burned at all. He had a failure, and he worked to solve that failure. Sounds like a normal day in IT.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
Again - I haven't looked at the thread - but it seems to me that you're saying that the OP is indicating that he's not going to fix his backups? So what is he going to do instead?
And what does having or not having a working backup system have anything to do with a second backup (typically offsite).
-
@coliver said in Burned by Eschewing Best Practices:
I know the 3-2-1 method is preferred and best practice.
3-2-1 is a rule of thumb, definitely not a best practice.
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
Again - I haven't looked at the thread - but it seems to me that you're saying that the OP is indicating that he's not going to fix his backups? So what is he going to do instead?
And what does having or not having a working backup system have anything to do with a second backup (typically offsite).
The issue is the concern that is backup system is broken, he cannot create a new backup while this is being restored.
The backup system is running on a physical windows server (which could easily have been virtualized for the same effect). Which then would've allowed him to avoid the issues of drivers breaking his on-site backup device.
He was able to finally get the system restored after many hours of working with Dell.
The secondary backup offsite is likely fed from the primary onsite backup. Meaning that any new backups, still would not function.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
He definitely got burned, but I think it was more just... got burned for not knowing what he was doing. Less any specific practice. He didn't keep his systems updated, he didn't call proper support, he didn't know where to turn before things were needed, he didn't have virtualization, his systems are ancient, he's running BackupExec, he didn't take any backups of the backup configs... it all came together to bite him. But at any stage, had he just known what to do, he'd have been okay.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
The issue is the concern that is backup system is broken, he cannot create a new backup while this is being restored.
Right, but they COULD have a working one quickly any time that they want one, they just are choosing not to. That's the important part there. They are choosing not to take, say, manual backups right now. It's not that they can't back things up.
