Burned by Eschewing Best Practices
-
Scott Alan Miller May 27, 2017 at 9:44 AM
Niagara Technology Group (NTG) is an IT service provider.
cavemanager wrote: omfg.. the people I work with...That's why you might want to take this thread to the CEO. I'm serious, engage someone at the top. Express REAL concerns. This isn't just a question of technical stuff or making money, this is a question of ethics. As someone that audits companies for this kind of stuff, these are the signs that you have someone in a senior position that is not doing their job and hiding behind sales people. It's a common tactic to make big salaries while not doing the job that they are hired to do. I'm talking about your CIO most likely. Someone is selling out the company and likely making a pretty penny doing it.
That doesn't imply that there are vendor kickbacks, but once someone is stealing their salary from the company, what's a kickback or two?
-
Scott Alan Miller
This is an industry standard problem in the SMB space and you'll see it all over this community, too. People who hide behind IT titles but don't actually do any IT work or act as the IT person / role but instead of "IT buyers" and just "buy IT" from vendors. But since they don't act in an IT role, they don't play the part of protecting their companies from predatory sales people and since "sales is free" and "expertise is expensive" to avoid budget concerns they go to sales people and get the "IT for free" and instead just get sold products that they claim that they need.
It's a really effective way to get paid as an IT pro, but do, quite literally, nothing. No need for any skills at all, no need to keep up, no need to even work. Sales people are always happy to do this "work" for free - but of course the only thing that they do is sell you things that you don't need because that's their actual job that they are paid to do. One hour of IT consulting would have protected your institution from all of these issues, that's all it takes. But lacking ALL IT oversight, someone just let a sales organization into the company to "rape and pillage" at will.
Think of IT like the castle guards. They are paid to protect the business. But in a case like this, it looks like the senior guard either decided doing his job was too much effort or got slipped some gold in a bag and literally opened the gates to let the known attackers in to have at the undefended castle. But the guards keep reporting to the king that all is well and the people stealing are not to be worried about.
Does that make sense? At some point, someone is making money and the IT protection isn't happening. These are literally the textbook examples of how this happens. If you look at SW, there are literally thousands (I really mean that) threads about how this exact scenario should never happen, and how VARs and vendors will take advantage of it, and how Dell does this specifically, etc. It could not fit the predicted scenario more. And the articles I linked, they are years old but fit your scenario exactly. There is a reason for that. This is the pattern that we look for for these kinds of problems.
-
I felt that we should grab all of that as the OP was threatening to delete it as it exposed his boss the CTO and his relationships with resellers; but it was such a good example of how one senior manager trying to get away without doing his job can sabotage a company.
-
We all totally understand that YOU are not the one at fault here. You have a boss that doesn't know what he is doing (or worse, that's the really scary part) and you are new to the company and IT in general. But you are in a position now to do something about it. Is it worth making waves? It is worth exposing what we assume must be a form of corruption (pretending to be a CTO level and not even being a viable entry level tech and getting paid to screw the company is a form of corruption here and appears to be the best case scenario?) Only you can know that. But I've worked in financials where, if I was in your position, I would be required by my social contract with the company to take this information to the CEO. That does NOT mean that you make claims of corruption, theft, ineptitude or impropriety. That's not how it works. YOU take this thread, print it out and say "look Mr. CEO, I had this conversation and a casual oversight of our organization left a large number of IT pros looking at us as having some very visible problems ranging from technical incompetence to outright fraud and inappropriate / unethical reliance on sales people at vendors to do the work that we are tasked with doing internally to protect the company. I'm new to IT and I'm new here, but they thought that I should pass this on to you, unfiltered, so that you could decide if they were right, wrong or if you even care either way. My job is only to bring it to your attention, not to pass judgment on it. So here it is, do with it what you will."That's all that YOU need to do. Then, whatever. but use this as a learning experience for yourself. You need to KNOW that your CTO isn't qualified to work for me as an intern and if I was a CEO (and I am) he's be fired and I'd seriously consider legal charges based off of the things you've said here. From a purely business, non-technical standpoint, he should be fired for the relationships he's build on impropriety. On a technical level, he should be fired for having allowed such obvious abuses of the company that anyone even moderately technical should have seen. No matter what his technical OR business levels, he should have protected the firm from what has happened thus far. And this is likely just the tip of the iceberg. Every little thing you've mentioned here is atrocious. Imagine what less obvious things are going on if this is what they were willing to do with full visibility!
-
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
-
I'm still trying to determine if this is a physical installation or not, but he is being burned because he doesn't provide relevant information to resolve the issue.
I'm also guessing at the size, he doesn't have a second domain controller.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
WHAT? Sure it's good to have 3 copies of your data, but having only 2 is acceptable.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
WHAT? Sure it's good to have 3 copies of your data, but having only 2 is acceptable.
No, 3-2-1
Production system, onsite backup, offsite backup.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
10/100 here
-
The bigger take away here. This was a backup system that was on a physical server not on a hypervisor. So he couldn't take a snap of the system prior to upgrading it. Always virtualize your infrastructure.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
-
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
This - so much this!
Is he doing the best option available - no, but not everyone can.
The fact that his Backup Server was not virtualized doesn't even play into his problem, most likely. He updated the hardware and was bricked. That's just as likely to happen to a hypervisor as it is a normal OS installed directly on the hardware.
Having the offsite copy of the data wouldn't really change his situation in any way, other than at this moment, if his main system dies, well he has no backup.
So I'm still asking where he was burned by not following best practice? His main system is still up and running, right? So he hasn't been burned at all. He had a failure, and he worked to solve that failure. Sounds like a normal day in IT.
-
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
Again - I haven't looked at the thread - but it seems to me that you're saying that the OP is indicating that he's not going to fix his backups? So what is he going to do instead?
And what does having or not having a working backup system have anything to do with a second backup (typically offsite).
-
@coliver said in Burned by Eschewing Best Practices:
I know the 3-2-1 method is preferred and best practice.
3-2-1 is a rule of thumb, definitely not a best practice.
-
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@coliver said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
@Dashrender said in Burned by Eschewing Best Practices:
@DustinB3403 said in Burned by Eschewing Best Practices:
Has a physical backup server, that after being updated with new firmware and drivers was bricked. OP was able to get the system restored and functional after several hours working with Dell.
This write-up doesn't indicate what whas Eschewed here?
Having a backup system that is physical in nature, without have another backup.
He had no backup for his backup, blindly updated his only backup system, and bricked it. So it does indicate what was eschewed. He failed to backup his backup.
I know the 3-2-1 method is preferred and best practice. But bricking his backup system doesn't really matter. It's a backup of a production system. Restore the software and take fresh backups of your production environment.
That is what I recommended, but the complaint was it woudl take forever (IIRC . . )
It takes to long to make backups? Wut?
The concern was that it would take to long to get the backups operational again. So yea... "takes too long to restore". Which if he lost something in that downtime when backups weren't running, then he'd be SOL.
Again - I haven't looked at the thread - but it seems to me that you're saying that the OP is indicating that he's not going to fix his backups? So what is he going to do instead?
And what does having or not having a working backup system have anything to do with a second backup (typically offsite).
The issue is the concern that is backup system is broken, he cannot create a new backup while this is being restored.
The backup system is running on a physical windows server (which could easily have been virtualized for the same effect). Which then would've allowed him to avoid the issues of drivers breaking his on-site backup device.
He was able to finally get the system restored after many hours of working with Dell.
The secondary backup offsite is likely fed from the primary onsite backup. Meaning that any new backups, still would not function.
