Was It the Last IT Guys Fault
-
Common story, working in IT in the SMB almost always means taking over the reigns from an IT guy or MSP that was there and is no more. Often there is a gap of a day or a year. It's the weird nature of IT in the SMB that there is almost never a handover period, no transition process.
This makes for an interesting and problematic culture in IT. Invariably we spend our careers managing systems that we did not design, did not choose, have to answer for but had no influence in selecting and we often lack documentation, tribal knowledge and even insight into what decisions led to where the business is when we take over. Often systems are only half implemented or some degree of "in process changes" exist and we have to determine if they were halted because someone left the company or because they were completed or they were abandoned!
This process of constantly taking over for someone that was fired, quit, or just disappeared (happens more than you think) creates a culture of all problems instantly being blamed on "the old IT guy / firm." Why? Because, why not? They were responsible for making the mess, lacking the documentation, not sticking around or whatever. This was "their network" and it has problems, who else is there to blame? The constantly changing IT position(s) in companies makes it feel natural to just expect that the last guy did a bad job and we have to fix it. So much so that it is almost expected and it has become cultural to point fingers at the last guy by everyone. It's the excuse for the technical debt that stops the current IT people from doing a good job and it keeps everyone from having to assign blame to the management that is still around. No need for management to accept any blame as there can be no proof of their involvement and the new IT guy or firm has no basis for suspecting them. Even if management is to blame, the new guy won't take the bold step of assigning blame to their new bosses and the bosses have no reason to not save face when there is a nameless person no longer around as the perfect scape goat.
Having worked in SMB IT in both the on staff as well as a consulting role, and many years as a service provider, I've seen this countless times and nearly every first day conversation with a new job or customer is going to be about how many the last guy or firm was, what they screwed up, how they didn't finish anything and so forth. There is always much complaining. The customer or employer always has so many issues to expose and so much blame to lay. And often, from the IT person's side, looking at the system there are so many unanswerable questions: "Why are there no backups?", "What do you mean that the RAID array has been degraded for two years?", "How do you not have any passwords to these systems?", "Why are there three email servers here but email is hosted with Google?"
But in the majority of cases, as the weeks or months or maybe years pass, it starts to become apparent, that budgets aren't approved, management is taking over IT decision roles, projects get canceled halfway through and so forth. Pretty soon you start to realize "I'm building the same awful problems that they blamed on the last guy!" It's not you, and it wasn't him... it's management. This is how the company runs! The last guy wasn't fired, he quit. The last MSP wasn't replaced, they stopped service from the client being unable to pay! The truth starts to become obvious.
Are there exceptions? Of course. And we know pretty reliably that the average IT Pro isn't very good and the average MSP isn't very good and there really are terrible situations being left behind all over the place and good reasons to place some or a lot of blame on people building houses of cards for others to deal with. But this situation is nearly impossible if management is hiring, managing and auditing well. Management is always at the core of IT, even when they are hands off to IT management (as they should be.) They are the core of the business and influence who is IT, and how they are treated.
Is there a simple answer here? No, but understanding this culture in IT and how it happens and that it does happen is important. Knowing how management has a "nothing to lose" stance shows us why this will just keep happening as long as SMB IT has high turn over. There is just always someone to blame. It's too convenient.
As IT Pros, we can work to be a little less apt to throw someone under the bus when we don't now why they did what they did. Maybe they were following orders. Maybe they got caught with changing demands or requirements. Maybe they were forced to work on something that they didn't understand. Maybe they were not approved to update needed systems. We just don't know. But by looking at the forensics of old problems and looking at current management practice, perhaps we can start to piece together a coherent story that suggests that the management making problems today is the common thread with the problems of yesterday
-
I have had a very similar experience with SMB companies as well. It's always the last guy's fault, but then you realize it's really the company's fault. Either they don't have enough of a budget, the pay sucks, or you're working 60-70 weeks just to keep your head above water. Most likely all 3.
-
@scottalanmiller
I got to say that sometime your post is just too long to read. Thanks to translate.google with voice over text.... I'm able to follow through.. -
@scottalanmiller said in Was It the Last IT Guys Fault:
"Why are there no backups?", "What do you mean that the RAID array has been degraded for two years?", "How do you not have any passwords to these systems?", "Why are there three email servers here but email is hosted with Google
I laughed so much cause of those.
And yes I can totally relate, I start working with group of folks that got me after they purchased the server with 3 disks so RAID 5 it is, I dont like it I wish I was more involved in the process and had more time to order disk, however since it was SAS disk, and harder to find where I live, we went live with 3 disks and RAID 5... And i dont like it one bit, I wish it was 4 disks and RAID 10.
And now after we have gone in production I dont dare to touch or change that array, now when I leave this will be on me for sure, but its not my fault.
-
@stess said in Was It the Last IT Guys Fault:
@scottalanmiller
I got to say that sometime your post is just too long to read. Thanks to translate.google with voice over text.... I'm able to follow through..You are listening to it translated and read? In what language? How well does that work?
-
This post is deleted! -
This is one of my favorite posts you have ever written because I have lived it. You are so right. Rather than the blame game of putting it on the last person there, we really should be questioning management and their decision making.
-
What people don't realize when they are considering a new job is that you're interviewing management too. You just have to learn to ask the right questions in the right way before you can really decide if management will end up being the problem. And I don't know that you can fully grasp the full picture until you get into the day-to-day of what the role is like.
-
@NetworkNerd said in Was It the Last IT Guys Fault:
What people don't realize when they are considering a new job is that you're interviewing management too. You just have to learn to ask the right questions in the right way before you can really decide if management will end up being the problem. And I don't know that you can fully grasp the full picture until you get into the day-to-day of what the role is like.
http://www.smbitjournal.com/2011/06/hiring-it-the-reverse-interview/
-
As of right now I have lots of stuff undone. We just migrated ecommerce stores, there are a lot of procedures and how-to docs now out of date. Lots of reports done in Excel that use scripts will no longer work with the CSV exports of the new system, they need updated.
Web tools I programmed to work with old data, all useless now and need to be rewritten.
I have an in-house server and Drobo not fully configured and automated with what I wanted to do with them.
Backup software is not up to date on all workstations.
Lots of passwords not copied over from my LastPass into KeyPass which is easier to hand over to management or the new guy.
A local MySQL DB full of data that needs moved to our new in house server but haven't got around to it, as well as local WAMP internal projects.
Lots of research docs trying to find good tools for ticketing, WIKI or KB management, file sharing tools, web services for our online properties, time tracking, calendaring, two dozen other research projects. My todo app probably has 80 entries, some long term projects, some quick things, some on the backburner.The point is, at not time are we 100% just "done" and everything is smooth. If I were to leave now and a new person step in my place, they would experience what you wrote. Everything would seem undone, slapshod, incomplete, or whatever. But I have four points to think about:
-
Everything has a reason.
Every single Word or Excel file stored in every folder, every installed program, every web services signed up for, every bit of research that was started and not finished, had an original purpose and reason.
We shouldn't make the mistake of thinking the last guy just did random things for no reason. I don't create files or sign up for things or begin configuring tools if there wasn't an original purpose or request. But like many things, management perhaps decided to go another direction, or it became a non-priority, etc.
I personally don't like to delete things I've started, "just in case" I ever need to return to it again. Thus lots of things that seem half-done. -
Not everything can have perfect documentation.
Frankly, I personally know why any given thing is the way it is. But I cannot spend my day thinking "how should I document this somewhere so if I'm replaced tomorrow, the next person won't be confused?" It never happens.
I don't have the time to spend making sure at all times I can be easily replaced and the next person won't have to figure anything out on their own. Documentation takes so much friggin time, and you have to balance how terse you can write about things. Am I explaining it to non-tech people? Or can I do quick summaries that only another nerd will understand?
There are 4 laptops sitting on my shelf, none have perfect documentation. But I personally know who owned them and what we might use them for and what is not working well on them etc. But I don't have the time to write a dissertation about every piece of gear scattered around the office. Or every software installed, and every service I have an account for.
I document important things, procedures and information on our most important critical services and accounts, but I don't document why I installed every piece of software I do, and what every single file was created for. -
Some "useless" stuff is done by habit, or at some upper commander's request
I know for a fact I waste time doing some stuff that doesn't matter. Collecting data that nobody will look at or act upon, but management just KNOWS they need it. I have a file drawer full of paper reports and data because they don't trust computer files.
We do a lot of stuff because someone thought it was important, or because we do it by habit. I record how many black and white copies and color copies we make on the copier, because the boss keeps telling me it's important information. But they never look at the file or want to see the data or do anything with it. They just think it's really important.
There is a honeymoon period when a new tech is hired. The new tech gets to make all kinds of suggestions and buy new stuff and declare what is useful and what isn't, and the boss will listen for a while.
The old tech, however, loses their influence over time and the boss can easily reject any suggestions, leaving us to continue in our old habits where they feel comfortable, not wanting to make big changes. -
Sloppy work may not be incompetence, just circumstances.
Not enough time to do something "right". Not enough money to buy the "right" thing. Not enough time given by the boss to do something big so we have to compromise to get it done. "Quick and dirty and cheap" often wins out over "industry best practices". Often the ROI of doing something right versus just getting it done now, is not good enough.
We don't need a better product image, we don't need it to look that nice, we don't need it to be that fast, we don't need it for very long so security isn't a huge deal. Whatever the case may be. I don't want that better software, because I once used this old decrepit version in college 22 years ago. Don't buy a new thing, this old junker is working "just fine".
The list goes on.
It's been said, IT doesn't own the business, the boss does, so we can only make suggestions, but they ultimately sink the ship. If they want to take risks, if they want to keep running on bald tires, they want to keep using the software they bought in 1998 cause they paid "good money for it" and it "still works", then that is their choice.
Just because there is always a better way to do something, doesn't mean the tech was incompetent. Sometimes we have to use duct tape and super glue just like anybody else. We work with the resources we have.
Be nice to your predecessors, they paved the way for you to make the next set of improvements at the company.
-
-
One point here that people seem to forget about blaming the last IT guy is that he may have just been that bad. Management may have made decisions based on the input from the person they trusted to give them reliable information to make IT/business decisions(the last IT Guy), and wouldnt know any better if the information they are given is bad or not, due to them being managers and not technical people.
They give incomplete or incorrect information to decision makers. Then they implement the decision makers edicts. Then things break. They keep breaking. They break because the initial information was wrong. The foundation for their entire infrastructure/email/IT project, wrong. IT guy doesnt know what to do(due to incompetence), flails around, quits or gets fired after a year. Repeat that cycle for a few years in a row, and that is the typical SMB IT environment.
-
@momurda said in Was It the Last IT Guys Fault:
One point here that people seem to forget about blaming the last IT guy is that he may have just been that bad.
That's possible, and certainly happens a lot. But the knee jerk blaming of a guy that isn't there to explain why he did crazy things is a problem because all of us are that guy and generally have good explanations for it. It's rare in the SMB that the "IT guy" is also the person making IT decisions, it's not even the norm. The person who gets blamed is very rarely the person actually in charge.
-
@momurda said in Was It the Last IT Guys Fault:
Management may have made decisions based on the input from the person they trusted to give them reliable information to make IT/business decisions(the last IT Guy), and wouldnt know any better if the information they are given is bad or not, due to them being managers and not technical people.
Then a critical question would be, why was management making any decision? That's why I point out the "who is the head of IT" problem. If someone who doesn't understand what they are doing is making decisions rather than entrusting the person who is tasked with understanding, that itself is an issue - either that the wrong person was hired or they are not empowered to do the job that needs to be done.
Sure, in many cases your example is true. But it's also very common to come in, be told that the last guy did crazy things, then immediately find out that the actual "last guy" is still there, as the boss.
-
@momurda said in Was It the Last IT Guys Fault:
Repeat that cycle for a few years in a row, and that is the typical SMB IT environment.
I totally agree. Now, no hiring process is perfect, not even close. But who bears all responsibility for who is hired? If a failed process is being repeated over and over and the "IT guy" is the piece getting replaced, it's a pretty good assumption that the IT guy isn't the problem. They might be a problem, but not the root one.
-
I wonder if I'd fall into this category... hopefully not. I document everything I do, and hand it out to the team members.
Leaving or getting hit by a bus, I wouldn't want to be in those shoes or leave someone in those shoes saying "WTF was he thinking. . . "
-
@scottalanmiller said in Was It the Last IT Guys Fault:
@stess said in Was It the Last IT Guys Fault:
@scottalanmiller
I got to say that sometime your post is just too long to read. Thanks to translate.google with voice over text.... I'm able to follow through..You are listening to it translated and read? In what language? How well does that work?
English. There's a speaker button that read what you typed (in this case copy/paste) into the box. It's not perfect, but for someone whos English is not their primary language, it helps a lot.
-
Probably the main thing that puts me off moving jobs is that it means moving in to someone else's shit, which you then have to spend months, or even years, sorting out.
It is like moving house, only where you have to leave all your own, cool stuff in your old house and have to make do with whatever the previous tenants of your new house have left you. Even if they're relatively neat and tidy, you'll still be like "uugh, my new closet is full of brown cords, I miss my old blue jeans". And that's the best case. Worst case is they've left a half-eaten, three week old pizza under the sofa.
-
@stess said in Was It the Last IT Guys Fault:
@scottalanmiller said in Was It the Last IT Guys Fault:
@stess said in Was It the Last IT Guys Fault:
@scottalanmiller
I got to say that sometime your post is just too long to read. Thanks to translate.google with voice over text.... I'm able to follow through..You are listening to it translated and read? In what language? How well does that work?
English. There's a speaker button that read what you typed (in this case copy/paste) into the box. It's not perfect, but for someone whos English is not their primary language, it helps a lot.
Oh, so you are hearing it in English, but listening is easier than reading. I get it now.
-
@Carnival-Boy said in Was It the Last IT Guys Fault:
Probably the main thing that puts me off moving jobs is that it means moving in to someone else's shit, which you then have to spend months, or even years, sorting out.
It is like moving house, only where you have to leave all your own, cool stuff in your old house and have to make do with whatever the previous tenants of your new house have left you. Even if they're relatively neat and tidy, you'll still be like "uugh, my new closet is full of brown cords, I miss my old blue jeans". And that's the best case. Worst case is they've left a half-eaten, three week old pizza under the sofa.
I came into role that was half documented, with everything all over the places. Took me 2 months just to grasp what is going on, and another 6 months to sort things just to keep it afloat.
-
Was it the last IT guy's fault? You can be the judge.
When I first started with my current company, I was suppose to meet the last IT guy so he can show me things, it never happened, he never showed up! After a day and a half of waiting, my boss gave me the last IT guy's credential and I logged into the IT computer, I found he had his google account logged in, I can see his "google hangouts" buddies, I can see his gmails, I can see he was chatting real time with his buddies and when he was going to start playing games with them. I reported the finding to my boss and let it be. When I looked for documents, they were 7 months behind, just so happen the last IT guy was here 7 months.
During the first week, three physical servers froze, two were DC servers. During the second week, virtual email server ran out of disk space, I had to come in on a Saturday to fix it. Even though the email server is a virtual server, I found the email data volume is spanned across three drives. I think my heart skipped a beat or 10. I had to add another drives to the spanned volume just to get the email server working. I fixed the spanned volume the week after, by enlarging the main drive, defragging the volume and removing the additional drives.
VMware vSphere Essentials Plus Kit, (over $5,000) was purchased, yet, the company servers are only setup to use VMware vSphere Essentials Kits (slightly over $500) worth of functions (backup API and vCenter). Company could have chosen HyperV and saved over $5,000 and will not make a difference in daily operations! Only two ESXi virtual server were setup. vCenter was also installed on a physical server running windows server, I replace it with a vCenter appliance running on one of the the ESXi host and saved a server license.
In the third week, I proposed and got approval to buy a few used HP DL380G6 and use them as virtual server host, buy more RAM, and new 2TB harddrives to refresh the current virtual host servers as well as converting all physical servers to virtual. DL380G6 was $50 each, RAM and harddrives were the main expense, total was maybe $5,000. A week or two after I have got three HP DL380G6 servers with new harddrives and RAM hosting the virtual servers and running happily and couple spare HP DL380G6 shells/RAM/harddrives for DR.
At one point of time, the physical windows file server was running out of space, but instead of buying more harddrives for the server, a decision was made to buy QNAP NAS, set to RAID 5, established an iSCSI connection to the server, then the server shared out the files. Result was a unstable file server that would forget permission of files when the server is restarted. There was an old HP server on the IT desk, the document left behind by two IT guy ago said this would make a good file server, but he didn't think he has time to setup up the server. During the first week there, I plug the kvm and power to that server, server will not even boot!!
Two months in, the physical windows file server finally crashed. File system was in a read only mode and would not allow any write until the CHKDSK was done. I saw this as an opportunity to get rid of the physical server and QNAP at the same time. I gave my boss two options: I can attempt to repair the current physical file server or I can spend time and energy to build a new virtual file server and then copy everything over. Boss chose the latter and I had the new virtual server up and sharing its files by the next morning. I set it up so quickly, I wondered why the two IT guy ago even bother to write the document stating he didn't have the time.
So, by the start of the third month, I had all virtual servers on fresh harddrives running RAID10, I have three virtual hosts and no other physical servers, I had cold server shells with RAM and harddrives for DR, no more QNAP running RAID5, plenty of disk space for all the virtual servers, all virtual servers were backed up, no more frozen servers since the virtual conversion and I can sleep better.
