Junior Dev destroys PROD DB on first day.
-
@Kelly said in Junior Dev destroys PROD DB on first day.:
The levels of organizational failure there are incredible.
It almost goes beyond failures to simply issue credentials with access and soars right into professional malpractice on the CTO's part.
-
For those not wanting to click a Reddit link:
Today was my first day on the job as a Junior Software Developer and was my first non-internship position after university. Unfortunately i screwed up badly.
I was basically given a document detailing how to setup my local development environment. Which involves run a small script to create my own personal DB instance from some test data. After running the command i was supposed to copy the database url/password/username outputted by the command and configure my dev environment to point to that database. Unfortunately instead of copying the values outputted by the tool, i instead for whatever reason used the values the document had.
Unfortunately apparently those values were actually for the production database (why they are documented in the dev setup guide i have no idea). Then from my understanding that the tests add fake data, and clear existing data between test runs which basically cleared all the data from the production database. Honestly i had no idea what i did and it wasn't about 30 or so minutes after did someone actually figure out/realize what i did.
While what i had done was sinking in. The CTO told me to leave and never come back. He also informed me that apparently legal would need to get involved due to severity of the data loss. I basically offered and pleaded to let me help in someway to redeem my self and i was told that i "completely fucked everything up".
So i left. I kept an eye on slack, and from what i can tell the backups were not restoring and it seemed like the entire dev team was on full on panic mode. I sent a slack message to our CTO explaining my screw up. Only to have my slack account immediately disabled not long after sending the message.
I haven't heard from HR, or anything and i am panicking to high heavens. I just moved across the country for this job, is there anything i can even remotely do to redeem my self in this situation? Can i possibly be sued for this? Should i contact HR directly? I am really confused, and terrified. -
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
-
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
-
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
Agreed!
-
@dafyre said in Junior Dev destroys PROD DB on first day.:
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
Agreed!
I used to have a user (CFO) that was half ass proficient in MS Access. All day she would have her Access & Excel forms up that created a RW connection to the main DB so she could run queries against the data. Told her many times how dangerous it was to do that but she insisted and being over the IT dept. we didn't have a choice. Well one day she messed up and committed a change to a table and managed to break it. Luckily we had live backups of the data and nothing was lost.
In the end she still insisted on doing it her way and even though she messed up she wasn't going to change her ways.
-
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
Well, that was all he thought he had.
He was following instructions.
Even if he copy/pasted the wrong thing, said instructions should not have been able to do what it did.
-
Yes, whoever made those instructions with admin db user creds on them, that is who should be fired. Probably the CTO who fired the kid.
-
@Kyle said in Junior Dev destroys PROD DB on first day.:
@Kelly said in Junior Dev destroys PROD DB on first day.:
The levels of organizational failure there are incredible.
It almost goes beyond failures to simply issue credentials with access and soars right into professional malpractice on the CTO's part.
Not really. CTO is an engineer, no production tie in. Why does the CTO have that access is more of a question. That department should not have production access.
-
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
It's because the creds were public and put into a doc that EVERYONE has. Zero security.
-
@momurda said in Junior Dev destroys PROD DB on first day.:
Yes, whoever made those instructions with admin db user creds on them, that is who should be fired. Probably the CTO who fired the kid.
CTO should face legal action. Firing someone as a cover up is a big no no. Someone higher up, like the board, should step in and look at what's going on there.
-
And look... no backups. And they have the developers trying to do the restores. Apparently they thought that they could get away without an IT team?
-
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
Agreed!
I used to have a user (CFO) that was half ass proficient in MS Access. All day she would have her Access & Excel forms up that created a RW connection to the main DB so she could run queries against the data. Told her many times how dangerous it was to do that but she insisted and being over the IT dept. we didn't have a choice. Well one day she messed up and committed a change to a table and managed to break it. Luckily we had live backups of the data and nothing was lost.
In the end she still insisted on doing it her way and even though she messed up she wasn't going to change her ways.
Was the CEO aware?
-
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
All developers, even the CTO, should be read only at best. PEOPLE don't need RW access to databases at all. Developers should be working on development, not production. There are cases when devs need to see production, but that's really, really rare.
-
@Kelly said in Junior Dev destroys PROD DB on first day.:
The levels of organizational failure there are incredible.
Hopefully this is public enough that the CEO stepped in and let the CTO go.
-
So the dude followed the instructions a little to closely and was fired due the results of that. F that noise. Not his fault at all.
No backup to restore? Also, not his fault.
Legal action against him, not bloody likely. If anything he has a case against them for wrongful termination.
Still with that much incompetence going around, and a supervisor who is gonna fire you rather than educate you and/or put the blame where it actually belongs, I think the guy is way better off staying fired and looking for a job at a company.
-
@scottalanmiller said in Junior Dev destroys PROD DB on first day.:
@Kelly said in Junior Dev destroys PROD DB on first day.:
The levels of organizational failure there are incredible.
Hopefully this is public enough that the CEO stepped in and let the CTO go.
How is it public?
It is 100% anon.
Unless someone names names and also points the CEO or board to Reddit, this is nothing.
-
@jrc said in Junior Dev destroys PROD DB on first day.:
So the dude followed the instructions a little to closely and was fired due the results of that. F that noise. Not his fault at all.
No backup to restore? Also, not his fault.
Legal action against him, not bloody likely. If anything he has a case against them for wrongful termination.
Wrongful termination to cover up the person that fired you's mistake and then threatening you on top of it? That's a good combination for the HR to hear.
-
@JaredBusch said in Junior Dev destroys PROD DB on first day.:
@scottalanmiller said in Junior Dev destroys PROD DB on first day.:
@Kelly said in Junior Dev destroys PROD DB on first day.:
The levels of organizational failure there are incredible.
Hopefully this is public enough that the CEO stepped in and let the CTO go.
How is it public?
It is 100% anon.
Unless someone names names and also points the CEO or board to Reddit, this is nothing.
The incident is public. Hopefully someone from there runs it up the flag pole.
-
@scottalanmiller said in Junior Dev destroys PROD DB on first day.:
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
@Kyle said in Junior Dev destroys PROD DB on first day.:
@dafyre said in Junior Dev destroys PROD DB on first day.:
That's rough though. Why would they have a junior dev with full control credentials on a production database on day one anyhow?
Only a couple people should have full read/write access to a PROD DB. A first day Junior should only have Read Only. He only needs write access to his sandbox.
Agreed!
I used to have a user (CFO) that was half ass proficient in MS Access. All day she would have her Access & Excel forms up that created a RW connection to the main DB so she could run queries against the data. Told her many times how dangerous it was to do that but she insisted and being over the IT dept. we didn't have a choice. Well one day she messed up and committed a change to a table and managed to break it. Luckily we had live backups of the data and nothing was lost.
In the end she still insisted on doing it her way and even though she messed up she wasn't going to change her ways.
Was the CEO aware?
Yes, Didn't care. He said if she deemed it "safe" then it was OK.
