XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.

It might, but it's RAID 5 so multiple drives failing close in time to each other is common.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.

You got VERY lucky. Did they tell you why the array went down if the drive didn't fail?

scottalanmiller

@travisdh1 said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.

Uhm, full backup created first?

This ^^^^

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.

Take down some desktops, use them as backup devices. Go get AetherStore and make a storage device out of your desktops. Do something.

scottalanmiller

@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.

XenServer is able to create backups of its VMs, but, from my experience, its not in the GUI. You'd have to go to the CLI. Here are a few helps from what I have found.

http://tecadmin.net/backup-running-virtual-machine-in-xenserver/#
https://nchrissos.wordpress.com/2013/09/05/free-backup-for-citrix-xenserver-live-virtual-machines/

He needs a backup target

scottalanmiller

Or skip the backups and remind the owner that "nothing here can fail." Why worry about backups if the boss doesn't?

scottalanmiller

Okay, lot of links coming. This is at the request of the OP, so don't think that I'm berating him. He requested the info so that he could show it upstream.

scottalanmiller

CitrixNewbJD

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

And yet.... he's sure that this "thing" can't fail and demands to be in charge of IT.

NerdyDad

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

Is this trusted employee also kinfolk?

CitrixNewbJD

@NerdyDad Close. Friend of the family. He's also a really good guy, and probably a genius. He's on my side. Not necessarily a correlation to his genius, just sayin'.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@NerdyDad Close. Friend of the family. He's also a really good guy, and probably a genius. He's on my side. Not necessarily a correlation to his genius, just sayin'.

Does he have solid IT knowledge, or just generally smart? If he's smart, why is he working for someone so.... not smart? What's the benefit that the owner is bringing to the equation? Why doesn't the guy that does all the work go off on his own?

Real questions. What is the owner holding over everyone if he's not bringing anything to the table and is actively sabotaging the business?

scottalanmiller

That's more business consulting, but something I always say to people looking to start a business.... if you are the owner, what are you bringing to the table that keeps people from just running the business without you? Is it fame, marketing, sales, investment dollars, a truly brilliant idea, property, experience, hard work, special knowledge, etc. If you don't have something that makes you critical, your team will be better off being your competitor than your staff.

scottalanmiller

My question about the genius and IT is....

If he's smart AND knows IT, great, he should intervene directly.
If he's smart and does NOT know IT, that's fine, he should express to the owner how to listen to and trust the proper people.

At the base level, the issue here is an owner with flawed business decision process skills. He interjects banal opinions, doesn't listen to good advice, doesn't trust the people trying to help him, isn't thinking through obvious non-technical problems with his environment, is spending without purpose, is not applying common sense, is believing in magic, etc. These are things that need to be addressed that maybe the trusted genius can address.

scale

Glad to hear that you are starting to be able to recover some of your data. Definitely let us know if we can help in any way!

CitrixNewbJD

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

That's more business consulting, but something I always say to people looking to start a business.... if you are the owner, what are you bringing to the table that keeps people from just running the business without you? Is it fame, marketing, sales, investment dollars, a truly brilliant idea, property, experience, hard work, special knowledge, etc. If you don't have something that makes you critical, your team will be better off being your competitor than your staff.

A work visa. He's British/Canadian.

CitrixNewbJD

@scale It appears we may not be in as good as shape as we thought. The drives going into the SAN are failing. Trying to move everything off now.

scottalanmiller

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

That's more business consulting, but something I always say to people looking to start a business.... if you are the owner, what are you bringing to the table that keeps people from just running the business without you? Is it fame, marketing, sales, investment dollars, a truly brilliant idea, property, experience, hard work, special knowledge, etc. If you don't have something that makes you critical, your team will be better off being your competitor than your staff.

A work visa. He's British/Canadian.

Ah ha, that kinda explains the disaster

That the value is a visa, not that he's Canadian. lol

Dashrender

@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...

I do plan on updating my LinkedIn profile and resume over the next few days.

Who doesn't understand? The dev or the owner? and really, why does it matter? Unless the dev is the one who designed this mess you're dealing with, they should both be out of the equation on the design/solution.