XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
At this point, I'm even at a loss on trying to figure out where to begin to explain to him how this is possible, as he hasn't seem to accept the premise in the first place.
When is he due back in the office? I would try to get on his schedule first thing ASAP, sit down with him, and talk with him about this. Maybe he can't visualize it over the phone, but once you draw it out for him, then he might be able to understand.
If he still refuses, I'd have a sheet of paper that a lawyer has drawn up that takes the blame off of you that he has to sign. CYA all you can here. Document everything and how you found it. I'd also be looking actively for other opportunities as well with a resignation in hand as a last resort.
-
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
At this point, I'm even at a loss on trying to figure out where to begin to explain to him how this is possible, as he hasn't seem to accept the premise in the first place.
Not your job to. If he doesn't believe it, then the system is healthy and there is nothing more for you to do. His system being down is his decision at this point.
If this is truly the case, then the owner is either incompetant in IT, which is typically the case, or he is delusional and needs mental help. I'm not trying to make this a joke either. However, delusional might also show itself in other aspects in his life. I'm not a psychiatrist so don't take this as advice.
-
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
At this point, I'm even at a loss on trying to figure out where to begin to explain to him how this is possible, as he hasn't seem to accept the premise in the first place.
Not your job to. If he doesn't believe it, then the system is healthy and there is nothing more for you to do. His system being down is his decision at this point.
If this is truly the case, then the owner is either incompetant in IT, which is typically the case, or he is delusional and needs mental help. I'm not trying to make this a joke either. However, delusional might also show itself in other aspects in his life. I'm not a psychiatrist so don't take this as advice.
And if the owner is getting his hands in IT that makes him foolish in business.
-
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
At this point, I'm even at a loss on trying to figure out where to begin to explain to him how this is possible, as he hasn't seem to accept the premise in the first place.
Not your job to. If he doesn't believe it, then the system is healthy and there is nothing more for you to do. His system being down is his decision at this point.
If this is truly the case, then the owner is either incompetant in IT, which is typically the case, or he is delusional and needs mental help. I'm not trying to make this a joke either. However, delusional might also show itself in other aspects in his life. I'm not a psychiatrist so don't take this as advice.
It's true. Saying that rAID 5 can't fail is like saying you don't need brakes. Because. seat belts can't fail.
It's a lacking grasp on reality. That RAID WILL is obvious beyond reason.
-
First, thanks everyone for your help yesterday, here's a brief update, more to come after a couple of phone calls.
I've powered down the SAN and reseated the drives. No joy. About to call Dell and see what options I may have, if any. The good news is there's a second SAN, and some data for the VMs is on there, but few OSes. The good news, is that somehow, I really have no idea how, the Xen servers reattached to the local storage overnight, and to the one virtual disk (Spindle2) and one DC is available. I'll give more details later.I wouldn't blame you for thinking I'm an incompetent tool, but I appreciate your help, anyway.
-
Cool. That the local storage was not attached was weird. That the SAN failed was.... expected. So the combination was making things complicated.
-
The previous guy setup local and SAN storage? lol I mean I guess there might be cases for that that, but still.
Now that you have a working DC, personally I'd grab any ol' desktop, install Hyper-V or XS on it and install Windows server there and get yourself another DC ASAP.
-
@Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
The previous guy setup local and SAN storage? lol I mean I guess there might be cases for that that, but still.
If making the mistake of having SAN, you should always mitigate it with having local as well. That's actually the only good thing we've heard so far, other than using XenServer.
-
No, there was a second SAN of which I was unaware. I know how bad that sounds, but since I've started here, it's been crisis after crisis, and I've had a quote from Scale that would have made all of this obsolete, provided the owner approved it in a conversation next week. I'm finding new things every day. With this probably going away in a week, I didn't spend a lot of time on it. This entire environment has been triage since I walked in the door.
-
And yes, a cobbled together DC is in the plans after I get off the horn with Dell.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
And yes, a cobbled together DC is in the plans after I get off the horn with Dell.
And then disconnect anything from AD that doesn't need to be on AD. Too much AD dependency is a common SMB mistake.
-
On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
The good news, is that somehow, I really have no idea how, the Xen servers reattached to the local storage overnight, and to the one virtual disk (Spindle2) and one DC is available. I'll give more details later.
What's on the local storage? what is Spindle2?
-
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
Uhm, full backup created first?
-
Some VM OSes aren't coming up. Getting this on most (all?) of the VMs that had their OS on the downed SAN.
-
@travisdh1 said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
Uhm, full backup created first?
Second that ^^
-
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
XenServer is able to create backups of its VMs, but, from my experience, its not in the GUI. You'd have to go to the CLI. Here are a few helps from what I have found.
http://tecadmin.net/backup-running-virtual-machine-in-xenserver/#
https://nchrissos.wordpress.com/2013/09/05/free-backup-for-citrix-xenserver-live-virtual-machines/ -
@CitrixNewbJD Find a system (desktop or server) with some diskspace. Use native Windows backup and do system state if you have nothing currently. XenServer may have something built in. Awful Windows backups are better than none.
