XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
And yes, a cobbled together DC is in the plans after I get off the horn with Dell.
And then disconnect anything from AD that doesn't need to be on AD. Too much AD dependency is a common SMB mistake.
-
On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
The good news, is that somehow, I really have no idea how, the Xen servers reattached to the local storage overnight, and to the one virtual disk (Spindle2) and one DC is available. I'll give more details later.
What's on the local storage? what is Spindle2?
-
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
Uhm, full backup created first?
-
Some VM OSes aren't coming up. Getting this on most (all?) of the VMs that had their OS on the downed SAN.
-
@travisdh1 said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
Uhm, full backup created first?
Second that ^^
-
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
XenServer is able to create backups of its VMs, but, from my experience, its not in the GUI. You'd have to go to the CLI. Here are a few helps from what I have found.
http://tecadmin.net/backup-running-virtual-machine-in-xenserver/#
https://nchrissos.wordpress.com/2013/09/05/free-backup-for-citrix-xenserver-live-virtual-machines/ -
@CitrixNewbJD Find a system (desktop or server) with some diskspace. Use native Windows backup and do system state if you have nothing currently. XenServer may have something built in. Awful Windows backups are better than none.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
On the phone with Dell, and the logs indicate that the two drives went down within seconds of each other, which may indicate that one of the drives isn't truly dead. They're going to try to revive it. Fingers crossed.
It might, but it's RAID 5 so multiple drives failing close in time to each other is common.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
You got VERY lucky. Did they tell you why the array went down if the drive didn't fail?
-
@travisdh1 said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@Dashrender @scottalanmiller Systems are up. New disk in, VD is degraded and being rebuilt.
Uhm, full backup created first?
This ^^^^
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
Take down some desktops, use them as backup devices. Go get AetherStore and make a storage device out of your desktops. Do something.
-
@NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@travisdh1 @seal Full backup created with what? As I noted yesterday there is no backup solution in place as of yet, unless you're referring to something I'm not grokking.
XenServer is able to create backups of its VMs, but, from my experience, its not in the GUI. You'd have to go to the CLI. Here are a few helps from what I have found.
http://tecadmin.net/backup-running-virtual-machine-in-xenserver/#
https://nchrissos.wordpress.com/2013/09/05/free-backup-for-citrix-xenserver-live-virtual-machines/He needs a backup target
-
Or skip the backups and remind the owner that "nothing here can fail." Why worry about backups if the boss doesn't?
-
Okay, lot of links coming. This is at the request of the OP, so don't think that I'm berating him. He requested the info so that he could show it upstream.
-
https://mangolassi.it/topic/11324/scott-alan-miller-smb-system-architectural-patterns
https://mangolassi.it/topic/10337/defining-high-availability/
https://mangolassi.it/topic/8822/why-dual-controllers-is-not-a-risk-mitigation-strategy-alone
https://mangolassi.it/topic/8821/risk-3-2-1-stock-inverted-pyramid-design/
https://mangolassi.it/topic/8743/risk-single-server-versus-the-smallest-inverted-pyramid-design/
https://mangolassi.it/topic/8090/how-does-local-storage-offer-high-availability/
https://mangolassi.it/topic/6634/the-inverted-pyramid-of-doom-challenge/
https://mangolassi.it/topic/6653/is-this-a-single-point-of-failure-or-spof
https://mangolassi.it/topic/6190/redundancy-is-never-a-goal-reliability-is-a-goal-redundancy-is-a-tool/
https://mangolassi.it/topic/5084/why-non-uniform-ure-distribution-may-make-parity-raid-riskier-than-thought
https://mangolassi.it/topic/3010/understanding-dependency-chains
https://mangolassi.it/topic/8757/why-is-raid-not-a-backup
https://mangolassi.it/topic/121/raid-link-blast
https://mangolassi.it/topic/9971/smb-system-architecture-link-blast -
@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...
I do plan on updating my LinkedIn profile and resume over the next few days.
-
@CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:
@scottalanmiller He's been talking to his most trusted employee, a developer. Apparently he doesn't even understand that that Xen and the SAN are two completely different things...
I do plan on updating my LinkedIn profile and resume over the next few days.
And yet.... he's sure that this "thing" can't fail and demands to be in charge of IT.
