ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

    IT Discussion
    xenserver xenserver 6.2 iscsi san
    10
    243
    46.1k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      NerdyDad
      last edited by

      I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

      S 1 Reply Last reply Reply Quote 0
      • S
        scottalanmiller @momurda
        last edited by

        @momurda said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD
        No i dont think so.
        You got some work to do though
        Probably need to do:
        log into the san and change the creds for iscsi, disable ad authentication in xc, and then probably forget/reattach the sr

        I didn't see it on the iSCSI, darn it. Still fixable, but more work.

        1 Reply Last reply Reply Quote 0
        • S
          scottalanmiller @NerdyDad
          last edited by

          @NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

          I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

          That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

          N 1 Reply Last reply Reply Quote 3
          • N
            NerdyDad @scottalanmiller
            last edited by

            @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

            @NerdyDad said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

            I always try to make sure that I have an administrative local user to every system, just in case I cannot get to AD.

            That, and never have AD on top of something that depends on it. That's like locking your keys in your car.

            Yup. Our SAN is isolated from the rest of the house network physically. All authentication between the SAN and the host should never even touch AD just in case of things like this.

            Sometimes playing what-if is a good exercise to keep bad things from happening as far as security, reliability, and recovery.

            1 Reply Last reply Reply Quote 1
            • M
              momurda @scottalanmiller
              last edited by

              @scottalanmiller
              Yes the cifs repos can be forgotten for now. I doubt there is anything important there in the iso repo.
              @CitrixNewbJD
              Before you do anything else, make sure you have the root pw for your xs servers as disabling AD integration will force you to use them.

              M C 2 Replies Last reply Reply Quote 3
              • M
                momurda @momurda
                last edited by

                Unless you have a DC that you can get working, ie one not attached to your shared storage. Perhaps you have a physical dc somewhere? Unlikely i know.

                C 1 Reply Last reply Reply Quote 1
                • S
                  seal
                  last edited by

                  @CitrixNewbJD This is Frank. I hope Scott and the gang here can help get you back operational quickly. To touch on a topic that was mentioned earlier, VMs from Xen can be imported and converted on the fly. As long as the Scale nodes can browse to the VM files (.vhd) on the storage, the XenServer functional state doesn't matter.

                  C 1 Reply Last reply Reply Quote 1
                  • C
                    CitrixNewbJD @momurda
                    last edited by

                    @momurda

                    Having been through this once before, and learning the hard way, I do normally have a physical DC. Despite my warnings, because I know that we do not currently have one here, I was told to bring it all down. And here we are. We do not have a physical DC.

                    S 1 Reply Last reply Reply Quote 0
                    • C
                      CitrixNewbJD @momurda
                      last edited by

                      @momurda

                      I've been using the root authentication for everything.

                      S 1 Reply Last reply Reply Quote 0
                      • C
                        CitrixNewbJD @seal
                        last edited by

                        @seal

                        Frank, can we speak on the phone for a minute so I can be sure I can intelligently talk to the guy when demanding his money?

                        1 Reply Last reply Reply Quote 0
                        • S
                          scottalanmiller @CitrixNewbJD
                          last edited by

                          @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                          @momurda

                          Having been through this once before, and learning the hard way, I do normally have a physical DC.

                          This is absolutely the wrong response. You should never have a physical DC, ever. There is zero issues here with virtualization. There are two problems....

                          • Zero AD redundancy
                          • An inverted pyramid of doom (single storage for all systems)

                          Fixing either of those anti-practices would have saved you. Physical would have zero benefit and is the polar opposite of the reaction that you should have.

                          D 1 Reply Last reply Reply Quote 1
                          • S
                            scottalanmiller @CitrixNewbJD
                            last edited by

                            @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                            @momurda

                            I've been using the root authentication for everything.

                            So we are safe there.

                            1 Reply Last reply Reply Quote 0
                            • S
                              scottalanmiller
                              last edited by

                              More on the IPOD: http://www.smbitjournal.com/2013/06/the-inverted-pyramid-of-doom/

                              And in video form from MangoCon:

                              Youtube Video

                              1 Reply Last reply Reply Quote 1
                              • C
                                CitrixNewbJD
                                last edited by

                                So, when looking for places to turn off AD integration, I see this...

                                0_1482873847039_Screenshot 2016-12-27 15.23.59.png

                                1 Reply Last reply Reply Quote 0
                                • S
                                  scottalanmiller
                                  last edited by

                                  It's not pool integration that is the issue, it's SAN integration. Check the SAN (PowerVault) interface instead.

                                  1 Reply Last reply Reply Quote 2
                                  • C
                                    CitrixNewbJD
                                    last edited by

                                    @seal Just came across these two items on the SAN interface. Dental_Data, Spindlemedia, are critical and it looks like those VDs failed.

                                    PROFILE FOR STORAGE ARRAY: MDS-Spindle01 (12/27/16 3:28:58 PM) 
                                     
                                     
                                    STANDARD VIRTUAL DISKS------------------------------ 
                                     
                                    SUMMARY 
                                     
                                       Number of standard virtual disks: 3 
                                     
                                       See other Virtual Disks sub-tabs for premium feature information. 
                                     
                                     
                                       NAME          STATUS  CAPACITY  RAID LEVEL  DISK GROUP  DRIVE TYPE   
                                       Dental_Data   Failed  1.495 TB  5           0           SAS          
                                       SpindleMedia  Failed  2.862 TB  5           0           SAS          
                                       Virtual       Failed  1.367 TB  5           0           SAS          
                                     
                                    DETAILS 
                                     
                                     
                                       Virtual Disk name:                      Dental_Data                                       
                                                                                                                                 
                                          Virtual Disk status:                 Failed                                            
                                                                                                                                 
                                          Capacity:                            1.495 TB                                          
                                          Virtual Disk world-wide identifier:  60:02:4e:80:00:7b:78:6a:00:00:04:13:4a:96:70:f3   
                                          Subsystem ID (SSID):                 1                                                 
                                          Associated disk group:               0                                                 
                                          RAID level:                          5                                                 
                                                                                                                                 
                                          Physical Disk type:                  Serial Attached SCSI (SAS)                        
                                          Enclosure loss protection:           No                                                
                                                                                                                                 
                                          Preferred owner:                     RAID Controller Module in slot 1                  
                                          Current owner:                       RAID Controller Module in slot 1                  
                                     
                                     
                                          Segment size:                                       128 KB     
                                          Capacity reserved for future segment size changes:  Yes        
                                          Maximum future segment size:                        2,048 KB   
                                          Modification priority:                              High       
                                     
                                     
                                          Read cache:                            Enabled    
                                          Write cache:                           Enabled    
                                             Write cache without batteries:      Disabled   
                                             Write cache with mirroring:         Enabled    
                                          Flush write cache after (in seconds):  10.00      
                                          Dynamic cache read prefetch:           Enabled    
                                                                                            
                                          Enable background media scan:          Enabled    
                                          Media scan with consistency check:     Enabled    
                                                                                            
                                          Pre-Read consistency check:            Disabled   
                                     
                                     
                                       Virtual Disk name:                      SpindleMedia                                      
                                                                                                                                 
                                          Virtual Disk status:                 Failed                                            
                                                                                                                                 
                                          Capacity:                            2.862 TB                                          
                                          Virtual Disk world-wide identifier:  60:02:4e:80:00:70:ed:06:00:00:07:f5:4d:ba:7b:fb   
                                          Subsystem ID (SSID):                 2                                                 
                                          Associated disk group:               0                                                 
                                          RAID level:                          5                                                 
                                                                                                                                 
                                          Physical Disk type:                  Serial Attached SCSI (SAS)                        
                                          Enclosure loss protection:           No                                                
                                                                                                                                 
                                          Preferred owner:                     RAID Controller Module in slot 0                  
                                          Current owner:                       RAID Controller Module in slot 1                  
                                     
                                     
                                          Segment size:                                       128 KB     
                                          Capacity reserved for future segment size changes:  Yes        
                                          Maximum future segment size:                        2,048 KB   
                                          Modification priority:                              High       
                                     
                                     
                                          Read cache:                            Enabled    
                                          Write cache:                           Enabled    
                                             Write cache without batteries:      Disabled   
                                             Write cache with mirroring:         Enabled    
                                          Flush write cache after (in seconds):  10.00      
                                          Dynamic cache read prefetch:           Enabled    
                                                                                            
                                          Enable background media scan:          Enabled    
                                          Media scan with consistency check:     Enabled    
                                                                                            
                                          Pre-Read consistency check:            Disabled   
                                     
                                     
                                       Virtual Disk name:                      Virtual                                           
                                                                                                                                 
                                          Virtual Disk status:                 Failed                                            
                                                                                                                                 
                                          Capacity:                            1.367 TB                                          
                                          Virtual Disk world-wide identifier:  60:02:4e:80:00:70:ed:06:00:00:04:31:4a:96:73:09   
                                          Subsystem ID (SSID):                 0                                                 
                                          Associated disk group:               0                                                 
                                          RAID level:                          5                                                 
                                                                                                                                 
                                          Physical Disk type:                  Serial Attached SCSI (SAS)                        
                                          Enclosure loss protection:           No                                                
                                                                                                                                 
                                          Preferred owner:                     RAID Controller Module in slot 0                  
                                          Current owner:                       RAID Controller Module in slot 1                  
                                     
                                     
                                          Segment size:                                       128 KB     
                                          Capacity reserved for future segment size changes:  Yes        
                                          Maximum future segment size:                        2,048 KB   
                                          Modification priority:                              High       
                                     
                                     
                                          Read cache:                            Enabled    
                                          Write cache:                           Enabled    
                                             Write cache without batteries:      Disabled   
                                             Write cache with mirroring:         Enabled    
                                          Flush write cache after (in seconds):  10.00      
                                          Dynamic cache read prefetch:           Enabled    
                                                                                            
                                          Enable background media scan:          Enabled    
                                          Media scan with consistency check:     Enabled    
                                                                                            
                                          Pre-Read consistency check:            Disabled   
                                    

                                    0_1482874726865_Screenshot 2016-12-27 15.38.36.png

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      scottalanmiller
                                      last edited by

                                      Oh look, on top of everything else, they left you with RAID 5, too. Figures. Whoever set this up really set you up for failure.

                                      1 Reply Last reply Reply Quote 2
                                      • S
                                        scottalanmiller
                                        last edited by

                                        Your predecessor definitely pulled this on you: https://mangolassi.it/topic/11852/why-it-builds-a-house-of-cards

                                        1 Reply Last reply Reply Quote 1
                                        • S
                                          scottalanmiller
                                          last edited by

                                          Looks like, on top of other problems, the SAN has died. It's hard to tell from this, but it looks like those are the LUNs that hold all of your VMs?

                                          1 Reply Last reply Reply Quote 2
                                          • M
                                            momurda
                                            last edited by

                                            So 2 drives failed at once? You should be able to go into the server room and see some sort of blinky light pattern that indicates what/how many drives are gone.
                                            Did you lose a RAID Controller?

                                            S 1 Reply Last reply Reply Quote 1
                                            • 1
                                            • 2
                                            • 9
                                            • 10
                                            • 11
                                            • 12
                                            • 13
                                            • 11 / 13
                                            • First post
                                              Last post