ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!

    IT Discussion
    xenserver xenserver 6.2 iscsi san
    10
    243
    44.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DashrenderD
      Dashrender @scottalanmiller
      last edited by

      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

      We still need to see the logs 🙂

      [root@xen2 ~]# tail -n 100 messages
tail: cannot open `messages' for reading: No such file or directory

      That's not good. That means that something really went wrong. Probably the disk had filled up.

      Is there another way to view log entries?

      Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?

      I used no cd command. I just tried "cd /var/logs" and was told there was no directory.

      We are definitely seeing a problem here 🙂

      Scott, why is the formatting so messed up?

      scottalanmillerS 1 Reply Last reply Reply Quote 0
      • scottalanmillerS
        scottalanmiller @Dashrender
        last edited by

        @Dashrender said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

        We still need to see the logs 🙂

        [root@xen2 ~]# tail -n 100 messages
tail: cannot open `messages' for reading: No such file or directory

        That's not good. That means that something really went wrong. Probably the disk had filled up.

        Is there another way to view log entries?

        Not if they are not there. That's not a log viewer, that's the logs itself. Are you use that you used the cd command to get into the right directory before trying to view the logs?

        I used no cd command. I just tried "cd /var/logs" and was told there was no directory.

        We are definitely seeing a problem here 🙂

        Scott, why is the formatting so messed up?

        No idea, but it's not from ML, that's from his cut and paste. Those spaces are all in his original.

        1 Reply Last reply Reply Quote 0
        • C
          CitrixNewbJD
          last edited by 

          Dec 27 13:13:16 xen2 last message repeated 2 times
Dec 27 13:13:16 xen2 kernel: [18030.894547] device-mapper: multipath: Failing path 8:128.
Dec 27 13:13:16 xen2 kernel: [18030.894934] device-mapper: multipath: Failing path 8:80.
Dec 27 13:13:16 xen2 kernel: [18030.895002] device-mapper: multipath: Failing path 8:176.
Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151320 UNIX /var/xapi/xapi|session.logout D:9b0536d5f01c|xapi] Session.destroy trackid=e8dec3c23c8d667dfcfbfd772ecabd9e
Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151321 UNIX /var/xapi/xapi|session.login_with_password D:72d677979b15|xapi] Session.create trackid=2db03e1031b1b474ab4e3a61a73c4824                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:16 xen2 kernel: [18031.409957] sd 4:0:0:1: [sdc] Unhandled sense code
Dec 27 13:13:16 xen2 kernel: [18031.409960] sd 4:0:0:1: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 13:13:16 xen2 kernel: [18031.409963] sd 4:0:0:1: [sdc] Sense Key : Hardware Error [current]
Dec 27 13:13:16 xen2 kernel: [18031.409965] sd 4:0:0:1: [sdc] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 13:13:16 xen2 kernel: [18031.409969] sd 4:0:0:1: [sdc] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00
Dec 27 13:13:16 xen2 kernel: [18031.409974] end_request: I/O error, dev sdc, sector 2936012672
Dec 27 13:13:16 xen2 kernel: [18031.409989] device-mapper: multipath: Failing path 8:32.
Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151359 INET 0.0.0.0:80|session.logout D:6b5f0e1950d3|xapi] Session.destroy trackid=95373089b3a53ad07d348ea483840e40
Dec 27 13:13:16 xen2 multipathd: 8:32: mark as failed
Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15
Dec 27 13:13:16 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0
Dec 27 13:13:16 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151365 UNIX /var/xapi/xapi|session.logout D:7808df6ecf1f|xapi] Session.destroy trackid=2db03e1031b1b474ab4e3a61a73c4824
Dec 27 13:13:16 xen2 xapi: [ info|xen2|1151366 UNIX /var/xapi/xapi|session.login_with_password D:107e4867f954|xapi] Session.create trackid=398819a8505258b63db68cd0d33af4b1                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: sdi - rdac checker reports path is up
Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:17 xen2 multipathd: 8:128: reinstated
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1
Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151408 UNIX /var/xapi/xapi|session.logout D:96b0ad24c6c1|xapi] Session.destroy trackid=398819a8505258b63db68cd0d33af4b1
Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151409 UNIX /var/xapi/xapi|session.login_with_password D:f87fb7e4fce8|xapi] Session.create trackid=a797b1b90698608ddd5f6919804b5484                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:17 xen2 kernel: [18032.440371] sd 6:0:0:1: [sdi] Unhandled sense code
Dec 27 13:13:17 xen2 kernel: [18032.440374] sd 6:0:0:1: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 13:13:17 xen2 kernel: [18032.440376] sd 6:0:0:1: [sdi] Sense Key : Hardware Error [current]
Dec 27 13:13:17 xen2 kernel: [18032.440379] sd 6:0:0:1: [sdi] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 13:13:17 xen2 kernel: [18032.440382] sd 6:0:0:1: [sdi] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00
Dec 27 13:13:17 xen2 kernel: [18032.440387] end_request: I/O error, dev sdi, sector 2936012672
Dec 27 13:13:17 xen2 kernel: [18032.440404] device-mapper: multipath: Failing path 8:128.
Dec 27 13:13:17 xen2 multipathd: 8:128: mark as failed
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15
Dec 27 13:13:17 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0
Dec 27 13:13:17 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151447 UNIX /var/xapi/xapi|session.logout D:acca552975a2|xapi] Session.destroy trackid=a797b1b90698608ddd5f6919804b5484
Dec 27 13:13:17 xen2 xapi: [ info|xen2|1151448 UNIX /var/xapi/xapi|session.login_with_password D:5f3e3fccf3d5|xapi] Session.create trackid=53cb15330f2848d21867cbf051db9ed9                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: sdl - rdac checker reports path is ghost
Dec 27 13:13:18 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:18 xen2 multipathd: 8:176: reinstated
Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled
Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode
Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1
Dec 27 13:13:18 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #4
Dec 27 13:13:18 xen2 kernel: [18033.217827] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, queueing MODE_SELECT command
Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151482 UNIX /var/xapi/xapi|session.logout D:39dea3d23bb2|xapi] Session.destroy trackid=53cb15330f2848d21867cbf051db9ed9
Dec 27 13:13:18 xen2 xapi: [ info|xen2|1151483 UNIX /var/xapi/xapi|session.login_with_password D:487d0c440a6f|xapi] Session.create trackid=31c6550a7ad5c485e04e1ce80f3588dd                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:18 xen2 kernel: [18033.500501] sd 7:0:0:1: rdac: array MDS-Spindle01, ctlr 1, MODE_SELECT completed
Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151517 UNIX /var/xapi/xapi|session.logout D:6b32233bc36f|xapi] Session.destroy trackid=31c6550a7ad5c485e04e1ce80f3588dd
Dec 27 13:13:19 xen2 kernel: [18034.028373] sd 7:0:0:1: [sdl] Unhandled sense code
Dec 27 13:13:19 xen2 kernel: [18034.028376] sd 7:0:0:1: [sdl] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 13:13:19 xen2 kernel: [18034.028379] sd 7:0:0:1: [sdl] Sense Key : Hardware Error [current]
Dec 27 13:13:19 xen2 kernel: [18034.028381] sd 7:0:0:1: [sdl] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 13:13:19 xen2 kernel: [18034.028385] sd 7:0:0:1: [sdl] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00
Dec 27 13:13:19 xen2 kernel: [18034.028390] end_request: I/O error, dev sdl, sector 2936012672
Dec 27 13:13:19 xen2 kernel: [18034.028406] device-mapper: multipath: Failing path 8:176.
Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151537 INET 0.0.0.0:80|session.logout D:7e74bbbad814|xapi] Session.destroy trackid=57a035032e821d9289ab2ddc9c56ef23
Dec 27 13:13:19 xen2 multipathd: 8:176: mark as failed
Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15
Dec 27 13:13:19 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0
Dec 27 13:13:19 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:19 xen2 xapi: [ info|xen2|1151540 UNIX /var/xapi/xapi|session.login_with_password D:d750b0705d7f|xapi] Session.create trackid=86b83edaef195351a3b0fab7cc6cc21e                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151585 UNIX /var/xapi/xapi|session.logout D:3e57cedfef7a|xapi] Session.destroy trackid=86b83edaef195351a3b0fab7cc6cc21e
Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: sdf - rdac checker reports path is up
Dec 27 13:13:20 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:20 xen2 multipathd: 8:80: reinstated
Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled
Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode
Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1
Dec 27 13:13:20 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #3
Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151605 UNIX /var/xapi/xapi|session.login_with_password D:1379b8d2955b|xapi] Session.create trackid=3fa7db291a48d0f98ea6e364b1dccd5b                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:20 xen2 xapi: [ info|xen2|1151611 INET 0.0.0.0:80|session.logout D:e6afd1da2cf0|xapi] Session.destroy trackid=a246f22117a6ddf72bd492a7e8505ddd
Dec 27 13:13:21 xen2 kernel: [18035.743234] sd 5:0:0:1: [sdf] Unhandled sense code
Dec 27 13:13:21 xen2 kernel: [18035.743237] sd 5:0:0:1: [sdf] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Dec 27 13:13:21 xen2 kernel: [18035.743240] sd 5:0:0:1: [sdf] Sense Key : Hardware Error [current]
Dec 27 13:13:21 xen2 kernel: [18035.743242] sd 5:0:0:1: [sdf] <<vendor>> ASC=0x84 ASCQ=0x0ASC=0x84 ASCQ=0x0
Dec 27 13:13:21 xen2 kernel: [18035.743246] sd 5:0:0:1: [sdf] CDB: Read(10): 28 00 ae ff ff 80 00 00 08 00
Dec 27 13:13:21 xen2 kernel: [18035.743251] end_request: I/O error, dev sdf, sector 2936012672
Dec 27 13:13:21 xen2 kernel: [18035.743266] device-mapper: multipath: Failing path 8:80.
Dec 27 13:13:21 xen2 multipathd: 8:80: mark as failed
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Entering recovery mode: max_retries=15
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 0
Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151653 UNIX /var/xapi/xapi|session.logout D:57acea842a91|xapi] Session.destroy trackid=3fa7db291a48d0f98ea6e364b1dccd5b
Dec 27 13:13:21 xen2 xapi: [ info|xen2|1151654 UNIX /var/xapi/xapi|session.login_with_password D:e547785afb43|xapi] Session.create trackid=8fd799a70bdc34fb2a4d8bd676873ede                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: sdc - rdac checker reports path is ghost
Dec 27 13:13:21 xen2 multipathd: Path event for 360024e800070ed06000004314a967309, request call of mpathcount
Dec 27 13:13:21 xen2 multipathd: 8:32: reinstated
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: queue_if_no_path enabled
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: Recovered to normal mode
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: remaining active paths: 1
Dec 27 13:13:21 xen2 multipathd: 360024e800070ed06000004314a967309: switch to path group #2
Dec 27 13:13:21 xen2 kernel: [18036.256313] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, queueing MODE_SELECT command
Dec 27 13:13:21 xen2 kernel: [18036.525838] sd 4:0:0:1: rdac: array MDS-Spindle01, ctlr 0, MODE_SELECT completed
Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151678 UNIX /var/xapi/xapi|session.logout D:34466f735dad|xapi] Session.destroy trackid=8fd799a70bdc34fb2a4d8bd676873ede
Dec 27 13:13:22 xen2 xapi: [ info|xen2|1151679 UNIX /var/xapi/xapi|session.login_with_password D:cd7b037c4106|xapi] Session.create trackid=f258157e7fefe65077464bbd3c912521                                                                   pool=false uname=root is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
          1 Reply Last reply Reply Quote 0
          • C
            CitrixNewbJD
            last edited by

            Better?

            scottalanmillerS 1 Reply Last reply Reply Quote 0
            • scottalanmillerS
              scottalanmiller @CitrixNewbJD
              last edited by

              @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

              Better?

              Yup, thanks.

              1 Reply Last reply Reply Quote 0
              • scottalanmillerS
                scottalanmiller
                last edited by

                We need to see the Xen logs, too. Those are the logs with "xen" in the name.

                C 1 Reply Last reply Reply Quote 0
                • C
                  CitrixNewbJD @scottalanmiller
                  last edited by

                  @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                  We need to see the Xen logs, too. Those are the logs with "xen" in the name.
                  I can't upload the .gz to here. Can I open the contents in wordpad? Sorry if I missed something.

                  0_1482866277151_Screenshot 2016-12-27 13.17.43.png

                  C 1 Reply Last reply Reply Quote 0
                  • scottalanmillerS
                    scottalanmiller
                    last edited by

                    No current logs at all?

                    C 1 Reply Last reply Reply Quote 0
                    • C
                      CitrixNewbJD @CitrixNewbJD
                      last edited by

                      @CitrixNewbJD

                      It ate my text. How do I display the logs to you? Sorry. A lot of new knowledge in a short period of time, combined with stress.

                      1 Reply Last reply Reply Quote 0
                      • C
                        CitrixNewbJD @scottalanmiller
                        last edited by

                        @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                        No current logs at all?

                        0_1482866370090_Screenshot 2016-12-27 13.19.21.png

                        C 1 Reply Last reply Reply Quote 0
                        • C
                          CitrixNewbJD @CitrixNewbJD
                          last edited by

                          @CitrixNewbJD

                          Sorry, ate my text again. Unless I should be looking somewhere else...

                          scottalanmillerS 1 Reply Last reply Reply Quote 0
                          • C
                            CitrixNewbJD
                            last edited by

                            Is there any way to tell why the hosts can't even attach to their own local storage, or why it was unplugged in the first place?

                            1 Reply Last reply Reply Quote 0
                            • scottalanmillerS
                              scottalanmiller @CitrixNewbJD
                              last edited by

                              @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                              @CitrixNewbJD

                              Sorry, ate my text again. Unless I should be looking somewhere else...

                              Get us the SMlog tail...

                              tail -n 100 SMlog
                              C 1 Reply Last reply Reply Quote 0
                              • scottalanmillerS
                                scottalanmiller
                                last edited by

                                Then let's see what is in the xen folder, too.

                                1 Reply Last reply Reply Quote 0
                                • C
                                  CitrixNewbJD @scottalanmiller
                                  last edited by

                                  @scottalanmiller said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                  @CitrixNewbJD said in XenServer 6.2 servers down. I have no Xen skill. Most likely networking? Help!:

                                  @CitrixNewbJD

                                  Sorry, ate my text again. Unless I should be looking somewhere else...

                                  Get us the SMlog tail...

                                  tail -n 100 SMlog

                                  Dec 27 13:24:41 xen2 SM: [6456]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6456]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6456] multipathd>
Dec 27 13:24:41 xen2 SM: [6456] MPATH:  Set val: [1, 4]
Dec 27 13:24:41 xen2 SM: [6456] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:41 xen2 SM: [6456] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:41 xen2 SM: [6456] MPATH: Update done
Dec 27 13:24:41 xen2 SM: [6456] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6456] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6456] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: acquired /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: released /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6579] MPATH: I get the lock
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [1, 4]
Dec 27 13:24:41 xen2 SM: [6579] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Trying to acquire the lock
Dec 27 13:24:41 xen2 SM: [6663] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6663] lock: tried lock /var/lock/sm/mpathcount2/host, acquired: True (exists: True)
Dec 27 13:24:41 xen2 SM: [6663] Failed to lock /var/lock/sm/mpathcount1/host on first attempt, blocked by PID 6579
Dec 27 13:24:41 xen2 SM: [6666] MPATH: Trying to acquire the lock
Dec 27 13:24:41 xen2 SM: [6666] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6666] lock: tried lock /var/lock/sm/mpathcount2/host, acquired: False (exists: True)
Dec 27 13:24:41 xen2 SM: [6666] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6666] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6579] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6579] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6579] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6579] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6579]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6579] multipathd>
Dec 27 13:24:41 xen2 SM: [6579] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6579] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6579] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6579] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6579] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6579] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6579]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6579]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6579] multipathd>
Dec 27 13:24:41 xen2 SM: [6579] MPATH:  Set val: [2, 4]
Dec 27 13:24:41 xen2 SM: [6579] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:41 xen2 SM: [6579] MPATH: Update done
Dec 27 13:24:41 xen2 SM: [6579] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6579] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6579] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6663] lock: acquired /var/lock/sm/mpathcount1/host
Dec 27 13:24:41 xen2 SM: [6663] lock: released /var/lock/sm/mpathcount2/host
Dec 27 13:24:41 xen2 SM: [6663] MPATH: I get the lock
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:41 xen2 SM: [6663] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:41 xen2 SM: [6663] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [2, 4]
Dec 27 13:24:41 xen2 SM: [6663] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:41 xen2 SM: [6663] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:41 xen2 SM: [6663] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:41 xen2 SM: [6663] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:41 xen2 SM: [6663] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:41 xen2 SM: [6663] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:41 xen2 SM: [6663] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:41 xen2 SM: [6663]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:41 xen2 SM: [6663]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:41 xen2 SM: [6663] multipathd>
Dec 27 13:24:41 xen2 SM: [6663] mpath cmd: show map 360024e800070ed06000004314a967309 topology
Dec 27 13:24:42 xen2 SM: [6663] mpath output: multipathd> 360024e800070ed06000004314a967309 dm-0 DELL,MD3000i
Dec 27 13:24:42 xen2 SM: [6663] size=1.4T features='3 queue_if_no_path pg_init_retries 50' hwhandler='1 rdac' wp=rw
Dec 27 13:24:42 xen2 SM: [6663] |-+- policy='round-robin 0' prio=4 status=active
Dec 27 13:24:42 xen2 SM: [6663] | |- 6:0:0:1 sdi 8:128 active ghost  running
Dec 27 13:24:42 xen2 SM: [6663] | `- 4:0:0:1 sdc 8:32  failed faulty running
Dec 27 13:24:42 xen2 SM: [6663] `-+- policy='round-robin 0' prio=3 status=enabled
Dec 27 13:24:42 xen2 SM: [6663]   |- 5:0:0:1 sdf 8:80  failed faulty running
Dec 27 13:24:42 xen2 SM: [6663]   `- 7:0:0:1 sdl 8:176 active ready  running
Dec 27 13:24:42 xen2 SM: [6663] multipathd>
Dec 27 13:24:42 xen2 SM: [6663] Matched SCSIid, updating 360024e800070ed06000007f54dba7bfb
Dec 27 13:24:42 xen2 SM: [6663] MPATH: Updating entry for [360024e800070ed06000007f54dba7bfb], current: [0, 4]
Dec 27 13:24:42 xen2 SM: [6663] MPATH: Update done
Dec 27 13:24:42 xen2 SM: [6663] lock: closed /var/lock/sm/mpathcount2/host
Dec 27 13:24:42 xen2 SM: [6663] lock: released /var/lock/sm/mpathcount1/host
Dec 27 13:24:42 xen2 SM: [6663] lock: closed /var/lock/sm/mpathcount1/host
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Trying to acquire the lock
Dec 27 13:24:42 xen2 SM: [6728] lock: tried lock /var/lock/sm/mpathcount1/host, acquired: True (exists: True)
Dec 27 13:24:42 xen2 SM: [6728] MPATH: I get the lock
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e80007b786a000004134a9670f3
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e80007b786a000004134a9670f3], current: [2, 4]
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e800054baef0000ddd550c85082
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e800054baef0000ddd550c85082], current: [4, 4]
Dec 27 13:24:42 xen2 SM: [6728] Matched SCSIid, updating 360024e800070ed06000004314a967309
Dec 27 13:24:42 xen2 SM: [6728] MPATH: Updating entry for [360024e800070ed06000004314a967309], current: [2, 4]
Dec 27 13:24:42 xen2 SM: [6728] mpath cmd: show map 360024e800070ed06000004314a967309 topology
                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    CitrixNewbJD
                                    last edited by

                                    0_1482866750909_Screenshot 2016-12-27 13.25.40.png

                                    scottalanmillerS 1 Reply Last reply Reply Quote 0
                                    • scottalanmillerS
                                      scottalanmiller @CitrixNewbJD
                                      last edited by

                                      @CitrixNewbJD okay, no need to look in the xen folder, that's not going to be a useful log 🙂

                                      1 Reply Last reply Reply Quote 0
                                      • scottalanmillerS
                                        scottalanmiller
                                        last edited by

                                        I'm discussing conversion options with Scale. Looks like moving the disk images over would not be too hard as long as you can pull them off of the SAN.

                                        1 Reply Last reply Reply Quote 0
                                        • scottalanmillerS
                                          scottalanmiller
                                          last edited by

                                          If you do this...

                                          cd ..

                                          It will take you back to the main logs. What is the output of this...

                                          ls | grep -i xen
                                          C 1 Reply Last reply Reply Quote 0
                                          • scottalanmillerS
                                            scottalanmiller
                                            last edited by

                                            While you are working on that, this is a good time to discuss triage. In all honesty, this is an old, unmaintained cluster. What is the hardware of the compute nodes?

                                            The storage is poor. Even if we fix things, the system is very old in both software and hardware. So it's only a bandaid. Which is very doable, but it is what it is.

                                            Doing a migration, like right now, might be the logical path forward. Let's get your specs for the current cluster, concerns for a new one and discuss if that's a real possibility. Also, what is the moment to moment impact of the outage?

                                            C 1 Reply Last reply Reply Quote 1
                                            • 1
                                            • 2
                                            • 3
                                            • 4
                                            • 5
                                            • 6
                                            • 7
                                            • 8
                                            • 12
                                            • 13
                                            • 6 / 13
                                            • First post
                                              Last post