Dell PERC Question (Server Down)

travisdh1

@BRRABill said in Dell PERC Question (Server Down):

I have not put any more data back on there yet, but the server has been up (with the array intact) for 24 hours now. Seems to be no issues, but the array had no issues for months until I started using it in production, so we'll have to wait and see. (Assuming it was the extra activity that got it.)

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:

dd if=/dev/random of=/dev/array bs=4k count=1000000000

will write 4gb of random stuff to the array.

BRRABill

As part of this discussion about booting off of USB, I decided that since I needed to redo my entire XS installation anyway, I would install to USB.

But as I was doing a little research, I found online people recommending against this, since XS (and other hypervisors, too, I'm not sure) writes a lot to the boot disk. The fear is that this constant writing with fry a USB stick that doesn't have the smarter controllers that SSDs have. I spoke with @scottalanmiller a little bit about this, and he said it made sense, and to definitely get the logs off there.

So my question is ... how many of you that boot off USB have had issues with your USBs not booting? Do you agree this might be an issue? (Yes, I know that you can clone another USB, but that;s not really the point. )

And how many of you that boot XS off USB are sending your logs elsewhere?

BRRABill

@travisdh1 said

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:
dd if=/dev/random of=/dev/array bs=4k count=1000000000
will write 4gb of random stuff to the array.

The problem is that it was pretty random. I'm not even sure that kind of test would make me feel any better.

Dashrender

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

DustinB3403

@Dashrender said in Dell PERC Question (Server Down):

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

BRRABill

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

I did set that up. However, I noticed that the logs were still recording themselves on my boot device.

I Googled this, and apparently you have to go into /var/lib/syslog.conf and comment out all the local locations. And then you have to restart the syslog daemon ... which overwrites the config file. It does this on reboot, too.

Is that what you meant? If so, take a look in your logs directory and you'll probably see the same thing I saw ... even though you are forwarding, it is still writing to the boot drive. Check it out!

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

This is the article I am referencing:
http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

No, he just means mounting an NFS device at the logging location.

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

Dashrender

@scottalanmiller said in Dell PERC Question (Server Down):

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

While Windows can do this, this is so rarely used to be more myth than anything

BRRABill

@Dashrender said

While Windows can do this, this is so rarely used to be more myth than anything

I used it once to set up a folder to my USB thumb drive.

BRRABill

@BRRABill said

This is the article I am referencing:
http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

I ended up trying the "dirty, dirty" trick in the comments tonight. Worked like a charm, and thankfully didn't blow anything up.

BRRABill

@BRRABill said

I ended up trying the "dirty, dirty" trick in the comments tonight. Worked like a charm, and thankfully didn't blow anything up.

Still not logging. Sweet.

I wonder why he said he didn't recommend doing this? I can't think of any reason. (Other than it gets re-written on updates.)

BRRABill

File this under the "you learn something new every day" category...

I have two SanDisk Cruzer Fit USB sticks I am running my two XSs off of.

The one has no LED, and the other flashes every 3 seconds.

Odd, I thought. I contacted SanDIsk, thinking perhaps one was broken or something.

This is the chart of "acceptable" behaviors, and they said it's possible two of the same drives bought at the same time could exhibit different behaviors.

LED is a solid light and blinks only when transferring files to the device.
LED blinks rapidly during initialization then turns off. LED then only blinks when data is being transferred.
LED breaths (slowly fades in and out) when device is not being used and blinks rapidly when data is being transferred.

scottalanmiller

Basically they are saying that the lights are meaningless and pointless and even they don't know why they are there.

StrongBad

Seems a bizarre waste of money for a light that doesn't have a purpose.

BRRABill

@scottalanmiller said

Basically they are saying that the lights are meaningless and pointless and even they don't know why they are there.

How dare you read something so cynical into such a well crafted three point list.

scottalanmiller

What's the latest on this project?

BRRABill

@scottalanmiller said in Dell PERC Question (Server Down):

What's the latest on this project?

Well, I still do not have production servers on it yet. (I was waiting for XS7 to come out.)

But I put a Splunk instance on it (as well as XO, and a few other things) and it hasn't had any issues yet. Of course it didn't with the EDGE drives until I really got things running on it, so we shall see. But nothing was writing to it like the Splunk machine is.

But so far, so good.