Dell PERC Question (Server Down)

@scottalanmiller said

Sure, what difference does the boot speed make realistically?

I was theorizing ... none.

And the initial update is a PITA, but that's a one time thing.

If it ads ten minuets or something, sure. That could be a conceivable issue for a lot of systems. That's a ten minute risk. But generally it's like an extra 10 - 60 seconds kind of thing. Hard to have that end up being a risk, especially when we are talking it only coming up during a pretty major event. It's not like an old server without virtualization that would reboot weekly. These tend to stay up for a season.

BRRABill

Well, over the weekend I got all the important data off this array, and this morning I TRASHED IT!

I unplugged all the drives, then re-plugged in all DELL-only drives.

The arrays built, initialized, and thus far seem fine.

I guess there is no way to test except actually putting the data back on there, so that will be the next step.

Oh, and prayer.

DustinB3403

Wish you the best.

BRRABill

@DustinB3403 said

Wish you the best.

Thanks!

BRRABill

When I plugged the DELL drives in last week, two of the three of them failed.

Today, those same DELL drive, in the same slots, are all working fine.

I wonder if there really was something corrupted in the array because of the other drives. DELL mentioned that as a good possibility ... but is that really possible?

But so far it seems like the PERC and the backplane are working fine.

StrongBad

What's the latest on this cluster (pun intended.)

BRRABill

I have not put any more data back on there yet, but the server has been up (with the array intact) for 24 hours now. Seems to be no issues, but the array had no issues for months until I started using it in production, so we'll have to wait and see. (Assuming it was the extra activity that got it.)

travisdh1

@BRRABill said in Dell PERC Question (Server Down):

I have not put any more data back on there yet, but the server has been up (with the array intact) for 24 hours now. Seems to be no issues, but the array had no issues for months until I started using it in production, so we'll have to wait and see. (Assuming it was the extra activity that got it.)

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:

dd if=/dev/random of=/dev/array bs=4k count=1000000000

will write 4gb of random stuff to the array.

BRRABill

As part of this discussion about booting off of USB, I decided that since I needed to redo my entire XS installation anyway, I would install to USB.

But as I was doing a little research, I found online people recommending against this, since XS (and other hypervisors, too, I'm not sure) writes a lot to the boot disk. The fear is that this constant writing with fry a USB stick that doesn't have the smarter controllers that SSDs have. I spoke with @scottalanmiller a little bit about this, and he said it made sense, and to definitely get the logs off there.

So my question is ... how many of you that boot off USB have had issues with your USBs not booting? Do you agree this might be an issue? (Yes, I know that you can clone another USB, but that;s not really the point. )

And how many of you that boot XS off USB are sending your logs elsewhere?

BRRABill

@travisdh1 said

Is the array actually mounted somewhere? I'd try hammering it for a bit. IE:
dd if=/dev/random of=/dev/array bs=4k count=1000000000
will write 4gb of random stuff to the array.

The problem is that it was pretty random. I'm not even sure that kind of test would make me feel any better.

Dashrender

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

DustinB3403

@Dashrender said in Dell PERC Question (Server Down):

I only just started using XS, and it is installed on an SD card. The logs have not be moved yet.. but really, the logs should be move regardless to your central log catching server.

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

BRRABill

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

I did set that up. However, I noticed that the logs were still recording themselves on my boot device.

I Googled this, and apparently you have to go into /var/lib/syslog.conf and comment out all the local locations. And then you have to restart the syslog daemon ... which overwrites the config file. It does this on reboot, too.

Is that what you meant? If so, take a look in your logs directory and you'll probably see the same thing I saw ... even though you are forwarding, it is still writing to the boot drive. Check it out!

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

This is the article I am referencing:
http://xenserver.org/discuss-virtualization/virtualization-blog/entry/log-rotation-and-syslog-forwarding.html

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

@DustinB3403 said

You can even specify a default location (NFS / SMB) for your logs to get pushed too so they aren't on the boot device.

Do you mean be using the syslog forwarding in XC? (And also in the CLI?)

No, he just means mounting an NFS device at the logging location.

scottalanmiller

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

Dashrender

@scottalanmiller said in Dell PERC Question (Server Down):

@BRRABill said in Dell PERC Question (Server Down):

(Unless you meant actually moving the local logging location to another storage device which is also something I saw mentioned, but it much more entailed.)

It's actually very simple and standard. You just move the old log location from /var/log to /var/log_old. Then you make mkdir /var/log and then you mount an NFS share to that spot. That's all and then the logs go directly to the NFS device.

While Windows can do this, this is so rarely used to be more myth than anything