Proxmox install for use with a ceph cluster
-
@scottalanmiller said in Proxmox install for use with a ceph cluster:
@DustinB3403 said in Proxmox install for use with a ceph cluster:
The argument of "why spend money on expensive hardware" also doesn't hold up because this equipment is all lab equipment that has no cost to acquire. Pulled it out of a decom and used it for this purpose.
The system is designed around paying customers, not people receiving the hardware later.
Of course, but when any administrator specifically goes into the controller and wipes the configuration and tries to boot the system, it immediately attempts to go back and recreate the very same array.
If I wanted an HP Server using JBOD that should also be an option, regardless if the system has hardware raid.
-
Which it is an option, but you have to fud around with the controller at start-up
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
Of course, but when any administrator specifically goes into the controller and wipes the configuration and tries to boot the system, it immediately attempts to go back and recreate the very same array.
Only if you don't make something else. it has to do "something", no matter what, there has to be some configuration. If it did anything else, we'd still be having this conversation. You wiped it, but didn't configure it, so it was in a situation of having to make a "judgement call" to try to help you and it made what is, far and away, the only reasonable choice other than stopping the boot completely and forcing you to manually decide - which given that you had already opted out of that, isn't a great choice.
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
If I wanted an HP Server using JBOD that should also be an option, regardless if the system has hardware raid.
That's a different discussion. That controller explicitly doesn't offer JBOD at all. By keeping the hardware in place, you have informed the hardware not to allow JBOD. If JBOD was your goal (which is totally different than wanting RAID 0), then wiping the controller isn't the right action, removing it is. It's a RAID controller, it's one purpose is to avoid JBOD.
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
Which it is an option, but you have to fud around with the controller at start-up
It's not, you only mimic JBOD in a bad way. It's not safe, you should remove the controller for better safety. but why?
-
Can that controller do raid 0 with one drive to do fake jbod?
-
@jt1001001 said in Proxmox install for use with a ceph cluster:
Can that controller do raid 0 with one drive to do fake jbod?
No, its also not worth bothering any more as the controller can still see the hardware controller when booted from USB.
So this lab experiment is over.
-
@jt1001001 said in Proxmox install for use with a ceph cluster:
Can that controller do raid 0 with one drive to do fake jbod?
Yes, but it blocks SMART so you never want to do it, it undermines the stability of the JBOD. There's always a standard controller on the MOBO for the JBOD connections.
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
No, its also not worth bothering any more as the controller can still see the hardware controller when booted from USB.
We could have told you that. True hardware RAID cannot be bypassed, it's physically in the path, if you cut it out, the drives have to vanish.
-
@scottalanmiller said in Proxmox install for use with a ceph cluster:
@DustinB3403 said in Proxmox install for use with a ceph cluster:
No, its also not worth bothering any more as the controller can still see the hardware controller when booted from USB.
We could have told you that. True hardware RAID cannot be bypassed, it's physically in the path, if you cut it out, the drives have to vanish.
Your snarky remarks aren't helping today.
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
@scottalanmiller said in Proxmox install for use with a ceph cluster:
@DustinB3403 said in Proxmox install for use with a ceph cluster:
No, its also not worth bothering any more as the controller can still see the hardware controller when booted from USB.
We could have told you that. True hardware RAID cannot be bypassed, it's physically in the path, if you cut it out, the drives have to vanish.
Your snarky remarks aren't helping today.
Not meant to be snarky, just explaining how it works. This way you don't have to test different scenarios, because just knowing it is hardware RAID tells you what you need to know. And knowing that you can remove the card completely and get the JBOD functionality you are seeking, is key.
-
@scottalanmiller I can remove the card for sure, but its not a practical lab exercise for what I'm working on.
I would do this in my personal lab possibly to do that, but not here, in this lab.
-
@DustinB3403 said in Proxmox install for use with a ceph cluster:
@scottalanmiller I can remove the card for sure, but its not a practical lab exercise for what I'm working on.
I would do this in my personal lab possibly to do that, but not here, in this lab.
You can just the change the controller to HBA mode. In HBA mode it will work like a HBA.
On older cards you have to flash the firmware, on newer cards it's often just a setting.
From a hardware perspective a RAID card is a HBA + more powerful hardware for parity calcs + larger memory cache.Hang on a sec see if I'll find the link on how to do it.
A newer way to set HP controller to HBA mode:
https://ahelpme.com/servers/hewlett-packard/smart-array-p440-enable-or-disable-hba-mode-using-smart-storage-administrator/This is a older longer way to do it:
Youtube Video -
@Pete-S said in Proxmox install for use with a ceph cluster:
@DustinB3403 said in Proxmox install for use with a ceph cluster:
@scottalanmiller I can remove the card for sure, but its not a practical lab exercise for what I'm working on.
I would do this in my personal lab possibly to do that, but not here, in this lab.
You can just the change the controller to HBA mode. In HBA mode it will work like a HBA.
On older cards you have to flash the firmware, on newer cards it's often just a setting.
From a hardware perspective a RAID card is a HBA + more powerful hardware for parity calcs + larger memory cache.Hang on a sec see if I'll find the link on how to do it.
A newer way to set HP controller to HBA mode:
https://ahelpme.com/servers/hewlett-packard/smart-array-p440-enable-or-disable-hba-mode-using-smart-storage-administrator/This is a older longer way to do it:
Youtube VideoI was just about to reply to turn passthrough mode (HBA) on on the controller but you nailed it! On the other hand, Proxmox works fine with hardware RAID. As a matter of fact, this is what the vendor themselves recommend: https://pve.proxmox.com/wiki/Raid_controller. Software ZFS RAID can potentially be faster but it needs to be configured properly with direct access to disks, plenty of RAM and ZIL for caching.
-
@taurex said in Proxmox install for use with a ceph cluster:
On the other hand, Proxmox works fine with hardware RAID. As a matter of fact, this is what the vendor themselves recommend: https://pve.proxmox.com/wiki/Raid_controller. Software ZFS RAID can potentially be faster but it needs to be configured properly with direct access to disks, plenty of RAM and ZIL for caching.
That's possible. I haven't played with it yet so I don't know.
-
@scottalanmiller said in Proxmox install for use with a ceph cluster:
Because it HAS to default to SOMETHING. If you wanted anything, you'd have selected it. So they default to what is safest and most common. Why pay for a hardware controller if you didn't have a use for it? The key features of a hardware controller are disabled with R0.
HPE doesn't sell an actual pass through HBA. All their HBA devices are duel use parts. (To be fair, the equivalent line from Broadcom/Avago like the 3008 in theory can be sold with a RAID 1 no cache support, but plenty of OEM's like Dell sell Pure HBA firmware (Sometimes called The IT firmware) such as the HBA 330. Now the Gen9's might of still offered controllers from both ODMs (by Gen10 though that was gone, and I'm pretty sure it was a lot earlier like gen7 where HPE last used Avago parts).
-
@scottalanmiller said in Proxmox install for use with a ceph cluster:
We could have told you that. True hardware RAID cannot be bypassed, it's physically in the path, if you cut it out, the drives have to vanish.
Some RAID controllers have a way of running pass through. For the 3108 based cards (Example P730) there is a pass through. Note historically it's been "pretty damn buggy", and it took a lot of joint engineering to get it stable enough for our purposes (don't even dare try it with the 2208 based 6Gbps Avago parts). Now this is kinda moot as everyone's using 3008 pure HBA firmware parts (We stopped certifying RAID controllers from Broadcom for pass through) and the other thing that's making it moot is NVMe running "proper" talks directly to the PCI-E bus. There are "Tri-Mode" RAID controllers that can raid NVMe but I just don't see a point. You bottleneck throughput pretty hard pretty fast.
-
@scottalanmiller said in Proxmox install for use with a ceph cluster:
Yes, but it blocks SMART so you never want to do it, it undermines the stability of the JBOD. There's always a standard controller on the MOBO for the JBOD connections.
Also blocks TRIM commands (not that I trust the Linux TRIM driver to ATA drives given how many one off exceptions to disable they've had to write).
Operationally it's messy because on a drive failure you have to go in with PERCLI etc, and rebuild the RAID 0s. We used to run this but it was just a royal pain in the ass.
