OS/Filesystem for file server ~ 8Tb

Doyler3000

£600 will keep me in beer for quite a while. I don't drink like I used to

So with Centos as the VM, would you use LVM snapshotting? ZFS on linux and BTRFS are not quite production ready seems to be a common (possibly wrong) consensus. There would be complaints if we lost the file system snapshotting that ZFS allows.

coliver

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

£600 will keep me in beer for quite a while. I don't drink like I used to

So with Centos as the VM, would you use LVM snapshotting? ZFS on linux and BTRFS are not quite production ready seems to be a common (possibly wrong) consensus. There would be complaints if we lost the file system snapshotting that ZFS allows.

LVM will do snapshots at the filesystem level yes.

DustinB3403

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

£600 will keep me in beer for quite a while. I don't drink like I used to

So with Centos as the VM, would you use LVM snapshotting? ZFS on linux and BTRFS are not quite production ready seems to be a common (possibly wrong) consensus. There would be complaints if we lost the file system snapshotting that ZFS allows.

Use CentOS as the server, use Veeam for your backups. You get a bit more functionality with it, especially if you aren't familiar with the process to restore files using the backed in snapshot functionality.

Doyler3000

Yes I'd keep using Veeam for backups. I'd class the snapshotting and backups as different functions really. I'd be hoping to use the snapshots for restoring single files or directories, the backups more for restoring the file server in a disaster.

scottalanmiller

@DustinB3403 said in OS/Filesystem for file server ~ 8Tb:

With Essentials do you get the ability to backup your VM's without an agent? Or really any other benefit?

I thought it was Essentials Plus.

With Essentials, isn't that just to allow for more cores and memory?

Essentials Plus gives you HA and support. Essentials is no support, and all you get are backups.

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

and yes it's snapshotting at the file level I'd need to replicate the daily zfs snapshots we've got now.

Why replicate here rather than with Veeam? Veeam does this well. Are you using ZFS' internal syncing protocol?

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

Yes I'd keep using Veeam for backups. I'd class the snapshotting and backups as different functions really. I'd be hoping to use the snapshots for restoring single files or directories, the backups more for restoring the file server in a disaster.

That can make sense. Although I'd consider re-evaluating your Veeam usage instead. Veeam will do this in a single place, more powerfully than doing snaps inside the VM will.

This is really Veeam's strong suite.

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

£600 will keep me in beer for quite a while. I don't drink like I used to

So with Centos as the VM, would you use LVM snapshotting? ZFS on linux and BTRFS are not quite production ready seems to be a common (possibly wrong) consensus. There would be complaints if we lost the file system snapshotting that ZFS allows.

They are both production ready, but the big question is always... why look for niche solutions when the mature, normal ones are SO good at it? LVM and XFS are the mature choice for this for two decades.

ZFS has always been considered silly on Linux because snapshotting was always there, since the 1990s. ZFS doesn't offer new functionality there. It was new on Solaris, either. Solaris had snaps in the 1990s as well. Long before they added ZFS.

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

Without wanting to draw out the virtualisation platform questions anymore let me just say that the £600 or something we paid for essentials is not outrageous given that it's the platform

That's a very unhealthy way of looking at it. What you should ask is "what did you get for that money?" And the answer is "screwed."

In absolute terms, £600 is nothing for any viable business to spend. So it sounds reasonable when looked at in that way and if it was getting you valuable stuff, then absolutely without question, just spend it.

But we have to work in relative terms. For £600 what did you actually get? You got a very limited system without the features you would get for free from any other system, that's it. You are literally getting a negative value out of spending £600.

To put it another way, if ESXi Essentials was free instead of £600... we would still be having this conversation about choosing the only platform out there that doesn't provide a certain base set of features. The £600 is a red herring, it's that ESXi isn't up to snuff here, it's not a viable option. ESXi isn't really a consideration until you are talking about Essentials Plus for much, much more money and mostly that is because you are getting support. It's not that ESXi is bad, it's awesome. But you are getting screwed if you are trying to use it below the Essentials Plus tier.

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

We're almost entirely a linux operation here though we have a few windows clients.

Just have to mention that this fact makes ESXi a weird choice as you need a range of skills instead of being able to focus on the ones that you have. If you have Linux skills, you are ready for KVM or Xen

scottalanmiller

CentOS, Fedora, openSuse and Ubuntu are your choices here and are all fine. If you want to use ZFS, honestly I'd probably use Ubuntu.

If I was building from scratch what you want to do here, I'd likely use openSuse with XFS. All of the choices are fine.

scottalanmiller

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

I'm most familar with and

the people in the company who would have to step in if I wasn't around are most familiar with

Very valid, in a case like that, though, I'd think other factors would step in. Do you feel that if you were gone that you really still have the expertise to maintain any system (you meaning the other people in the company?) I would think that this would either trigger bringing in a backup ITSP that can help you with whatever platform make the most sense for you and solve this issue entirely. Or bring in someone like @Scale HC3 where the platform is all self managing and they are there to support anything that you need so that you need no expertise or knowledge of it at all. Or both, of course.

matteo nunziati

@Doyler3000 xfs+lvm or zfs. In any case consider an hba and passthru disks/hba to the vm making soft raid if you can manage it.

scottalanmiller

@matteo-nunziati said in OS/Filesystem for file server ~ 8Tb:

@Doyler3000 xfs+lvm or zfs. In any case consider an hba and passthru disks/hba to the vm making soft raid if you can manage it.

Definitely don't do that. Passthrough to a VM is terrible. Very complex and it bypasses loads of the critical abstraction.

scottalanmiller

It would also keep ESXi from being able to use those disks in any way as an additional problem. ESXi can't use software RAID in any form. So no way to even share them.

matteo nunziati

@scottalanmiller I was thinking about a dedicated array not something used by esxi.
Does passthru cause all these issues? I've used it some time ago and it was really nice. Anyway it was not esxi

scottalanmiller

@matteo-nunziati said in OS/Filesystem for file server ~ 8Tb:

@scottalanmiller I was thinking about a dedicated array not something used by esxi.
Does passthru cause all this issues? I've used it some time ago and it was relly nice. Anyway it was not esxi

Passthrough will do things like break backups and snapshotting, for example. It makes the system have two different consistency layers. Requires lots of complexity on the hardware side where you have to know which drives and physically handled in what way.

And then your RAID has to be handled in the VM rather than in the hardware, which isn't horrific, but it isn't great. That's stuff you want at super low latency that putting into a VM means there is more overhead. And things like restores, failovers... become very hard and complex itself of simply restoring from backup.

Doyler3000

@scottalanmiller Sorry, my choice of terminology didn't make things very clear there. By replicating I mean replicating the functionality rather than the snapshots themselves. At the moment if I developer deletes a file by mistake they can go to the .zfs folder in the root of their home and restore the file from one of the daily snapshots. I think if they were no longer able to do the equivalent of this, that I might get some resistance.

Doyler3000

@scottalanmiller said in OS/Filesystem for file server ~ 8Tb:

@Doyler3000 said in OS/Filesystem for file server ~ 8Tb:

We're almost entirely a linux operation here though we have a few windows clients.

Just have to mention that this fact makes ESXi a weird choice as you need a range of skills instead of being able to focus on the ones that you have. If you have Linux skills, you are ready for KVM or Xen

In my (shaky) defence I had stronger VMWare skills than Linux skills at the time

Doyler3000

Thanks all, I very much appreciate lots of great advice.
I've already got a XenServer installed and I'm going to take a look at Xen Orchestra now. As regards OS/Filesystems OpenSuse (or Centos) with XFS and LVM will get some testing in the near future.