Someone doesn't like local storage for large amounts of data
-
@olivier said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller I was talking about the architecture.
Do you mean that if you have massive file serving needs that it makes the most sense to have that on unique hardware that is then shared to other VMs? That makes sense, but seems like a standard capacity algorithm solution rather than a special case. Any workload that becomes dramatically unbalanced from the others would operate in that way.
-
@scottalanmiller Because that's not my point aaarrrgghh. I don't care, that's a not something I wanted to focus in my opinion at the first place.
-
I think that we are all lost at this point. Maybe start over and word it fresh. What @Dashrender had in the original post was not at all what you had said.
I thought that you meant physical file server was better than a virtual one, but that wasnt it either.
I don't know what was originally said that prompted the conversation so only working from what is in the thread.
-
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
-
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Yup, that's it. Because a lot of file storage will mean a large VDI, which is "dangerous".
-
@olivier said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Yup, that's it. Because a lot of file storage will mean a large VDI, which is "dangerous".
Okay, that part makes sense
-
\o/
-
I'm blaming @Dashrender for that one.
-
I'm blaming myself for doing multiple things at once. Got a trip early tomorrow, so I'm going to bed
See ya! -
@olivier said in Someone doesn't like local storage for large amounts of data:
I'm blaming myself for doing multiple things at once. Got a trip early tomorrow, so I'm going to bed
See ya!Good night.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
-
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
I'm blaming @Dashrender for that one.
Whatever
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
-
@Dashrender said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
Use direct access to the storage rather than a VDI. The size of the VDI is the concern.
So Xen as an example, you can use a raw LVM partition for a VM rather than VDI file. This fixes the large VDI problem.
A typical setup would be to have one smaller VDI, say 20GB, for the OS and then a raw partition, say 30TB, for the files.
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@Dashrender said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
Use direct access to the storage rather than a VDI. The size of the VDI is the concern.
So Xen as an example, you can use a raw LVM partition for a VM rather than VDI file. This fixes the large VDI problem.
A typical setup would be to have one smaller VDI, say 20GB, for the OS and then a raw partition, say 30TB, for the files.
How does this make the situation any better? It still takes hours to migrate that data from one host to another. Does being raw somehow enable faster access to that data?
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@Dashrender said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
Use direct access to the storage rather than a VDI. The size of the VDI is the concern.
So Xen as an example, you can use a raw LVM partition for a VM rather than VDI file. This fixes the large VDI problem.
A typical setup would be to have one smaller VDI, say 20GB, for the OS and then a raw partition, say 30TB, for the files.
Why complicate things like that? Why not just make a 20GB LVM for the OS, and a 30TB LVM for the data?
-
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@Dashrender said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
Use direct access to the storage rather than a VDI. The size of the VDI is the concern.
So Xen as an example, you can use a raw LVM partition for a VM rather than VDI file. This fixes the large VDI problem.
A typical setup would be to have one smaller VDI, say 20GB, for the OS and then a raw partition, say 30TB, for the files.
Isn't this 2016 or did I read my calendar wrong?
-
@dafyre said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@Dashrender said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@FATeknollogee said in Someone doesn't like local storage for large amounts of data:
@scottalanmiller said in Someone doesn't like local storage for large amounts of data:
@olivier said in Someone doesn't like local storage for large amounts of data:
The thing, initially, was about having VMs with large VDIs. Which is for me not a good practice.
But if you need to store a large amount of data, it's better to connect to a remote file share in the VM and keep small system disks (excepts for db/web usage, which are not huge in general).
That's all.
edit: is it more clear now?
Let's see if I reword it correctly....
If your VM needs a lot of file storage.... then it is better to mount that from a file server rather than keeping it in the original VM?
Ok, I get that, but this goes against the "new fangled" HCI (call it what you want) use of local "attached" storage?
It doesn't not really. That's what caught @Dashrender it's more two things...
- Split up workloads to keep size down of individual loads
- Resort to raw storage when containerized storage gets too large and the above cannot be actioned
what does resort to raw storage mean?
Use direct access to the storage rather than a VDI. The size of the VDI is the concern.
So Xen as an example, you can use a raw LVM partition for a VM rather than VDI file. This fixes the large VDI problem.
A typical setup would be to have one smaller VDI, say 20GB, for the OS and then a raw partition, say 30TB, for the files.
Why complicate things like that? Why not just make a 20GB LVM for the OS, and a 30TB LVM for the data?
How does it complicate anything? Having two LVMs is just as complicated, if not moreso.
