Storage for Video Files - Suggestions?
-
Looks like our newly-assembled marketing team is going to be upping the ante by doing some promotional videos. That means, I need a place to store all this stuff.
I am already planning on replacing our 2 existing R720XD servers (running esxi). Most likely, with R740XD servers early next year. They just told me that their current project is at least 500GB of critical data that needs to be backed up. Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
They have told me that they will be doing more of these, but we don't know what to expect, as this is the first real project.
Any suggestions on how to handle this influx of data? Should I be asking for more information from them? What questions would you ask?
-
You need to be given real definitions. They need to project appropriately so you can then go buy new hardware Monday.
Do not try and bandaid your existing gear until next year. If the business has the need now, then the business needs to purchase the gear to run the business.
-
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
-
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
9TB usable, 18TB raw. Used storage is about 6TB, plus the 500GB they are going to be uploading next week, so 6.5TB.
I have been trying to get people to go through old stuff, but it is hard. We also have tons of higher res photos of our products' components. The manager just quit, so I am hoping that we can get rid of a lot of the stuff he did. Still, that is maybe 200 GB.
We also have our marketing and catalog stuff that our graphic design department has created over the years. That is about 3TB of data. I have already asked them to look into stuff we no longer need. We will see what they can live without (probably not much).
-
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
9TB usable, 18TB raw. Used storage is about 6TB, plus the 500GB they are going to be uploading next week, so 6.5TB.
I have been trying to get people to go through old stuff, but it is hard. We also have tons of higher res photos of our products' components. The manager just quit, so I am hoping that we can get rid of a lot of the stuff he did. Still, that is maybe 200 GB.
We also have our marketing and catalog stuff that our graphic design department has created over the years. That is about 3TB of data. I have already asked them to look into stuff we no longer need. We will see what they can live without (probably not much).
This is exactly what I mean. This is data that does not need to be hot, but if you are going to keep it hot, then put it on slow storage.
Edit: Don’t ask. Tell. You are designing new servers. The existing shares should be moved during the server migration. Then they should work active projects on the fast storage. A script should then be setup to alert you to old files that can be moved.
-
@jaredbusch Got it. Currently, all my server storage is slow storage lol. Going forward, I would like to spring for the SSDs configuration with spinning rust as you outlined.
-
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
9TB usable, 18TB raw. Used storage is about 6TB, plus the 500GB they are going to be uploading next week, so 6.5TB.
I have been trying to get people to go through old stuff, but it is hard. We also have tons of higher res photos of our products' components. The manager just quit, so I am hoping that we can get rid of a lot of the stuff he did. Still, that is maybe 200 GB.
We also have our marketing and catalog stuff that our graphic design department has created over the years. That is about 3TB of data. I have already asked them to look into stuff we no longer need. We will see what they can live without (probably not much).
This is exactly what I mean. This is data that does not need to be hot, but if you are going to keep it hot, then put it on slow storage.
Edit: Don’t ask. Tell. You are designing new servers. The existing shares should be moved during the server migration. Then they should work active projects on the fast storage. A script should then be setup to alert you to old files that can be moved.
I have been looking at the FSRM reports in 2016 that can track the file access for this very reason.
-
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
9TB usable, 18TB raw. Used storage is about 6TB, plus the 500GB they are going to be uploading next week, so 6.5TB.
I have been trying to get people to go through old stuff, but it is hard. We also have tons of higher res photos of our products' components. The manager just quit, so I am hoping that we can get rid of a lot of the stuff he did. Still, that is maybe 200 GB.
We also have our marketing and catalog stuff that our graphic design department has created over the years. That is about 3TB of data. I have already asked them to look into stuff we no longer need. We will see what they can live without (probably not much).
This is exactly what I mean. This is data that does not need to be hot, but if you are going to keep it hot, then put it on slow storage.
Edit: Don’t ask. Tell. You are designing new servers. The existing shares should be moved during the server migration. Then they should work active projects on the fast storage. A script should then be setup to alert you to old files that can be moved.
I have been looking at the FSRM reports in 2016 that can track the file access for this very reason.
I had a simple power shell script for this but I have no idea where it went.
-
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch said in Storage for Video Files - Suggestions?:
@wrx7m said in Storage for Video Files - Suggestions?:
Between the 2 servers, I only have 9TB of usable space (OBR10). One server only has 1.1TB of free space and the other has 2TB.
First, you need to know what is eating all of your space. You “only” have 9TB usable? I’ve got no clients that actually have that kind of real usage once things were seriously looked into.
Do you really have 9TB of active storage? Or is it terabytes of stale data that the company seems important enough to have hot?
You can easily get any amount of slow storage to move stale data to that has no need to be hot.
Obviously your CPU needs are the other big thing. You have two servers now with two processors in each? Do you need that much?
Memeory doesn't matter most likely. You can put undgodly amounts of memory on each bus in new servers.
You can get 4x 6TB non-SSD drives and drop them in a RAID10 to handle storage for very low prices. That gives you 12TB of slow storage.
Then you can get some amount of SSD to fill the rest of the trays in a RAID 5 for the active storage.
9TB usable, 18TB raw. Used storage is about 6TB, plus the 500GB they are going to be uploading next week, so 6.5TB.
I have been trying to get people to go through old stuff, but it is hard. We also have tons of higher res photos of our products' components. The manager just quit, so I am hoping that we can get rid of a lot of the stuff he did. Still, that is maybe 200 GB.
We also have our marketing and catalog stuff that our graphic design department has created over the years. That is about 3TB of data. I have already asked them to look into stuff we no longer need. We will see what they can live without (probably not much).
This is exactly what I mean. This is data that does not need to be hot, but if you are going to keep it hot, then put it on slow storage.
Edit: Don’t ask. Tell. You are designing new servers. The existing shares should be moved during the server migration. Then they should work active projects on the fast storage. A script should then be setup to alert you to old files that can be moved.
Or you could use something like Enmotus... It works well and costs pennies.
-
Do you have drive bays to put disks in?
If you have, preferably 3.5", put two enterprise SATA or SAS drives in there. Dedicate those two drives to a VM and run software raid on it and use as a file server. Or make a raid 1 array and give that to the VM, if you have hardware raid on the VM host.
Absolutely no need for anything faster than 2 x 7200 rpm drives unless you are on a 10 GbE network. Then you need SSD storage.
Get something like two Seagate Exos X10 10TB or 12TB. SATA or SAS version are almost the same price. https://www.amazon.com/Seagate-256MB-Cache-Enterprise-ST10000NM0086/dp/B01LXXV880
You don't need anything more than raid 1 for small requirements like what you have. If you said 50TB or something like that, then it's time to look at raid 6.
PS. Also do the math how long it takes to move files. 1 GbE is roughly 100 MB/sec. It will be sustained sequential writes for large files.
-
@obsolesce said in Storage for Video Files - Suggestions?:
Enmotus
An option if the need to move things often. But it sounds like his environment could easily handle a script alerting him to old data that can be sent to the appropriate manager to get moved.
-
@jaredbusch - For this setup, I could just have 2 virtual disks presented on a single VM. One would be on the flash datastore, and the other would be on the slow datastore?
-
@wrx7m said in Storage for Video Files - Suggestions?:
@jaredbusch - For this setup, I could just have 2 virtual disks presented on a single server. One would be on the flash datastore, and the other would be on the slow datastore?
Pretty much, I would be more granular than that, but yes.
Edit: your hypervisor would have two drives mounts. Then make multiple disks as needed for the servers on the appropriate drives.
-
@jaredbusch Right. The datastores are at the hypervisor level. The vmdks would be on respective datastores and then in the VM, they would be also be shown as separate disks.
