Fedora 26 No Space Left on Device with Plenty of Space Available
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
Usually comes from orphaned INodes. Can happen if the process halts or server is shutdown incorrectly. This is what the lost+found directory is for.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
So my question is: Why did a reboot fix this? @scottalanmiller's available inodes should be exactly the same, unless he has a container running that is creating and deleting files very frequently
It did not. Deleting millions of unneeded files did.
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems weird to me that this can happen. This inodes thing is completely new to me. I had to look it up, and know almost nothing about it. Seems like a major Linux turn-off if this happens for no reason.
Maybe I need a good explanation, if someone doesn't mind.
It's one of those weird things that on tests and in classrooms people talk about constantly as if it is an every day thing. But in the real world, with thousands of servers over decades of use, I've actually had this happen now... once. I am an idiot for not checking that straight away, but it is so insanely rare in the real world that it never occurred to me that it was a real possibility. Especially when the system had no reason to be making so many files.
-
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
-
@tim_g said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Seems like a major Linux turn-off if this happens for no reason.
It's not Linux related. It's filesystem related. And it is so insanely rare that it's not a thing to worry about in the real world. Windows has inodes too, they call them File ID.
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
-
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
-
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
Empty I think. Tons were just folders.
-
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? More files = more inodes being used
-
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant. Pay attention to what is said.
-
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
-
Ya I've never run into it in the real world only on trick questions
-
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Ya I've never run into it in the real world only on trick questions
That's why I was so surprised. This never happens!
-
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?
People writing bad scripts that use lock files and forget to remove them.
That wouldn't create enough to do this though.
-
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
Ya I've never run into it in the real world only on trick questions
That's why I was so surprised. This never happens!
I understand. I've done a lot on Linux over the years and I've never run into anything having to do with inodes. That's why I was clueless and had to ask.
-
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?
People writing bad scripts that use lock files and forget to remove them.
That wouldn't create enough to do this though.
If you start using configuration management tools to manage infrastructure with code you get the chance to see some of these one-off oddities in the wild a little more frequently than you'd expect. Like having Java developers not use Java's log facilities to manage log rotation, and then having a generic log rotation configuration completely bork things by delete application logs that are still being accessed by the Java application.
I got to see this issue a few dozen times a few months ago before another of our Engineers disassociated the Java applications from our generic log rotation recipe.
Rebooting was the quick fix for us prior to fixing the actual problem.
-
@ramblingbiped said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@stacksofplates said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@jaredbusch said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@travisdh1 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@wirestyle22 said in Fedora 26 No Space Left on Device with Plenty of Space Available:
@scottalanmiller I could've sworn there was a post here related to a reboot solving it
Someone mentioned rebooting to try to solve it. But it did not (had already done that.) It was real files causing the issue, nothing orphaned. Literally, we were making 30K files an hour or so.
Tiny files to, right?
That is not relevant except for that fact that large files would have filled drive space and likely been noticed.
How is that not relevant? more files = more inodes being used
Size of the files is not relevant.
It actually makes sense that @scottalanmiller said it was mostly directories. Files of any size will almost always run out of drive space before inodes run out in 99.9999% of situations. This is the first time I've actually heard of this happening, ever.
Only not relevant for those who actually know about inodes already. The only reason I even know about them is they got mentioned in SGI's IRIX Sysadmin courses.
The one major exception is marker files. "touch thishappened" as a file automatically and never clean up and you are using inodes without using any space. That's who you can easily learn about inode depletion. But who does that?
People writing bad scripts that use lock files and forget to remove them.
That wouldn't create enough to do this though.
If you start using configuration management tools to manage infrastructure with code you get the chance to see some of these one-off oddities in the wild a little more frequently than you'd expect. Like having Java developers not use Java's log facilities to manage log rotation, and then having a generic log rotation configuration completely bork things by delete application logs that are still being accessed by the Java application.
I got to see this issue a few dozen times a few months ago before another of our Engineers disassociated the Java applications from our generic log rotation recipe.
Rebooting was the quick fix for us prior to fixing the actual problem.
Oh wow. Everything I have is managed with Ansible ( I can't even log in to servers), however all of the devs use Oracle APEX on separate systems so they don't really touch anything on my stuff. I'm sure I'd have it much worse if I had to manage their stuff.
