What Is Eating CentOS Disk Space

scottalanmiller

Here is some sample output from a web server I happen to be logged into at the moment. I added the "2> /dev/null" and the "tail" portions to make it easier to read and use. Make sure you are root before doing this to make things easy.

[root@to-lnx-web /]# **whoami**
root
[root@to-lnx-web /]# **pwd**
/
[root@to-lnx-web /]# **du -smx * 2> /dev/null| sort -n | tail -n 5**
153     boot
403     tmp
554     lib
899     usr
6070    var
[root@to-lnx-web /]# **cd /var**
[root@to-lnx-web var]# **du -smx * 2> /dev/null| sort -n | tail -n 5**
70      tmp
73      spool
184     lib
1708    www
3957    log
[root@to-lnx-web var]# **cd log**
[root@to-lnx-web log]# **du -smx * 2> /dev/null| sort -n | tail -n 5**
316     httpd
413     maillog-20140223
627     maillog
1043    maillog-20140302
1267    maillog-20140309

scottalanmiller

From my output above, you can see that I started in / and found that var was the directory using the most space under it. So I moved into var and did it again. Under var we saw that log was using the most space. So we moved until log and ran it again.

The 2>/dev/null removes extraneous error output that you don't care about.

The sort -n | tail -n 5 portion shows you only the five largest files or directories from each run. You could adult the "5" to "8" or "12" or whatever is most useful to you.

ajin.c

root@trvbackup [/]# du -smx * | sort -n
^C
root@trvbackup [/]#

Waited arround half an hour ...but no output ....still waiting

scottalanmiller

If the drive is full, this will likely take some time. Because it is sorting the output it will show nothing until it completes.

ajin.c

Boss.....Still waiting for the output.......

ajin.c

root@trvbackup [/]# du -smx * | sort -n
du: cannot access proc/11877/task/11877/fd/4': No such file or directory du: cannot access proc/11877/task/11877/fdinfo/4': No such file or directory
du: cannot access proc/11877/fd/4': No such file or directory du: cannot access proc/11877/fdinfo/4': No such file or directory
0 proc
0 scripts
0 sys
1 backup
1 dev
1 lost+found
1 media
1 mnt
1 quota.user
1 razor-agent.log
1 selinux
1 srv
3 tmp
7 bin
8 root
14 sbin
29 etc
30 lib64
38 opt
43 boot
234 lib
5401 usr
17480 var
148041 home

scottalanmiller

This is easy. It's someone storing stuff in their home directory. This is not a system problem but a user problem. Just just the same command but with /home instead of just / and it will produce the list of your offending users.

scottalanmiller

That is 148GB of user data.

ajin.c

root@trvbackup [/home]# du -smx * | sort -n

right ?

scottalanmiller

@ajin.c said:

root@trvbackup [/home]# du -smx * | sort -n

right ?

Correct

ajin.c

Hi SAM,

since the server was down , i had to install and configure a new one. i will come back as soon as the temperory issues are sorted out .

scottalanmiller

In the future, you might want to consider separating the /home directory out into its own filesystem so that end users cannot impact the system in this way. Or using quotas to limit how much damage that they can do.

scottalanmiller

@ajin.c said:

root@trvbackup [/home]# du -smx * | sort -n

right ?

I just noticed from you df -h above, /home is already a separate logical volume. That is not the problem. The issue is that your /var is too big. Run this instead...

du -smx /var/ 2> /dev/null | sort -n | tail -n 5*

ajin.c

Hi Sam,

I had mounted a 2 TB hdd on my server, when i had this issue ...
@ arround 10 Am IST server got stuck. And i started building new one ...................Removed the HDD and mounted to the new one.

ajin.c

As soon as i un mounded the External hdd ., i tried df -h
and got the output..........

root@trvbackup [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_trvbackup-lv_root
50G 28G 19G 60% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 53M 407M 12% /boot
/dev/mapper/vg_trvbackup-lv_home
402G 145G 236G 39% /home
/usr/tmpDSK 1.6G 38M 1.5G 3% /tmp

That means 19 gb free as soon as i unmounted my External Hdd.

ajin.c

And please find the output of this without The external Hdd

root@trvbackup [~]# du -smx /var/* 2> /dev/null | sort -n | tail -n 5
4 /var/tmp
30 /var/cache
377 /var/cpanel
5323 /var/log
17030 /var/lib
root@trvbackup [~]#

scottalanmiller

@ajin.c said:

As soon as i un mounded the External hdd ., i tried df -h
and got the output..........

root@trvbackup [~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_trvbackup-lv_root
50G 28G 19G 60% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 485M 53M 407M 12% /boot
/dev/mapper/vg_trvbackup-lv_home
402G 145G 236G 39% /home
/usr/tmpDSK 1.6G 38M 1.5G 3% /tmp

That means 19 gb free as soon as i unmounted my External Hdd.

That would be coincidental. Possibly a process was running and holding open files. Removing the drive might have killed the process releasing the storage for the filesystem to clean up.

scottalanmiller

@ajin.c said:

And please find the output of this without The external Hdd

root@trvbackup [~]# du -smx /var/* 2> /dev/null | sort -n | tail -n 5
4 /var/tmp
30 /var/cache
377 /var/cpanel
5323 /var/log
17030 /var/lib
root@trvbackup [~]#

Yes, that supports my theory. It appears to have cleaned up by the time that you ran this the first time.

ajin.c

So Is there any way to figure it out , so that i can take necessary steps for the next time..

scottalanmiller

@ajin.c said:

So Is there any way to figure it out , so that i can take necessary steps for the next time..

Most important thing is to have monitoring watching disk space on a regular basis so that you can see if it starts to fill up and get an alert. Every business has this issue. Logging or any number of things can cause it to fill up so having a way to keep tabs on it is always important.