Greg Albrecht

Sometimes, in UNIX, you’ll come across this problem:

Your disk is full but you can’t seem to find the file (or files) that are taking up the space. df -k shows 100% disk usage for the file system, but when you get down to the directory level and start running du -sk you can’t seem to account for that 100% usage. What can you do?

FreeBSD (and probably other OS’) have a utility called fstat which will show active files (aka file descriptors) on each file system on your computer. Each open FD (file descriptor) will show these fields in fstat:

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W

For more info on what each of these fields mean, see the fstat man page (‘man fstat‘). Here’s an example output from fstat:

USER     CMD          PID   FD MOUNT      INUM MODE         SZ|DV R/W
root     tail         216    3 /var       8389 -rw-------  212254704  r

This shows us that the tail program has FD3 open on /var and is taking up 212MB of storage. Clearly this is a hog. We can kill 216 (the PID – Process ID – of tail), instantly freeing up space. Of course, you might not always want to kill the program that’s using up your space, restarting or HUP’ing is sometimes sufficient. For example, in apache‘s httpd using apachectl graceful is sufficient:

Gracefully restarts the Apache httpd daemon. If the daemon is not running, it is started. This differs from a normal restart in that currently open connections are not aborted. A side effect is that old log files will not be closed immediately. This means that if used in a log rotation script, a substantial delay may be necessary to ensure that the old log files are closed before processing them. This command automatically checks the configuration files as in configtest before initiating the restart to make sure Apache doesn’t die. This is equivalent to apachectl -k graceful.
- From “apachectl – Apache HTTP Server Control Interface“