troubleshooting

One of the more perplexing issues Linux administrators encounter is the inability to create or edit files despite having plenty of available disk space. If you’ve ever faced this problem, you may have found yourself in a troubleshooting loop:

1. You attempt to create or modify a file, but receive a “disk full” error
2. You check available space with df -h and see plenty of free space
3. You verify you have the correct permissions, perhaps even as root
4. The error persists despite all conventional troubleshooting steps

This situation often points to a lesser-known but critical resource limitation in Linux filesystems: inode exhaustion.

What Are Inodes?

Inodes (index nodes) are fundamental data structures in Linux and Unix-like filesystems. Each inode stores metadata about a file, including:

• File size
• Owner and group IDs
• Access permissions
• Timestamps (creation, modification, access)
• File type
• Location of the file’s data blocks on disk

Critical concept: Every file on a Linux filesystem requires exactly one inode, regardless of the file’s size. This means a 1 byte file consumes one inode, and a 10 GB file also consumes one inode.

The Inode Exhaustion Problem

When a filesystem is created, it is allocated a fixed number of inodes based on the filesystem size and configuration. If all inodes are used, no new files can be created—even if there’s abundant disk space available.

This situation typically arises in environments with:

• Large numbers of very small files
• Mail servers (each email is typically stored as a separate file)
• Cache directories
• Source code repositories
• Log directories with extensive rotation
• Web servers hosting many small assets
• Improper partition sizing during setup

Diagnosing Inode Exhaustion

The tool to check inode usage is the same df command you’re familiar with, but with the -i flag (for inodes):

df -i

The output will look something like this:

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      1310720  1310720       0  100% /
/dev/sdb1      2621440   124672 2496768    5% /home
tmpfs           506716       1  506715    1% /dev/shm

In this example, the root partition (/dev/sda1) shows 100% inode usage despite potentially having free disk space. This is the classic symptom of inode exhaustion.

Common Solutions for Inode Exhaustion

Immediate Relief

1. Identify and clean up directories with many small files:

   find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n

   This command helps identify which directories contain the most files.
2. Remove temporary or cache files:

   rm -rf /tmp/*

   (Be careful with removal commands, especially with recursive options)
3. Compress multiple small files into archives where appropriate:

   tar -czf logs_archive.tar.gz /var/log/old_logs/

Long-term Prevention

1. Plan filesystem allocation with inodes in mind:
   • Partitions that will store many small files need more inodes
   • Separate partitions for directories with different file size characteristics
2. Monitor inode usage regularly: Add inode checks to your system monitoring:

   watch -n 60 "df -i"

3. Implement log rotation with compression: Ensure logs are not only rotated but also compressed to reduce the number of files.
4. Resize filesystems with appropriate inode density: For ext2/3/4 filesystems, you can specify inode settings during creation:

   mkfs.ext4 -i 16384 /dev/sdX

   This sets one inode per 16KB of space instead of the default (typically one per 4KB or 8KB).

Special Case: Temporary Filesystems

As mentioned in the original problem statement, the /dev/shm tmpfs partition is particularly susceptible to inode exhaustion when resized. This happens because when you increase the size of a tmpfs filesystem, the number of inodes doesn’t automatically scale with it.

To adjust inodes on a tmpfs mount, you can remount it with the nr_inodes parameter:

mount -o remount,nr_inodes=1000000 tmpfs /dev/shm

You can also make this change permanent by updating your /etc/fstab:

tmpfs /dev/shm tmpfs defaults,size=2G,nr_inodes=1000000 0 0

Conclusion

Inode exhaustion is often overlooked in system administration until it causes problems. By understanding how inodes work and monitoring their usage alongside disk space, you can prevent mysterious “disk full” errors that occur despite showing available space.

Remember, a well-planned filesystem considers not just the total storage requirements, but also the expected number and size distribution of files. Regular monitoring of both disk space (df -h) and inode usage (df -i) should be part of your standard system maintenance routine.

In a future post, I’ll explore more detailed strategies for resizing and optimizing inode allocation, particularly for tmpfs partitions where this issue commonly arises.