server management

Troubleshooting “Disk Full” Errors Despite Available Space: The Linux Inode Problem

By in Blog Tags: , , , , , , , , ,

One of the more perplexing issues Linux administrators encounter is the inability to create or edit files despite having plenty of available disk space. If you’ve ever faced this problem, you may have found yourself in a troubleshooting loop:

  1. You attempt to create or modify a file, but receive a “disk full” error
  2. You check available space with df -h and see plenty of free space
  3. You verify you have the correct permissions, perhaps even as root
  4. The error persists despite all conventional troubleshooting steps

This situation often points to a lesser-known but critical resource limitation in Linux filesystems: inode exhaustion.

What Are Inodes?

Inodes (index nodes) are fundamental data structures in Linux and Unix-like filesystems. Each inode stores metadata about a file, including:

  • File size
  • Owner and group IDs
  • Access permissions
  • Timestamps (creation, modification, access)
  • File type
  • Location of the file’s data blocks on disk

Critical concept: Every file on a Linux filesystem requires exactly one inode, regardless of the file’s size. This means a 1 byte file consumes one inode, and a 10 GB file also consumes one inode.

The Inode Exhaustion Problem

When a filesystem is created, it is allocated a fixed number of inodes based on the filesystem size and configuration. If all inodes are used, no new files can be created—even if there’s abundant disk space available.

This situation typically arises in environments with:

  • Large numbers of very small files
  • Mail servers (each email is typically stored as a separate file)
  • Cache directories
  • Source code repositories
  • Log directories with extensive rotation
  • Web servers hosting many small assets
  • Improper partition sizing during setup

Diagnosing Inode Exhaustion

The tool to check inode usage is the same df command you’re familiar with, but with the -i flag (for inodes):

df -i

The output will look something like this:

Filesystem      Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1      1310720  1310720       0  100% /
/dev/sdb1      2621440   124672 2496768    5% /home
tmpfs           506716       1  506715    1% /dev/shm

In this example, the root partition (/dev/sda1) shows 100% inode usage despite potentially having free disk space. This is the classic symptom of inode exhaustion.

Common Solutions for Inode Exhaustion

Immediate Relief

  1. Identify and clean up directories with many small files:
    find / -xdev -type f | cut -d "/" -f 2 | sort | uniq -c | sort -n
    This command helps identify which directories contain the most files.
  2. Remove temporary or cache files:
    rm -rf /tmp/*
    (Be careful with removal commands, especially with recursive options)
  3. Compress multiple small files into archives where appropriate:
    tar -czf logs_archive.tar.gz /var/log/old_logs/

Long-term Prevention

  1. Plan filesystem allocation with inodes in mind:
    • Partitions that will store many small files need more inodes
    • Separate partitions for directories with different file size characteristics
  2. Monitor inode usage regularly: Add inode checks to your system monitoring:
    watch -n 60 "df -i"
  3. Implement log rotation with compression: Ensure logs are not only rotated but also compressed to reduce the number of files.
  4. Resize filesystems with appropriate inode density: For ext2/3/4 filesystems, you can specify inode settings during creation:
    mkfs.ext4 -i 16384 /dev/sdX
    This sets one inode per 16KB of space instead of the default (typically one per 4KB or 8KB).

Special Case: Temporary Filesystems

As mentioned in the original problem statement, the /dev/shm tmpfs partition is particularly susceptible to inode exhaustion when resized. This happens because when you increase the size of a tmpfs filesystem, the number of inodes doesn’t automatically scale with it.

To adjust inodes on a tmpfs mount, you can remount it with the nr_inodes parameter:

mount -o remount,nr_inodes=1000000 tmpfs /dev/shm

You can also make this change permanent by updating your /etc/fstab:

tmpfs /dev/shm tmpfs defaults,size=2G,nr_inodes=1000000 0 0

Conclusion

Inode exhaustion is often overlooked in system administration until it causes problems. By understanding how inodes work and monitoring their usage alongside disk space, you can prevent mysterious “disk full” errors that occur despite showing available space.

Remember, a well-planned filesystem considers not just the total storage requirements, but also the expected number and size distribution of files. Regular monitoring of both disk space (df -h) and inode usage (df -i) should be part of your standard system maintenance routine.

In a future post, I’ll explore more detailed strategies for resizing and optimizing inode allocation, particularly for tmpfs partitions where this issue commonly arises.

Remounting Linux Filesystems on the Fly: A Quick Reference Guide

By in Blog Tags: , , , , , , , , ,

One of the many advantages of Linux over other operating systems is its flexibility when it comes to filesystem management. A particularly useful capability is the ability to unmount and remount filesystems without requiring a system reboot. This can be invaluable when you need to make configuration changes or perform maintenance tasks while minimizing downtime.

Basic Unmounting and Remounting Procedure

The process of remounting a filesystem “on the fly” is quite straightforward. For example, to work with a tmpfs filesystem:

  1. First, unmount the filesystem:
    umount tmpfs
    This command tells the operating system to unmount the filesystem identified as “tmpfs” in the /etc/fstab file.
  2. Then, remount the filesystem:
    mount tmpfs
    This command will mount the filesystem again, applying any configuration changes you’ve made in the /etc/fstab file.

Potential Challenges and Solutions

When attempting to unmount a filesystem, you might encounter error messages if the filesystem is currently in use. Common scenarios that can prevent unmounting include:

  • Having an SSH session with your current directory inside the filesystem
  • Files being open and in use by running processes
  • System processes accessing the filesystem

If you receive an error indicating that the filesystem is busy, you’ll need to:

  1. Exit any directories within the filesystem
  2. Close any open files on the filesystem
  3. Stop any processes that might be using files on the filesystem

Once you’ve addressed these dependencies, you can try the unmount command again, and it should succeed.

Real-World Applications

There are several practical scenarios where you might need to remount a filesystem:

1. Changing Filesystem Parameters

After modifying filesystem parameters in /etc/fstab, such as adjusting the size of a tmpfs partition (as described in my previous post about resizing RAM disks), you’ll need to remount the filesystem to apply the changes.

2. Changing Mount Options

You might need to add or modify mount options such as:

  • noexec (prevent execution of binaries)
  • nosuid (ignore SUID/SGID bits)
  • ro or rw (read-only or read-write access)
  • noatime (don’t update access times)

3. Filesystem Maintenance

Some maintenance tasks require remounting with specific options, such as remounting a filesystem as read-only to run certain types of filesystem checks.

Advanced Remounting Options

For more complex situations, you can use the mount command with the -o remount option to change mount options without fully unmounting first:

mount -o remount,option1,option2 /mount/point

This approach is particularly useful for system directories that are difficult to fully unmount during operation.

Monitoring Filesystem Status

To verify the current mount status of your filesystems, use:

df -h

Or for more detailed information including mount options:

mount | grep filesystem_name

Conclusion

The ability to remount filesystems without a system restart is one of many features that make Linux an excellent choice for servers and systems requiring high availability. By understanding these basic commands, you can make configuration changes to your filesystems with minimal disruption to running services.

Remember that while this functionality is powerful, it should be used carefully, especially on production systems. Always ensure you understand the dependencies and potential impacts before unmounting any critical filesystem.

Migrating cPanel Accounts to a New cPanel Server: The Command Line Approach

By in Blog Tags: , , , , , , , , ,

When it comes to server administration tools, cPanel stands out as a reliable solution for managing multiple websites on a single server. Over the years, I’ve experimented with various control panels including Ensim, Webmin, Plesk, and Parallels (the latter two being particularly problematic in my experience). While cPanel generally performs admirably, there are occasional situations where expected functionality doesn’t behave as anticipated.

The Problem with Built-in Migration

One such scenario involves cPanel’s built-in account migration feature. This tool is designed to transfer accounts from one server to another, regardless of whether the source uses cPanel or another web hosting panel. While this feature often works seamlessly, I recently encountered an issue when attempting a cPanel-to-cPanel migration. The system would establish a connection but consistently fail to transfer the backup file.

The Command Line Solution

Fortunately, cPanel offers robust command line capabilities that provide a straightforward alternative to the graphical migration wizard. In fact, this manual approach is often simpler than using the wizard interface.

Step 1: Create an Account Backup on the Source Server

To begin, log in to the source server as the root user and execute:

/scripts/pkgacct username

Replace username with the actual cPanel account username you wish to migrate.

This command creates a compressed tar.gz archive in the /home directory containing everything from the specified account. This comprehensive package includes all website files, databases, email accounts, and configuration settings.

Step 2: Transfer the Backup to the Destination Server

Next, copy the generated tar.gz file to your new server. Depending on the account size, this transfer might take considerable time. You can use SCP, SFTP, or any secure file transfer method you prefer.

Step 3: Restore the Account on the Destination Server

Once the backup file is on the new server, log in as root and run:

/scripts/restorepkg username

This command extracts the backup and recreates the account on the destination server with all its original content and settings. If an account with the same username already exists, it will be overwritten.

Handling Special Cases: Dedicated IP Addresses

If the account you’re migrating requires a dedicated IP address (common for websites with SSL certificates), you’ll need a slightly modified command:

/scripts/restorepkg --ip=y username

This assigns the next available dedicated IP address from your server’s pool to the restored account. Before running this command, ensure you have free dedicated IPs available on the destination server.

Important Considerations

While this command line approach is straightforward and often more reliable than the wizard, there’s one critical factor to consider: version compatibility. For optimal results, ensure that both the source and destination servers run similar versions of cPanel. Significant version disparities may lead to migration failures or unexpected behavior in the restored account.

Conclusion

The command line method for migrating cPanel accounts offers a simple yet powerful alternative when the graphical interface encounters issues. With just three basic steps—backup, transfer, and restore—you can efficiently move accounts between cPanel servers without navigating the more complex migration wizard, which requires additional connection details and configuration.

This approach has consistently proven more reliable in my experience, especially when dealing with larger accounts or when troubleshooting migration failures through the standard interface.

Essential Exim Mail Server Commands for System Administrators

By in Blog Tags: , , , , , , , , ,

If you’re managing an Exim mail server, you’ll inevitably need to perform various maintenance and troubleshooting tasks. Over my nine years of server administration, I’ve found certain commands particularly useful. This guide shares those essential Exim commands that have consistently proven their value in managing mail servers effectively.

Monitoring Current Activity

To get a real-time snapshot of what Exim is currently doing:

exiwhat

This command provides an immediate overview of Exim’s current processes, showing you what the mail server is handling at that precise moment.

Analyzing the Mail Queue

Count Messages in the Queue

One of the first indicators of potential problems is an unusually large mail queue. To get a simple count of messages currently in the queue:

exim -bpc

For effective monitoring, run this command several times over a few hours to establish whether the queue is growing, which could indicate a delivery issue.

Detailed Queue Analysis

For a more comprehensive analysis of your mail queue, including size and age statistics:

exim -bp | exiqsumm

This powerful combination displays:

  • Total message count
  • Queue volume
  • Age of oldest and newest messages
  • Destination domains
  • Summary statistics

The “oldest message” and “volume” metrics are particularly useful for identifying problematic domains or persistent delivery issues.

Managing the Mail Queue

Process the Entire Queue

To trigger an immediate delivery attempt for all queued messages with verbose output:

exim -q -v

This instructs Exim to process the entire queue according to its standard delivery rules, with the -v flag providing detailed information about each delivery attempt.

Process Only Local Mail

If external mail is causing blockages, you can prioritize the delivery of local mail (such as system notifications):

exim -ql -v

This command attempts to deliver only messages addressed to local recipients, which can help clear important internal communications when external delivery is experiencing issues.

Clean Up Old Messages

To remove messages that have been stuck in the queue for extended periods (potentially spam or bounces):

exiqgrep -o 604800 -i | xargs exim -Mrm

This command finds and removes all messages older than 7 days (604,800 seconds). Adjust the time value as needed—it’s calculated as seconds per day (86,400) multiplied by the number of days.

Final Thoughts

While Exim offers many more specialized commands, these are the ones I’ve found most useful in day-to-day server administration. They provide a solid foundation for monitoring, troubleshooting, and maintaining your Exim mail server.

For specific issues or advanced configurations, Exim’s comprehensive documentation is an excellent resource. However, these commands should handle the vast majority of routine mail server management tasks you’re likely to encounter.

How to Use Crontab to Schedule Tasks in Linux: A Complete Guide

By in Blog Tags: , , , , , , , , ,

One of the most essential tools in a system administrator’s toolkit is crontab. This powerful utility allows you to schedule jobs (programs or scripts) to run automatically at specified intervals—a capability that can significantly enhance server management and automation.

What Can You Schedule with Crontab?

The versatility of crontab is remarkable. You can schedule a wide variety of tasks, including:

  • Restarting servers or individual services
  • Creating, modifying, or deleting files
  • Changing file permissions
  • Running batch programs or scripts
  • Performing regular backups
  • Executing maintenance tasks
  • Sending automated emails or reports

Understanding Cron: Key Facts

Before diving into usage, here are some important facts about cron:

  1. Evolution with Consistency: While cron has evolved since its inception, the basic principles remain the same—making it a timeless skill for system administrators.
  2. One-Minute Resolution: Cron can run jobs at most once per minute—this is the smallest time interval available.
  3. Efficient Execution: Modern cron implementations don’t continuously check for jobs every minute. Instead, they load crontab files into memory and only execute when necessary.
  4. Dynamic Reloading: When you edit a crontab file, the cron service automatically reloads it, eliminating the need to restart the service.
  5. Concurrent Execution: Cron will start new jobs according to schedule even if previous instances haven’t finished. This can potentially overload poorly configured systems, so careful planning is essential.

Editing Your Crontab

To edit your personal crontab file:

crontab -e

This command opens your crontab file in the default system editor (typically vim, nano, or another configured editor).

Crontab Syntax and Format

A typical crontab entry looks like this:

1 0 * * * shutdown -r now

This example would reboot the server (shutdown -r now) at one minute past midnight every day.

Understanding the Fields

Each crontab entry consists of five time fields followed by the command to execute:

# .---------------- minute (0 - 59) 
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ... 
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat 
# |  |  |  |  |
# *  *  *  *  * command_to_execute

The asterisk (*) is a wildcard that matches all possible values. When used in a field, it means “every” (every minute, every hour, etc.).

Common Crontab Examples

Here are some practical examples to help you get started:

Run a Script Every Day at 3:30 AM

30 3 * * * /path/to/your/script.sh

Run a Backup Every Monday at 2:15 AM

15 2 * * 1 /usr/local/bin/backup.sh

Run a Database Cleanup on the First of Every Month

0 0 1 * * /usr/local/bin/db_cleanup.sh

Run a Script Every Hour

0 * * * * /path/to/hourly_script.sh

Run a Script Every 15 Minutes

*/15 * * * * /path/to/frequent_script.sh

Capturing Output

When a cron job executes, any output it generates can be captured and redirected to a file. Simply add a redirection operator to your command:

0 0 * * * /path/to/script.sh > /var/log/script_output.log

This will save the standard output to the specified log file. For both standard output and error messages, use:

0 0 * * * /path/to/script.sh > /var/log/script_output.log 2>&1

Best Practices for Crontab

  1. Document Your Crontab: Add comments (lines starting with #) to explain what each job does.
  2. Use Absolute Paths: Always use full paths for both the commands and any files they reference.
  3. Test Commands First: Before adding a command to crontab, test it manually from the command line.
  4. Monitor Job Output: Redirect output to log files and check them regularly.
  5. Consider Alternatives: For very frequent tasks or those requiring more sophisticated scheduling, consider alternatives like systemd timers.

Conclusion

Crontab is a powerful tool that allows for robust task automation in Linux systems. By mastering its syntax and capabilities, you can significantly enhance your system administration efficiency, ensuring that routine tasks are handled automatically and consistently.

In future posts, I’ll cover alternative scheduling methods for situations where cron might not be the optimal solution, including writing custom scheduling services.

Adding a New Swap Partition in Linux: Step-by-Step Guide

By in Blog Tags: , , , , , , , , ,

Following my previous post on creating a swap file, I realized it would be worthwhile to discuss another approach: adding an entirely new swap partition. While swap files are excellent for emergency situations, a dedicated swap partition often provides better performance for long-term use.

Prerequisites

Before we begin, you’ll need:

  • A separate hard drive that is either not currently mounted or can be safely unmounted using the standard umount command
  • Root/sudo access to your system

For this guide, I’ll assume the device is /dev/sdc, but your system might use a different device name (such as /dev/sdb, /dev/sda, /dev/hda1, etc.).

Step-by-Step Process

  1. Ensure the drive is unmounted If the drive already contains a swap partition that’s in use, disable it first:
    swapoff /dev/sdc
    If the drive is clean and doesn’t have any swap partitions, you can skip this step.
  2. Access the partition manager Launch the parted utility to manage the disk partitioning:
    parted /dev/sdc
    This will load the parted prompt, which provides an interface for manipulating disk partitions.
  3. View existing partitions At the parted prompt, enter:
    print
    This will display details of any existing partitions and available free space on the disk.
  4. Create the new swap partition Once you’ve identified how much space you want to allocate (at minimum, I recommend matching your RAM size), create the partition with:
    mkpartfs part-type linux-swap start end
    Replace start and end with appropriate values shown from the print command. For example, if the starting point is 1024MB and you want a 4GB swap partition, your end point would be 5120MB (1024 + 4096).
  5. Exit parted Type quit to exit the parted utility. The new partition will have been assigned a number, such as sdc2 (or higher if other partitions already exist).
  6. Format the new partition for swap Format the newly created partition as swap space:
    mkswap /dev/sdc2
    (Remember to replace sdc2 with your actual partition name)
  7. Activate the swap partition Enable the swap partition with:
    swapon /dev/sdc2

Making the Change Permanent

If you want the system to automatically use this swap partition on every boot, you need to add it to the file system table:

  1. Edit the fstab file
    vi /etc/fstab
  2. Add the swap entry Press Insert to enter edit mode, then add this line to the bottom of the file:
    /dev/sdc2 swap swap defaults 0 0
  3. Save and exit Press Escape, then type :wq or press Shift+Z twice to save and quit.

Verifying the New Swap Space

To confirm that your new swap partition has been successfully added and is active, you can check the system’s swap status:

cat /proc/swaps

This will display all active swap spaces on your system, including your newly created partition.

Final Thoughts

Adding a dedicated swap partition is generally considered a more efficient long-term solution than using a swap file, especially for systems that regularly require swap space. The performance difference is modest on modern systems, but can be noticeable under heavy load.

Remember that while swap space is important for system stability, it’s not a replacement for adequate RAM. If your system is consistently using large amounts of swap space, consider upgrading your physical memory for better performance.

Adding Swap Space Without a Partition in Linux: A Quick Emergency Fix

By in Blog Tags: , , , , , , , , ,

Recently, we encountered a challenging issue on one of the servers I help administer. The server would occasionally max out its RAM usage, forcing it to rely on swap space. When both RAM and swap space were exhausted, the server would grind to a halt, requiring a manual restart.

The Problem: Growing Needs, Limited Resources

The server in question had been in operation for several years, starting with modest traffic but gradually growing as the websites it hosted gained popularity and their databases expanded. Despite this growth, it was still running with only 512MB of RAM—a limitation we’ve since addressed with an upgrade.

However, we needed an immediate solution before the hardware upgrade could be implemented. The server already had a swap partition, but creating another one wasn’t feasible without significant downtime, as all available hard drives were in active use.

The Solution: Adding a Swap File

The answer turned out to be creating a swap file rather than a swap partition. This approach allows you to add additional swap space without reconfiguring your disk partitions—perfect for an emergency fix. Here’s how to do it:

  1. Navigate to a directory with plenty of available space: 

    cd /home
  2. Create a swap file using /dev/zero: 
    bash
    dd if=/dev/zero of=swapfile bs=1024 count=4194304
    This creates an empty file called “swapfile” with a block size of 1024 bytes and 4,194,304 blocks, resulting in a 4GB file. (To calculate the size in GB: 4,194,304 ÷ 1024 ÷ 1024 = 4GB)
  3. Set appropriate permissions on the file: 

    chmod 600 swapfile
  4. Format the file as swap space: 

    mkswap swapfile
  5. Activate the swap file: 

    swapon swapfile

To verify that your new swap space is active, you can run:

cat /proc/swaps

This command displays all active swap spaces on your system, and you should now see your newly created swap file listed alongside any existing swap partitions.

Important Considerations

This solution doesn’t automatically activate on system boot, which was fine for our purposes since we planned to upgrade the RAM soon. If you need a more permanent solution, you would need to add an entry to your /etc/fstab file.

Remember that while this approach can prevent system crashes, it’s only a temporary workaround. Swap space, especially on a traditional hard drive, is significantly slower than RAM. For optimal performance, it’s always better to address the root cause by:

  1. Upgrading physical RAM
  2. Optimizing your application configurations
  3. Considering server upgrades if necessary

That said, when you need a quick fix to keep a server stable until more permanent solutions can be implemented, adding a swap file is a valuable technique to have in your system administration toolkit.

Backup Backup Backup

By in Blog Tags: , , , , , ,

An incident earlier this week made me review the backup systems in place for my wife’s company (iHelm Enterprises).

One of our clients doesn’t exclusively use us, and they ran into a major issue when another hosting provider lost a server—with no backups available. Worse still, the client didn’t regularly back up their own data, assuming that the hosting provider would have backups—which, of course, they didn’t.

Our Backup Strategy—And Its Problems

We’ve always been a bit unusual in that we routinely perform backups, but we don’t always do them daily. Our system has been in place for a while, but the process is… well, let’s say, a bit unorthodox:

  1. Server 1 performs its backup by transferring to Server 2.
  2. Server 2 then transfers both its own backup and Server 1’s backup to the next server.
  3. This continues until the final server, where the files are placed in a repository that only this server can access.
  4. A third-party provider then performs off-site backups from this repository.

After hearing about our client’s catastrophic data loss, I decided to test our backup system—just to be sure. And here’s what I found:

  • (a) Our backup repository was completely full, and I hadn’t received any notifications about it.
  • (b) Getting files off the backup system was a nightmare. Some files were huge, and the server that had access to the repository didn’t have enough space to handle large transfers.
  • (c) Some of the servers weren’t properly notifying me of backup failures, so I had no idea whether some files were complete or corrupted.

The New Backup System

Clearly, this wasn’t ideal, so I decided to rethink our approach.

Now, our servers transmit backups directly to our office computers. One of these computers is always on (24/7). This machine then uses Windows file sharing (yes, despite being a Linux machine!) to copy the backups to a new Network Attached Storage (NAS) device, which is stored securely in the office.

Immediate Benefits We’ve Seen

  • We can easily check what files are backed up on the NAS.
  • Restoring data is much simpler—no more complex file transfers!
  • Each server has a dedicated backup schedule, so if something goes wrong, we only need to check that specific server and re-run the backup for just that machine.

The Takeaway: Always Check Your Backups!

If you provide backups for clients or even just back up your own work, take some time to review your backup system every now and then.

Our system had been in place for over a year, and while it worked at the time, changes in technology (like the affordability of NAS devices) meant that better solutions were available.

From now on, my new mantra is:
Backup – Backup – Backup – Check – Review – Backup – Backup – Backup – Check – Review!

Copyright Michael Helm 2026.
The earliest version of this site available is from October 2003 - possibly viewable here: https://web.archive.org/web/20031228013629/http://www.ihelm.org.uk:80/

↑ Back to top