Troubleshooting Full Filesystem or Inode Errors

Error messages may appear on the console or main system log indicating that the hard drive in the firewall is full, such as:

/: write failed, filesystem is full
/: create/symlink failed, no inodes free
Warning: session_start(): open(/tmp/sess_XXXXXX, O_RDWR) failed: No space left on device

The typical cause of such errors is rarely that the drive is full, but that the operating system is unable to contact the drive. In short, the disk (HDD, SSD, CF, etc) is dead or dying.

In cases when the drive is dying, the OS tries to write to the drive, and receives back an error code that there aren’t any inodes left. Typically other messages about the underlying problem are logged to the console but not passed back to PHP so they can be seen by the GUI. Usually those refer to things like g_vfs_done or Troubleshooting DMA and LBA Errors.

In order to eliminate the possibility that the drive is actually full, such as from a package that went crazy eating up space, try to get to a shell from the console or ssh, and run the following command:

: df -hi

The output shows disk space usage for both capacity and inodes, using human-readable numbers. The System Information widget on the Dashboard also shows the usage for all mounted partitions. The following output is from a system running with UFS:

Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/5ec430415d66e7cc     29G    1.9G     25G     7%     25k  4.0M    1%   /
devfs                          1.0K    1.0K      0B   100%       0     0  100%   /dev
/dev/md0                       3.4M    120K    3.0M     4%      43   979    4%   /var/run
devfs                          1.0K    1.0K      0B   100%       0     0  100%   /var/dhcpd/dev

Note

The devfs lines do NOT indicate an actual problem; The devfs filesystem is virtual and used for housing device nodes not for files.

Of special concern is the space on /var and /tmp on systems configured to use RAM disks for those directories. Large files, such as an abnormally large DHCP leases file, can in fact fill up the /var memory disk and that is one way to encounter the problem.

If the root (/) slice has space and inodes remaining, and so do /var and /tmp and so on, then the problem is most likely a failing disk.

If disk space has been exhausted, find a way to free it up. This usually involves uninstalling or removing packages or changing the settings so they use less space.

If the disk is failing, swap it out as soon as possible.

Often when the drive is failing or full the system will continue routing packets indefinitely until it would need to access the hard drive, at which point it would quit. Users have reported firewalls running for months with a dead drive unnoticed before, though of course that is not advisable.

Another possible cause is mild filesystem corruption, which could be helped by Troubleshooting Disk and Filesystem Issues.