Troubleshooting Full Filesystem or Inode Errors¶
Error messages may appear on the console or main system log indicating that the hard drive in the firewall is full, such as:
/: write failed, filesystem is full /: create/symlink failed, no inodes free Warning: session_start(): open(/tmp/sess_XXXXXX, O_RDWR) failed: No space left on device
The typical cause of such errors is rarely that the drive is full, but that the operating system is unable to contact the drive. In short, the disk (HDD, SSD, CF, etc) is dead or dying.
In cases when the drive is dying, the OS tries to write to the drive, and receives back an error code that there aren’t any inodes left. Typically other messages about the underlying problem are logged to the console but not passed back to PHP so they can be seen by the GUI. Usually those refer to things like g_vfs_done or Troubleshooting DMA and LBA Errors.
In order to eliminate the possibility that the drive is actually full, such as from a package that went crazy eating up space, try to get to a shell from the console or ssh, and run the following command:
: df -hi
The output shows disk space usage for both capacity and inodes, using human-readable numbers. The System Information widget on the Dashboard also shows the usage for all mounted partitions. The following output is from a system running with UFS:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on /dev/ufsid/5ec430415d66e7cc 29G 1.9G 25G 7% 25k 4.0M 1% / devfs 1.0K 1.0K 0B 100% 0 0 100% /dev /dev/md0 3.4M 120K 3.0M 4% 43 979 4% /var/run devfs 1.0K 1.0K 0B 100% 0 0 100% /var/dhcpd/dev
The devfs lines do NOT indicate an actual problem; The devfs filesystem is virtual and used for housing device nodes not for files.
Of special concern is the space on
/tmp on systems configured
to use RAM disks for those directories. Large files, such as an abnormally large
DHCP leases file, can in fact fill up the
/var memory disk and that is one
way to encounter the problem.
If the root (
/) slice has space and inodes remaining, and so do
/tmp and so on, then the problem is most likely a failing disk.
If disk space has been exhausted, find a way to free it up. This usually involves uninstalling or removing packages such as squid, or changing the settings so they use less space.
If the disk is failing, swap it out as soon as possible.
Often when the drive is failing or full the system will continue routing packets indefinitely until it would need to access the hard drive, at which point it would quit. Users have reported firewalls running for months with a dead drive unnoticed before, though of course that is not advisable.
Another possible cause is mild filesystem corruption, which could be helped by Troubleshooting Disk and Filesystem Issues.