Troubleshooting Disk Lifetime¶
An important part of keeping a firewall running reliably is to ensure its storage is in good condition.
If the disk in a firewall fails, it may continue to run in a reduced capacity until the system restarts. Exactly which parts may fail depends on the services and packages in use and what roles the firewall is performing. Packet filtering may continue to function indefinitely, but it may not be able to update rules, for example. Certain types of disks, such as SSD and eMMC disks, may fail into a read only state where disk writes fail or are discarded, but data can still be read.
Checking Disk Health & Lifetime¶
Contrary to popular rumors, the vast majority of modern flash storage found in SSD and eMMC drives is very resilient. Early SSDs were more prone to failure or only supported a comparatively small number of writes, but the technology has improved vastly over the years.
Traditional spinning platter hard drives have their own problems, such as failure of mechanical moving parts, which are harder to predict.
Over the lifetime of any disk, they typically will encounter failing spots and remap them to spares. This happens on traditional HDDs as well as SSDs. It’s normal to see a small number of these over time, but if the pool of spares is almost consumed, the disk should be replaced.
Depending on the hardware it may be possible to query the disk for information about its health. Not all platforms support each method, and disk OEMs track data in different ways. When in doubt, contact the manufacturer of the disk for details.
SSD¶
Many SSDs can report their health through S.M.A.R.T. data as described in S.M.A.R.T. Hard Disk Status as well as perform disk tests. Disks connected using SATA, mSATA, M.2, and other similar methods can typically use the same methods as traditional HDDs, but may have SSD-specific health information in their S.M.A.R.T. data. For example, some SSDs will include a life time estimate or media wear indication level.
NVMe disks require special handling, but can also be queried through S.M.A.R.T., though the data they report may be in a different format from typical SSDs.
eMMC¶
eMMC disks are unique in that they do not support S.M.A.R.T. but hardware which supports the correct revisions of the eMMC specification are capable of reporting health in their own way.
Install MMC Utilities¶
The first step is to install the mmc-utils package from an SSH or console
shell prompt:
# pkg install -y mmc-utils; rehash
Note
This package is currently only available on pfSense® Plus software and does not have a GUI component. It must be run from an SSH or console shell prompt.
Check MMC Health Status¶
To check the health of the first MMC disk, run the following command:
# mmc extcsd read /dev/mmcsd0rpmb
If the disk supports reporting its health, it should return output. For additional MMC disks, increase the device number in the command.
Interpreting MMC Health Data¶
The primary fields to look at in MMC health are the life time estimations and Pre-EOL estimation.
Note
Not all disks support all of these fields.
: mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x02
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
- Type A:
An estimate for life time of SLC (and pseudo-SLC) eraseblocks in steps of 10%.
- Type B:
An estimate for life time of MLC eraseblocks in steps of 10%.
- Type A and B Values:
The values of the A and B life time estimations are in 10% increments based on the hexadecimal value returned by the disk. This is only an estimate and the value can exceed 100%.
Possible values include:
Value
Meaning
0x00
Not defined
0x01
The disk has used 0%-10% of its estimated life time
0x02
The disk has used 10%-20% of its estimated life time
0x03
The disk has used 20%-30% of its estimated life time
0x04
The disk has used 30%-40% of its estimated life time
0x05
The disk has used 40%-50% of its estimated life time
0x06
The disk has used 50%-60% of its estimated life time
0x07
The disk has used 60%-70% of its estimated life time
0x08
The disk has used 70%-80% of its estimated life time
0x09
The disk has used 80%-90% of its estimated life time
0x0a
The disk has used 90%-100% of its estimated life time
0x0b
The disk has used 100%-110% of its estimated life time
Warning
This is only an estimation. Though useful as a general guideline, it does not necessarily indicate that a disk will fail at a given time.
- Pre-EOL:
Pre EOL information is an overall status for reserved blocks on the disks.
Possible values are:
Value
Severity
Meaning
0x00
Not defined.
0x01
Normal
The disk has consumed less than 80% of its reserved blocks
0x02
Warning
The disk has consumed more than 80% of its reserved blocks
0x03
Urgent
The disk has consumed more than 90% of its reserved blocks
HDD¶
Most HDDs can report their health through S.M.A.R.T. data as described in S.M.A.R.T. Hard Disk Status as well as perform disk tests. S.M.A.R.T. may need to be enabled both in the system BIOS and on the disk, though in modern systems these tend to both be enabled by default.
USB Disks¶
Disks connected through USB controllers are unlikely to support health queries, no matter what type of disk they are.
Taking Action¶
If a disk is showing signs that it might be failing, the safest action is to replace the disk. If a device has a built-in disk like an eMMC disk that cannot be replaced, it may be capable of taking an additional drive using another means such as NVMe, M.2, mSATA, or SATA. When in doubt, contact the device OEM for guidance.
If the disk is showing some wear but still has a lot of life left, consider making changes to reduce disk writes to potentially extend its remaining lifetime.