We had an issue earlier today where a customer’s Ubutnu Server decided to stop working. It simply refused to boot into the OS, instead displaying the following:
[3.958693] md/raid:md0: raid level 5 active with 4 out of 5 devices, algorithm 2
BusyBox v1.18.5 (Ubuntu 1:1.18.5-1ubuntu4.1) built-in shell (ash)
Enter ‘help’ for a list of built-in commands.
Note: In the event where you must load the OS you can press CTRL + D to proceed with booting the OS.
The first line, obviously indicated a degraded RAID array. Essentially stating that there were only 4 out of 5 disks active, for one reason or another the fifth had decided to drop from the array.
We confirmed this by issuing the following two commands:
Mmdadm --detail /dev/md0
This command display a bunch of useful information on the array. In our situation we were interested in the RAID devices / Total Devices values. As well as the small tables at the end of the output which provided details on each disk.
This provides slightly less information, however is very useful for identifying a degraded array and clearly showed a dropped disk.
The dropped disk however, was still installed in the machine and still appeared to be showing up fine:
ls -l /dev/sd*
To provide a speedy remediation we attempted to simply re-add the disk to the array:
mdadm --manage /dev/md0 --add /dev/sde1
The worked fine and the drive was re-added to the array, with rebuilding/synchronization commencing immediately. We can confirm this and monitor the rebuild by issuing:
mdadm --detail /dev/md0
Within the output there is a Rebuild Status item which provides a % on the rebuild status.
We prefer to monitor the rebuild using the following:
This provides more detailed information on the rebuild, most notably the remaining time for the rebuild to complete – in our case it will take about 1754 minutes (29hrs) to rebuild the array.
Once the rebuild completed, everything reverted back to normal; the OS booted and operated fine and the array once again showed all 5 disks.