Software-RAID HOWTO: Error Recovery

Next Previous Contents

## 4. Error Recovery

1. Q: I have a RAID-1 (mirroring) setup, and lost power while there was disk activity. Now what do I do?

   > A: The redundancy of RAID levels is designed to protect against a disk failure, not against a power failure. There are several ways to recover from this situation.
   > - Method (1): Use the raid tools. These can be used to sync the raid arrays. They do not fix file-system damage; after the raid arrays are sync'ed, then the file-system still has to be fixed with fsck. Raid arrays can be checked with `ckraid /etc/raid1.conf` (for RAID-1, else, `/etc/raid5.conf`, etc.) Calling `ckraid /etc/raid1.conf --fix` will pick one of the disks in the array (usually the first), and use that as the master copy, and copy its blocks to the others in the mirror. To designate which of the disks should be used as the master, you can use the `--force-source` flag: for example, `ckraid /etc/raid1.conf --fix --force-source /dev/hdc3` The ckraid command can be safely run without the `--fix` option to verify the inactive RAID array without making any changes. When you are comfortable with the proposed changes, supply the `--fix` option.
   > - Method (2): Paranoid, time-consuming, not much better than the first way. Lets assume a two-disk RAID-1 array, consisting of partitions `/dev/hda3` and `/dev/hdc3`. You can try the following:
   >   1. `fsck /dev/hda3`
   >   2. `fsck /dev/hdc3`
   >   3. decide which of the two partitions had fewer errors, or were more easily recovered, or recovered the data that you wanted. Pick one, either one, to be your new ``master'' copy. Say you picked `/dev/hdc3`.
   >   4. `dd if=/dev/hdc3 of=/dev/hda3`
   >   5. `mkraid raid1.conf -f --only-superblock`
   >   Instead of the last two steps, you can instead run `ckraid /etc/raid1.conf --fix --force-source /dev/hdc3` which should be a bit faster.
   > - Method (3): Lazy man's version of above. If you don't want to wait for long fsck's to complete, it is perfectly fine to skip the first three steps above, and move directly to the last two steps. Just be sure to run `fsck /dev/md0` after you are done. Method (3) is actually just method (1) in disguise.
   > In any case, the above steps will only sync up the raid arrays. The file system probably needs fixing as well: for this, fsck needs to be run on the active, unmounted md device.

   > With a three-disk RAID-1 array, there are more possibilities, such as using two disks to ''vote'' a majority answer. Tools to automate this do not currently (September 97) exist.

2. Q: I have a RAID-4 or a RAID-5 (parity) setup, and lost power while there was disk activity. Now what do I do?

   > A: The redundancy of RAID levels is designed to protect against a disk failure, not against a power failure. Since the disks in a RAID-4 or RAID-5 array do not contain a file system that fsck can read, there are fewer repair options. You cannot use fsck to do preliminary checking and/or repair; you must use ckraid first.

   > The `ckraid` command can be safely run without the `--fix` option to verify the inactive RAID array without making any changes. When you are comfortable with the proposed changes, supply the `--fix` option.

   > If you wish, you can try designating one of the disks as a ''failed disk''. Do this with the `--suggest-failed-disk-mask` flag.

   > Only one bit should be set in the flag: RAID-5 cannot recover two failed disks. The mask is a binary bit mask: thus:

   >         0x1 == first disk
   >         0x2 == second disk
   >         0x4 == third disk
   >         0x8 == fourth disk, etc.

   > Alternately, you can choose to modify the parity sectors, by using the `--suggest-fix-parity` flag. This will recompute the parity from the other sectors.

   > The flags `--suggest-failed-dsk-mask` and `--suggest-fix-parity` can be safely used for verification. No changes are made if the `--fix` flag is not specified. Thus, you can experiment with different possible repair schemes.

3. Q: My RAID-1 device, `/dev/md0` consists of two hard drive partitions: `/dev/hda3` and `/dev/hdc3`. Recently, the disk with `/dev/hdc3` failed, and was replaced with a new disk. My best friend, who doesn't understand RAID, said that the correct thing to do now is to ''`dd if=/dev/hda3 of=/dev/hdc3`''. I tried this, but things still don't work.

   > A: You should keep your best friend away from you computer. Fortunately, no serious damage has been done. You can recover from this by running:

   > > `

   > >     mkraid raid1.conf -f --only-superblock

   > > `

   > By using `dd`, two identical copies of the partition were created. This is almost correct, except that the RAID-1 kernel extension expects the RAID superblocks to be different. Thus, when you try to reactivate RAID, the software will notice the problem, and deactivate one of the two partitions. By re-creating the superblock, you should have a fully usable system.

4. Q: My version of `mkraid` doesn't have a `--only-superblock` flag. What do I do?

   > A: The newer tools drop support for this flag, replacing it with the `--force-resync` flag. It has been reported that the following sequence appears to work with the latest tools and software:

   > > `

   > >       umount /web (where /dev/md0 was mounted on)
   > >       raidstop /dev/md0
   > >       mkraid /dev/md0 --force-resync --really-force
   > >       raidstart /dev/md0

   > > `

   > After doing this, a `cat /proc/mdstat` should report `resync in progress`, and one should be able to `mount /dev/md0` at this point.

5. Q: My RAID-1 device, `/dev/md0` consists of two hard drive partitions: `/dev/hda3` and `/dev/hdc3`. My best (girl?)friend, who doesn't understand RAID, ran `fsck` on `/dev/hda3` while I wasn't looking, and now the RAID won't work. What should I do?

   > A: You should re-examine your concept of ``best friend''. In general, `fsck` should never be run on the individual partitions that compose a RAID array. Assuming that neither of the partitions are/were heavily damaged, no data loss has occurred, and the RAID-1 device can be recovered as follows:
   > 1. make a backup of the file system on `/dev/hda3`
   > 2. `dd if=/dev/hda3 of=/dev/hdc3`
   > 3. `mkraid raid1.conf -f --only-superblock`
   > This should leave you with a working disk mirror.

6. Q: Why does the above work as a recovery procedure?

   > A: Because each of the component partitions in a RAID-1 mirror is a perfectly valid copy of the file system. In a pinch, mirroring can be disabled, and one of the partitions can be mounted and safely run as an ordinary, non-RAID file system. When you are ready to restart using RAID-1, then unmount the partition, and follow the above instructions to restore the mirror. Note that the above works ONLY for RAID-1, and not for any of the other levels.

   > It may make you feel more comfortable to reverse the direction of the copy above: copy from the disk that was untouched to the one that was. Just be sure to fsck the final md.

7. Q: I am confused by the above questions, but am not yet bailing out. Is it safe to run `fsck /dev/md0` ?

   > A: Yes, it is safe to run `fsck` on the `md` devices. In fact, this is the only safe place to run `fsck`.

8. Q: If a disk is slowly failing, will it be obvious which one it is? I am concerned that it won't be, and this confusion could lead to some dangerous decisions by a sysadmin.

   > A: Once a disk fails, an error code will be returned from the low level driver to the RAID driver. The RAID driver will mark it as ``bad'' in the RAID superblocks of the ``good'' disks (so we will later know which mirrors are good and which aren't), and continue RAID operation on the remaining operational mirrors.

   > This, of course, assumes that the disk and the low level driver can detect a read/write error, and will not silently corrupt data, for example. This is true of current drives (error detection schemes are being used internally), and is the basis of RAID operation.

9. Q: What about hot-repair?

   > A: Work is underway to complete ``hot reconstruction''. With this feature, one can add several ``spare'' disks to the RAID set (be it level 1 or 4/5), and once a disk fails, it will be reconstructed on one of the spare disks in run time, without ever needing to shut down the array.

   > However, to use this feature, the spare disk must have been declared at boot time, or it must be hot-added, which requires the use of special cabinets and connectors that allow a disk to be added while the electrical power is on.

   > As of October 97, there is a beta version of MD that allows:

   > - RAID 1 and 5 reconstruction on spare drives
   > - RAID-5 parity reconstruction after an unclean shutdown
   > - spare disk to be hot-added to an already running RAID 1 or 4/5 array
   > By default, automatic reconstruction is (Dec 97) currently disabled by default, due to the preliminary nature of this work. It can be enabled by changing the value of `SUPPORT_RECONSTRUCTION` in `include/linux/md.h`.

   > If spare drives were configured into the array when it was created and kernel-based reconstruction is enabled, the spare drive will already contain a RAID superblock (written by `mkraid`), and the kernel will reconstruct its contents automatically (without needing the usual `mdstop`, replace drive, `ckraid`, `mdrun` steps).

   > If you are not running automatic reconstruction, and have not configured a hot-spare disk, the procedure described by Gadi Oxman < gadio@netvision.net.il> is recommended:

   > - Currently, once the first disk is removed, the RAID set will be running in degraded mode. To restore full operation mode, you need to:
   >   - stop the array (`mdstop /dev/md0`)
   >   - replace the failed drive
   >   - run `ckraid raid.conf` to reconstruct its contents
   >   - run the array again (`mdadd`, `mdrun`).
   >   At this point, the array will be running with all the drives, and again protects against a failure of a single drive.

   > Currently, it is not possible to assign single hot-spare disk to several arrays. Each array requires it's own hot-spare.

10. Q: I would like to have an audible alarm for ``you schmuck, one disk in the mirror is down'', so that the novice sysadmin knows that there is a problem.

    > A: The kernel is logging the event with a ```KERN_ALERT`'' priority in syslog. There are several software packages that will monitor the syslog files, and beep the PC speaker, call a pager, send e-mail, etc. automatically.

11. Q: How do I run RAID-5 in degraded mode (with one disk failed, and not yet replaced)?

    > A: Gadi Oxman < gadio@netvision.net.il> writes: Normally, to run a RAID-5 set of n drives you have to:

    > > `

    > >     mdadd /dev/md0 /dev/disk1 ... /dev/disk(n)
    > >     mdrun -p5 /dev/md0

    > > `

    > Even if one of the disks has failed, you still have to `mdadd` it as you would in a normal setup. (?? try using /dev/null in place of the failed disk ??? watch out) Then, The array will be active in degraded mode with (n - 1) drives. If ```mdrun`'' fails, the kernel has noticed an error (for example, several faulty drives, or an unclean shutdown). Use ```dmesg`'' to display the kernel error messages from ```mdrun`''. If the raid-5 set is corrupted due to a power loss, rather than a disk crash, one can try to recover by creating a new RAID superblock:

    > > `

    > >     mkraid -f --only-superblock raid5.conf

    > > `

    > A RAID array doesn't provide protection against a power failure or a kernel crash, and can't guarantee correct recovery. Rebuilding the superblock will simply cause the system to ignore the condition by marking all the drives as ``OK'', as if nothing happened.

12. Q: How does RAID-5 work when a disk fails?

    > A: The typical operating scenario is as follows:
    > - A RAID-5 array is active.
    > - One drive fails while the array is active.
    > - The drive firmware and the low-level Linux disk/controller drivers detect the failure and report an error code to the MD driver.
    > - The MD driver continues to provide an error-free `/dev/md0` device to the higher levels of the kernel (with a performance degradation) by using the remaining operational drives.
    > - The sysadmin can `umount /dev/md0` and `mdstop /dev/md0` as usual.
    > - If the failed drive is not replaced, the sysadmin can still start the array in degraded mode as usual, by running `mdadd` and `mdrun`.

13. Q:

    > A:

14. Q: Why is there no question 13?

    > A: If you are concerned about RAID, High Availability, and UPS, then its probably a good idea to be superstitious as well. It can't hurt, can it?

15. Q: I just replaced a failed disk in a RAID-5 array. After rebuilding the array, `fsck` is reporting many, many errors. Is this normal?

    > A: No. And, unless you ran fsck in "verify only; do not update" mode, its quite possible that you have corrupted your data. Unfortunately, a not-uncommon scenario is one of accidentally changing the disk order in a RAID-5 array, after replacing a hard drive. Although the RAID superblock stores the proper order, not all tools use this information. In particular, the current version of `ckraid` will use the information specified with the `-f` flag (typically, the file `/etc/raid5.conf`) instead of the data in the superblock. If the specified order is incorrect, then the replaced disk will be reconstructed incorrectly. The symptom of this kind of mistake seems to be heavy & numerous `fsck` errors.

    > And, in case you are wondering, yes, someone lost all of their data by making this mistake. Making a tape backup of all data before reconfiguring a RAID array is strongly recommended.

16. Q: The QuickStart says that `mdstop` is just to make sure that the disks are sync'ed. Is this REALLY necessary? Isn't unmounting the file systems enough?

    > A: The command `mdstop /dev/md0` will:
    > - mark it ''clean''. This allows us to detect unclean shutdowns, for example due to a power failure or a kernel crash.
    > - sync the array. This is less important after unmounting a filesystem, but is important if the `/dev/md0` is accessed directly rather than through a filesystem (for example, by `e2fsck`).

Next Previous Contents