Creating a RAID 5 Array

I wanted to combine the space on several USB thumb drives to appear as one drive. This can be achieved by making the disks into a RAID array. There are several RAID formats to choose from and I chose RAID 5 as it offers a performance increase as well as redundancy (it can withstand the failure of one drive). RAID 5 reserves the space equivalent to one drive to record parity data and spreads the files across the remaining drives. In the event one of the drives fails, the lost data can be recalculated from the data on the remaining drives plus the parity data.

In this example I used 4 16Gb USB thumb drives. This will provide usable storage space of 48Gb (3 x 16Gb) with the remaining 16Gb being used to store the parity data.

Steps

  • Install RAID software – mdadm
  • Partition the drives to be used in the RAID
  • Create the RAID drive
  • Save RAID settings and update initramfs
  • Format and mount the RAID drive
  • Remount the RAID drive on reboots

Install RAID software

sudo apt-get install mdadm

Partition the drives to be used in the RAID

To view the available drives:

sudo fdisk -l

Here you can see that I have 4, 16Gb USB thumb drives on /dev/sda, /dev/sdb, /dev/sdc, /dev/sdd (all highlighted in yellow):

pi@raspberrypi:~ $ sudo fdisk -l

Device         Boot  Start     End Sectors  Size Id Type
/dev/mmcblk0p1        8192  137215  129024   63M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      137216 3862527 3725312  1.8G 83 Linux

Disk /dev/sda: 14.9 GiB, 16000221184 bytes, 31250432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0cb6096b

Disk /dev/sdb: 14.9 GiB, 16000221184 bytes, 31250432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xb0badb8f

Disk /dev/sdc: 14.9 GiB, 16000221184 bytes, 31250432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Disk /dev/sdd: 14.9 GiB, 16000221184 bytes, 31250432 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

pi@raspberrypi:~ $

Whilst ‘fdisk’ is commonly used to partition drives, it only supports drives up to 2TB in size. Although my USB thumb drives are much smaller than 2TB, I recommend you create the partitions with ‘parted’ using GPT labels rather than ‘fdisk’. This is good practice as it is good for drives smaller and larger than 2TB. For each drive /dev/sda through to /dev/sdd, do the following:

pi@raspberrypi:~ $ sudo parted /dev/sda
GNU Parted 3.2
Using /dev/sda
Welcome to GNU Parted! Type ‘help’ to view a list of commands.
(parted) mklabel gpt Warning: The existing disk label on /dev/sda will be destroyed and all data on this disk will be lost. Do you want to continue? Yes/No? Yes
(parted) mkpart primary 1 -1
(parted) align-check
alignment type(min/opt)  [optimal]/minimal? optimal
Partition number? 1
1 aligned
(parted) quit
Information: You may need to update /etc/fstab.
pi@raspberrypi:~ $

Create the RAID drive

sudo mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[a-d]1

This will create a RAID 5 array using 4 disks on the partitions /dev/sda1 through to /dev/sdd1. The array will be mounted at /dev/md0. Once you enter the command it will return to the command line within seconds but don’t be fooled, building the array can take some time and is happening in the background. In this example it took about an hour on a Raspberry Pi 3. DO NOT restart the Raspberry Pi during this time. You can watch the progress with watch cat /proc/mdstat, (press Ctrl-C to exit):

watch cat /proc/mdstat

Every 2.0s: cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[4] sdc1[2] sdb1[1] sda1[0]
      46844928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_]
      [==>………………] recovery = 13.8% (2163080/15614976) finish=48.7min speed=4597K/sec

unused devices: <none>

Check the status of the array and take a note of the Chunk Size (highlighted in yellow – you’ll need it to efficiently format the drive later), should look like this:

sudo mdadm --detail /dev/md0

pi@raspberrypi:~ $ sudo mdadm –detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Wed Jul 13 11:33:26 2016
     Raid Level : raid5
     Array Size : 46844928 (44.67 GiB 47.97 GB)
  Used Dev Size : 15614976 (14.89 GiB 15.99 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Wed Jul 13 12:31:09 2016
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : raspberrypi:0 (local to host raspberrypi)
           UUID : 46941901:b008107d:afa7ff38:65905224
         Events : 18

    Number   Major   Minor   RaidDevice State
       0       8        1       0      active sync   /dev/sda1
       1       8       17       1      active sync   /dev/sdb1
       2       8       33       2      active sync   /dev/sdc1
       4       8       49       3      active sync   /dev/sdd1
pi@raspberrypi:~ $

Save RAID settings and update initramfs

sudo -i
mv /etc/mdadm/mdadm.conf /etc/mdadm/mdadm.conf.old
mdadm -Es >> /etc/mdadm/mdadm.conf
update-initramfs -u
exit

I’ve not had this on other linux systems, but for some reason with the Raspberry Pi I’ve found that for mdadm to initialise the array on bootup you need to edit /boot/cmdline.txt and add rootdelay=10 to the end of the line:

sudo nano /boot/cmdline.txt

So it will look something like this. (addition highlighted in yellow):

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait rootdelay=10

Ctrl-O to save
Ctrl-X to exit

Format and mount the RAID drive

You should format the array so that it is optimised for the RAID deployment. You’ll need to know:

  • Chunk Size – Highlighted above when checking the status of the array
  • Block Size – Recommend you use 4KB (which is 4096 bytes when entered on the command line)
  • Stride – Calculated by dividing the Chunck Size by the Block Size. In this case, 512KB / 4 KB = 128
  • Stripe Width – Calculated by: (number of disks in array – 1) multiplied by the Stride. In this case: (4-1) * 128 = 384

Then format the RAID drive with the following command:

sudo mkfs.ext4 -b 4096 -E stride=128,stripe-width=384 /dev/md0

Create a mount point and mount the RAID array, in this case I use /mnt/raid5:

sudo mkdir /mnt/raid5
sudo mount /dev/md0 /mnt/raid5

If you now change directories to /mnt/raid5 and do a directory listing you should see an empty drive with a ‘lost+found’ directory’:

pi@raspberrypi: ~ $ cd /mnt/raid5
pi@raspberrypi: /mnt/raid5 $ ls -al
total 24
drwxr-xr-x 3 root root 4096 Jul 13 13:56 .
drwxr-xr-x 3 root root 4096 Jul 13 14:03 ..
drwx—— 2 root root 16384 Jul 13 13:56 lost+found
pi@raspberrypi: /mnt/raid5 $

Reducing the reserved space

By default, ext4 reserves 5% of disk space for root and system services. This is to ensure that should a remote user fill the disk, that space remains for critical system services to continue and to assist trouble shooting by ensuring the root user could still log in and have some space. This makes sense for the drive the Operating System is loaded on, but isn’t necessary for a data drive. On a large RAID this can account for a lot of wasted space. I have 4x 6TB drives in a RAID 5 config as a media server and this 5% reserve would account for nearly 1TB of reserved/wasted space. You may therefore choose to reduce the percentage of space reserved. To reduce to 1%:

tune2fs -m 1 /dev/md0

To reduce to 0%:

tune2fs -m 0 /dev/md0

Remount the RAID drive on reboots

Add the following line to the end of /etc/fstab:
/dev/md0 /mnt/raid5 ext4 defaults 0 0

sudo nano /etc/fstab

So it will look something like this:

proc            /proc           proc    defaults          0       0
/dev/mmcblk0p1  /boot           vfat    defaults          0       2
/dev/mmcblk0p2  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use dphys-swapfile swap[on|off]  for that

/dev/md0        /mnt/raid5      ext4    defaults          0       0

Ctrl-O to save
Ctrl-X to exit

Check RAID status

To check the status of the RAID:

sudo mdadm --detail /dev/md0

or

cat /proc/mdstat

Replace disk on failure

I have removed one of the USB thumb drives to emulate a failure. In this case /dev/sdd1. The data on /mnt/raid5 is still readable. The status of the RAID looks like this, I’ve highlighted in yellow the fields which show an error has occurred:

pi@raspberrypi: ~ $ sudo mdadm –detail /dev/md0 /dev/md0: Version : 1.2 Creation Time : Wed Jul 13 11:33:26 2016 Raid Level : raid5 Array Size : 46844928 (44.67 GiB 47.97 GB) Used Dev Size : 15614976 (14.89 GiB 15.99 GB) Raid Devices : 4 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Jul 13 21:50:28 2016 State : clean, degraded Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : raspberrypi:0 (local to host raspberrypi) UUID : 46941901:b008107d:afa7ff38:65905224 Events : 44 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 2 0 0 2 removed 2 8 17 2 active sync /dev/sdb1 4 8 33 3 active sync /dev/sdc1 pi@raspberrypi:~ $

pi@raspberrypi: ~ $ cat /proc/mdstat Personalities : [raid6] [raid5] [raid4] md0 : active raid5 sda1[0] sdb1[1] sdc1[2] 46844928 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [UUU_] unused devices: <none> pi@raspberrypi:~ $

To recover, mark /dev/sdd1 as failed:

sudo mdadm --manage /dev/md0 --fail /dev/sdd1

Remove /dev/sdd1 from the RAID array:

sudo mdadm --manage /dev/md0 --remove /dev/sdd1

You may wish to check the serial number of the drives to ensure you’re keeping/replacing the correct drives:

sudo hdparm -i /dev/sdd

Power down and replace the failed drive with one which is no smaller in size than the smallest disk in the RAID array.Create the same partition on the replacement disk as is on the other disks in the array (see above).

Add the new disk to the RAID:

sudo mdadm --manage /dev/md0 --add /dev/sdd1

mdadm will detect that the RAID is degraded and that a new disk is available and rebuild the array. The synchronisation can be monitored with:

watch cat /proc/mdstat

Starting the RAID if it fails to start at bootup

If /dev/md0 is not found at bootup then mdadm has failed to initialise the RAID. This has two common causes:

  1. the /etc/mdadm/mdadm.conf file contains an error, or
  2. rootdelay=10 has not been added to the end of /boot/cmdline.txt

The RAID can be initialised at the command line by:

sudo mdadm -A /dev/md0 /dev/sd[a-d]1

Once initialised, use sudo mdadm –detail /dev/md0 and double check the UUID of the RAID is the same as in the /etc/mdadm/mdadm.conf file.

Inspecting a single partition

If you wish to see the status of a single partition use:

sudo mdadm --examine /dev/sda1

Erasing the MBR (Master Boot Record)

On a few occasions when I’ve tried to create the array, an error has been returned saying that the device is already in use and not suitable for this type of array. Often this is due to the disk being part of an old array which has been dismantled. In this case, use mdadm to see if there is a ‘superblock’ on the disk:

sudo mdadm --examine /dev/sda1

If there is then try erasing the MBR with the following command. Be VERY CAREFUL as this command will write zeros to the first 4096Kb of the disk so make sure you select the right disk! To avoid copy and paste errors, I’ve highlighted below and replaced the drive letter with an ‘x‘. Replace this with the drive you wish to erase.

sudo dd if=/dev/zero of=/dev/sdx bs=4096 count=1

If that fails, then try wiping the whole disk (this might take some time!):

sudo dd if=/dev/zero of=/dev/sdx

1 thought on “Creating a RAID 5 Array”

  1. Youre so cool! I dont suppose Ive read anything similar to this prior to. So nice to get somebody with many original thoughts on this subject. realy appreciate beginning this up. this website is one area that is needed on the internet, a person after some originality. beneficial task for bringing a new challenge for the net!

Leave a Reply

Your email address will not be published. Required fields are marked *