-
Linux software RAID
[EDIT] I apologize for the column being so wide. I tried to reformat the tiobench output to make it more sensible, but it doesn't look right any other way.
Let me begin by saying that this is as much a documentary as a how-to guide, because there are easier ways to do this with Linux distributions like Red Hat and Mandrake. I'm just the kind of guy who likes to play in the mud and do things the hard way. :D I've spent the last week fiddling with this, and I've had to start over a couple of times, and I lost all my data due to a hardware failure in the array. It's been an adventure, so this time through I decided to document my progress for anyone who might be interested in this sort of thing.
I don't really expect anybody to actually do this (at least not this way), but if you do, I'd like to hear about your results.
A couple of weeks ago, one of my hard drives died and I went down to Office Depot to replace it, and an idea hit me. As I was standing in line at the check-out counter with my new 60GB ATA-133 drive, I had a moment of clarity and an idea struck me. I only use Linux, and the Linux kernel is fully capable of performing software RAID and doubling my disk I/O, so why not buy 2 disks instead of one and give it a try? So I went back and got another one just like it, then went to the grocery store and got plenty of coffee for the weekend.
So the adventure begins......
First, I should begin with a little explanation of what software RAID is for those who might not be familiar with it. RAID is an acronym for Redundant Array of Independent Disks, and it means basically combining two or more disks to look and act like one. There are several reasons why you might want to do this, such as combining capacity to produce an extremely large volume or increasing disk throughput for performance. RAID may be done in hardware with a special hard drive controller, or the Linux kernel can do it in software.
There are several different types of RAID that can be set up for different purposes. The types of RAID that the Linux kernel currently supports are as follows:
Linear RAID: Two or more disks are combined to create one large volume. There is no redundancy for data protection, and there is no performance gain.
Level 0: Disk Striping - Both capacity and performance are increased by accessing the disks is parallel. There is no data redundancy in case of a disk failure.
Level 1: Disk Mirroring - Duplicate contents of one disk are written onto another disk for security.
Level 4: Data is striped across multiple drives and parity is written to a dedicated drive. Level 4 is typically implemented at the BLOCK level.
Level 5: Error correction data is striped at the block level across all the drives in the array. Reads and writes may be performed concurrently.
The RAID setups with mirroring and data redundancy are most commonly found on servers where critical data is stored and data integrity is the paramount concern. However, for home use I'm far more interested in performance for operations like video capture, so I'm going for RAID-0, or striping. I will be using 3 disks in the array: 2 60GB ATA-133 and an older 40GB ATA-100, for a total capacity of 160GB. I will also be installing Gentoo Linux, which is my distribution of choice. (But as I said, installing onto software RAID partitions is somewhat easier with Red Hat and Mandrake.) RAID is not for the faint of heart, however, and there are a few things to think about before beginning.
* You're taking a risk by striping disks, because if one disk in the array fails, the data on ALL the drives is lost. This has already happened to me once in the last week, so be careful and keep good backups. Unfortunately, unless you have a high-capacity tape drive, where the hell are you going to back up 160GB to?
* Each drive must be the ONLY device on the controller to prevent the IDE controller from becoming a bottleneck. That means if you have more than 2 IDE devices you will probably have to invest in some off-board ATA controllers. For my RAID array, I will be using two offboard controllers, a Promise ATA-100 and a Promise ATA-133 with one disk on each channel.
* I will not be doing a conventional Gentoo install, because at the time of this writing, the Gentoo 1.4RC1 installation CD does not support software RAID. I will be doing the RAID installation from a separate Linux installation. If you are going to attempt this and have never installed Gentoo before, I highly suggest that you become intimately acquainted with the Gentoo installation guide before beginning.
First, we need a working Linux installation on a disk that is not in the array. It can be any Linux distribution like Red Hat or Mandrake, as long as the kernel has multi-device and RAID support compiled in. For comparison purposes, I like to benchmark the drives before I begin so I will have a point of reference for my RAID performance. I will use tiobench as recommended by the Linux RAID FAQ.
A quick test of my new ATA-133 drive yields this output:
Code:
No size specified, using 1022 MB
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19 1022 4096 1 39.83 13.05% 0.098 12.69 0.00000 0.00000 305
2.4.19 1022 4096 2 31.02 10.47% 0.223 154.29 0.00000 0.00000 296
2.4.19 1022 4096 4 27.17 9.615% 0.524 196.28 0.00000 0.00000 283
2.4.19 1022 4096 8 25.29 8.688% 1.054 235.80 0.00000 0.00000 291
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19 1022 4096 1 0.90 0.690% 4.342 13.83 0.00000 0.00000 130
2.4.19 1022 4096 2 0.87 0.391% 7.743 26.28 0.00000 0.00000 223
2.4.19 1022 4096 4 0.88 0.449% 16.979 65.55 0.00000 0.00000 195
2.4.19 1022 4096 8 0.94 0.482% 26.287 136.73 0.00000 0.00000 195
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19 1022 4096 1 27.08 19.02% 0.103 384.58 0.00000 0.00000 142
2.4.19 1022 4096 2 26.51 18.75% 0.200 495.79 0.00000 0.00000 141
2.4.19 1022 4096 4 25.97 18.50% 0.397 697.91 0.00000 0.00000 140
2.4.19 1022 4096 8 26.21 19.39% 0.750 1104.93 0.00000 0.00000 135
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19 1022 4096 1 0.61 0.352% 0.020 0.56 0.00000 0.00000 174
2.4.19 1022 4096 2 0.61 0.314% 0.033 17.76 0.00000 0.00000 195
2.4.19 1022 4096 4 0.61 0.353% 0.041 22.03 0.00000 0.00000 174
2.4.19 1022 4096 8 0.64 0.286% 0.080 62.96 0.00000 0.00000 223
Now I'm ready to begin. First thing to do is partition my drives, keeping in mind the rules I outlined in my tutorial on Linux partitioning. My hde and hdg drives are my two 60GB drives, and hdi is the old 40GB drive. Since the kernel cannot boot from a RAID-0 volume, I have to create a small /boot partition to house the kernel. I will put this on hde1 for simplicity. The remainder of my partitions on hde and hdg will be mirrors of each other, so I will make them all logical partitions numbered starting at 5 to avoid mistakes in setting up the RAID array. I make hde5 and hdg5 256MB swap partitions because Linux will automatically stripe swap partitions with the same priority, giving me a total of 512 MB of swap space. Then I make the remaining system partitions as follows:
hde6/hdg6 50MB each (for /)
hde7/hdg7 128MB each (for /tmp)
hde8/hdg8 500MB each (for /var)
hde9/hdg9 2500MB each (for /usr)
For the /home partition, I will use 3 drives in the array instead of 2. I will use all the remaining space on the 60GB drives for hde10 and hdg10, as well as the 40GB drive hdi1. VERY IMPORTANT: The partition type on all RAID partitions must be set "Linux Raid Autodetect." In fdisk, use the t command and set the type to "fd" for each of the RAID partitions (i.e., everything except /boot and the 2 swap partitions). Set the 2 swap partitions to type 82, "Linux swap."
Now we're ready to actually make the RAID array. Create a file called /etc/raidtab and tell Linux which partitions go in which array. Each entry must specify the RAID level, the number of disks in the array, any spare disks (in this case none), the chunk size of the array, and whether the array is to have a persistent superblock. The chunk size is the amount of data that will be written to one disk before another disk is used, and it can be anywhere from 4k to 4MB. Through a little research on Google and some benchmarking, I have found 256k to be a good all-purpose chunk size. The arrays must have a persistent superblock for the kernel to autodetect them.
Code:
raiddev /dev/md0
raid-level 0
nr-raid-disks 2
nr-spare-disks 0
chunk-size 256k
persistent-superblock 1
device /dev/hde6
raid-disk 0
device /dev/hdg6
raid-disk 1
raiddev /dev/md1
raid-level 0
nr-raid-disks 2
nr-spare-disks 0
chunk-size 256k
persistent-superblock 1
device /dev/hde7
raid-disk 0
device /dev/hdg7
raid-disk 1
raiddev /dev/md2
raid-level 0
nr-raid-disks 2
nr-spare-disks 0
chunk-size 256k
persistent-superblock 1
device /dev/hde8
raid-disk 0
device /dev/hdg8
raid-disk 1
raiddev /dev/md3
raid-level 0
nr-raid-disks 2
nr-spare-disks 0
chunk-size 256k
persistent-superblock 1
device /dev/hde9
raid-disk 0
device /dev/hdg9
raid-disk 1
raiddev /dev/md4
raid-level 0
nr-raid-disks 3
nr-spare-disks 0
chunk-size 256k
persistent-superblock 1
device /dev/hde10
raid-disk 0
device /dev/hdg10
raid-disk 1
device /dev/hdi1
raid-disk 2
Once /etc/raidtab is in place, create the RAID arrays with the mkraid command like so:
Code:
mkraid /dev/md0
mkraid /dev/md1
mkraid /dev/md2
mkraid /dev/md3
mkraid /dev/md4
Then format with the filesystem of your choice. I recommend ext3 for its robustness. See my tutorial on filesystems for more information.
Code:
mke2fs -j /dev/hde1
mke2fs -j /dev/md0
mkesf2 -j /dev/md2
mkesf2 -j /dev/md3
mkesf2 -j /dev/md4
mkswap /dev/hde5
mkswap /dev/hdg5
The RAID arrays are now created and ready to mount. From here, proceed with the normal Gentoo installation beginning with Code Listing 17 (mount partitions). The only caveats:
- You must copy /etc/raidtab to /mnt/gentoo/etc at the same time as /etc/recolv.conf in code listing 19.
- /etc/fstab should treat the /dev/mdx devices as regular partitions
- Be sure to compile multi-device and RAID-0 directly into the kernel and not as modules.
- Be sure to install (emerge) raidtools before finishing the installation.
Once it's all compiled, configured, and rebooted, a quick benchmark of my new /home partition yields the following results:
Code:
No size specified, using 1022 MB
Unit information
================
File size = megabytes
Blk Size = bytes
Rate = megabytes per second
CPU% = percentage of CPU used during the test
Latency = milliseconds
Lat% = percent of requests that took longer than X seconds
CPU Eff = Rate divided by CPU% - throughput per cpu load
Sequential Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19-crypto-r7 1022 4096 1 78.00 34.56% 0.050 92.62 0.00000 0.00000 226
2.4.19-crypto-r7 1022 4096 2 69.47 30.47% 0.110 551.78 0.00000 0.00000 228
2.4.19-crypto-r7 1022 4096 4 68.19 30.63% 0.225 669.62 0.00000 0.00000 223
2.4.19-crypto-r7 1022 4096 8 65.76 30.30% 0.425 1288.70 0.00000 0.00000 217
Random Reads
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19-crypto-r7 1022 4096 1 0.76 0.870% 5.168 22.01 0.00000 0.00000 87
2.4.19-crypto-r7 1022 4096 2 1.18 0.967% 6.369 33.38 0.00000 0.00000 122
2.4.19-crypto-r7 1022 4096 4 1.59 1.218% 9.316 65.98 0.00000 0.00000 130
2.4.19-crypto-r7 1022 4096 8 1.85 1.162% 14.861 117.12 0.00000 0.00000 159
Sequential Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19-crypto-r7 1022 4096 1 45.53 43.53% 0.070 1942.27 0.00000 0.00000 105
2.4.19-crypto-r7 1022 4096 2 44.88 38.25% 0.131 2072.38 0.00115 0.00000 117
2.4.19-crypto-r7 1022 4096 4 44.84 40.09% 0.263 4166.85 0.00345 0.00000 112
2.4.19-crypto-r7 1022 4096 8 44.61 40.54% 0.483 2939.09 0.00577 0.00000 110
Random Writes
File Blk Num Avg Maximum Lat% Lat% CPU
Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff
---------------------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- -----
2.4.19-crypto-r7 1022 4096 1 2.61 1.923% 0.023 4.13 0.00000 0.00000 136
2.4.19-crypto-r7 1022 4096 2 2.67 2.545% 0.026 1.90 0.00000 0.00000 105
2.4.19-crypto-r7 1022 4096 4 2.90 2.634% 0.026 3.91 0.00000 0.00000 110
2.4.19-crypto-r7 1022 4096 8 2.99 2.695% 0.026 1.81 0.00000 0.00000 111
Not too shabby, eh? :D
Further resources:
RAID and Data Protection Solutions for Linux
The Software-RAID HOWTO
Linux-RAID FAQ
-
good tutorial, how many have you done? i saw you mentioned a couple in there. do you think you could send some to my email?
c0rporate_mupp3t@yahoo.com
-
Thanks, it's nice to know people really read these things. :) I have links to all of my tutorials in my AO profile, so you can browse them from there.
