Results 1 to 3 of 3

Thread: Linux software RAID

  1. #1
    Senior Member problemchild's Avatar
    Join Date
    Jul 2002
    Posts
    551

    Linux software RAID

    [EDIT] I apologize for the column being so wide. I tried to reformat the tiobench output to make it more sensible, but it doesn't look right any other way.

    Let me begin by saying that this is as much a documentary as a how-to guide, because there are easier ways to do this with Linux distributions like Red Hat and Mandrake. I'm just the kind of guy who likes to play in the mud and do things the hard way. I've spent the last week fiddling with this, and I've had to start over a couple of times, and I lost all my data due to a hardware failure in the array. It's been an adventure, so this time through I decided to document my progress for anyone who might be interested in this sort of thing.

    I don't really expect anybody to actually do this (at least not this way), but if you do, I'd like to hear about your results.

    A couple of weeks ago, one of my hard drives died and I went down to Office Depot to replace it, and an idea hit me. As I was standing in line at the check-out counter with my new 60GB ATA-133 drive, I had a moment of clarity and an idea struck me. I only use Linux, and the Linux kernel is fully capable of performing software RAID and doubling my disk I/O, so why not buy 2 disks instead of one and give it a try? So I went back and got another one just like it, then went to the grocery store and got plenty of coffee for the weekend.

    So the adventure begins......

    First, I should begin with a little explanation of what software RAID is for those who might not be familiar with it. RAID is an acronym for Redundant Array of Independent Disks, and it means basically combining two or more disks to look and act like one. There are several reasons why you might want to do this, such as combining capacity to produce an extremely large volume or increasing disk throughput for performance. RAID may be done in hardware with a special hard drive controller, or the Linux kernel can do it in software.

    There are several different types of RAID that can be set up for different purposes. The types of RAID that the Linux kernel currently supports are as follows:

    Linear RAID: Two or more disks are combined to create one large volume. There is no redundancy for data protection, and there is no performance gain.

    Level 0: Disk Striping - Both capacity and performance are increased by accessing the disks is parallel. There is no data redundancy in case of a disk failure.

    Level 1: Disk Mirroring - Duplicate contents of one disk are written onto another disk for security.

    Level 4: Data is striped across multiple drives and parity is written to a dedicated drive. Level 4 is typically implemented at the BLOCK level.

    Level 5: Error correction data is striped at the block level across all the drives in the array. Reads and writes may be performed concurrently.

    The RAID setups with mirroring and data redundancy are most commonly found on servers where critical data is stored and data integrity is the paramount concern. However, for home use I'm far more interested in performance for operations like video capture, so I'm going for RAID-0, or striping. I will be using 3 disks in the array: 2 60GB ATA-133 and an older 40GB ATA-100, for a total capacity of 160GB. I will also be installing Gentoo Linux, which is my distribution of choice. (But as I said, installing onto software RAID partitions is somewhat easier with Red Hat and Mandrake.) RAID is not for the faint of heart, however, and there are a few things to think about before beginning.

    * You're taking a risk by striping disks, because if one disk in the array fails, the data on ALL the drives is lost. This has already happened to me once in the last week, so be careful and keep good backups. Unfortunately, unless you have a high-capacity tape drive, where the hell are you going to back up 160GB to?

    * Each drive must be the ONLY device on the controller to prevent the IDE controller from becoming a bottleneck. That means if you have more than 2 IDE devices you will probably have to invest in some off-board ATA controllers. For my RAID array, I will be using two offboard controllers, a Promise ATA-100 and a Promise ATA-133 with one disk on each channel.

    * I will not be doing a conventional Gentoo install, because at the time of this writing, the Gentoo 1.4RC1 installation CD does not support software RAID. I will be doing the RAID installation from a separate Linux installation. If you are going to attempt this and have never installed Gentoo before, I highly suggest that you become intimately acquainted with the Gentoo installation guide before beginning.

    First, we need a working Linux installation on a disk that is not in the array. It can be any Linux distribution like Red Hat or Mandrake, as long as the kernel has multi-device and RAID support compiled in. For comparison purposes, I like to benchmark the drives before I begin so I will have a point of reference for my RAID performance. I will use tiobench as recommended by the Linux RAID FAQ.

    A quick test of my new ATA-133 drive yields this output:
    Code:
    No size specified, using 1022 MB
    
    Unit information
    ================
    File size = megabytes
    Blk Size  = bytes
    Rate      = megabytes per second
    CPU%      = percentage of CPU used during the test
    Latency   = milliseconds
    Lat%      = percent of requests that took longer than X seconds
    CPU Eff   = Rate divided by CPU% - throughput per cpu load
    
    Sequential Reads
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19                        1022  4096    1   39.83 13.05%     0.098       12.69   0.00000  0.00000   305
    2.4.19                        1022  4096    2   31.02 10.47%     0.223      154.29   0.00000  0.00000   296
    2.4.19                        1022  4096    4   27.17 9.615%     0.524      196.28   0.00000  0.00000   283
    2.4.19                        1022  4096    8   25.29 8.688%     1.054      235.80   0.00000  0.00000   291
    
    Random Reads
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19                        1022  4096    1    0.90 0.690%     4.342       13.83   0.00000  0.00000   130
    2.4.19                        1022  4096    2    0.87 0.391%     7.743       26.28   0.00000  0.00000   223
    2.4.19                        1022  4096    4    0.88 0.449%    16.979       65.55   0.00000  0.00000   195
    2.4.19                        1022  4096    8    0.94 0.482%    26.287      136.73   0.00000  0.00000   195
    
    Sequential Writes
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19                        1022  4096    1   27.08 19.02%     0.103      384.58   0.00000  0.00000   142
    2.4.19                        1022  4096    2   26.51 18.75%     0.200      495.79   0.00000  0.00000   141
    2.4.19                        1022  4096    4   25.97 18.50%     0.397      697.91   0.00000  0.00000   140
    2.4.19                        1022  4096    8   26.21 19.39%     0.750     1104.93   0.00000  0.00000   135
    
    Random Writes
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19                        1022  4096    1    0.61 0.352%     0.020        0.56   0.00000  0.00000   174
    2.4.19                        1022  4096    2    0.61 0.314%     0.033       17.76   0.00000  0.00000   195
    2.4.19                        1022  4096    4    0.61 0.353%     0.041       22.03   0.00000  0.00000   174
    2.4.19                        1022  4096    8    0.64 0.286%     0.080       62.96   0.00000  0.00000   223
    Now I'm ready to begin. First thing to do is partition my drives, keeping in mind the rules I outlined in my tutorial on Linux partitioning. My hde and hdg drives are my two 60GB drives, and hdi is the old 40GB drive. Since the kernel cannot boot from a RAID-0 volume, I have to create a small /boot partition to house the kernel. I will put this on hde1 for simplicity. The remainder of my partitions on hde and hdg will be mirrors of each other, so I will make them all logical partitions numbered starting at 5 to avoid mistakes in setting up the RAID array. I make hde5 and hdg5 256MB swap partitions because Linux will automatically stripe swap partitions with the same priority, giving me a total of 512 MB of swap space. Then I make the remaining system partitions as follows:

    hde6/hdg6 50MB each (for /)
    hde7/hdg7 128MB each (for /tmp)
    hde8/hdg8 500MB each (for /var)
    hde9/hdg9 2500MB each (for /usr)

    For the /home partition, I will use 3 drives in the array instead of 2. I will use all the remaining space on the 60GB drives for hde10 and hdg10, as well as the 40GB drive hdi1. VERY IMPORTANT: The partition type on all RAID partitions must be set "Linux Raid Autodetect." In fdisk, use the t command and set the type to "fd" for each of the RAID partitions (i.e., everything except /boot and the 2 swap partitions). Set the 2 swap partitions to type 82, "Linux swap."

    Now we're ready to actually make the RAID array. Create a file called /etc/raidtab and tell Linux which partitions go in which array. Each entry must specify the RAID level, the number of disks in the array, any spare disks (in this case none), the chunk size of the array, and whether the array is to have a persistent superblock. The chunk size is the amount of data that will be written to one disk before another disk is used, and it can be anywhere from 4k to 4MB. Through a little research on Google and some benchmarking, I have found 256k to be a good all-purpose chunk size. The arrays must have a persistent superblock for the kernel to autodetect them.
    Code:
    raiddev /dev/md0
            raid-level              0
            nr-raid-disks           2
            nr-spare-disks          0
            chunk-size              256k
            persistent-superblock   1
                    device          /dev/hde6
                    raid-disk       0
                    device          /dev/hdg6
                    raid-disk       1
    raiddev /dev/md1
            raid-level              0
            nr-raid-disks           2
            nr-spare-disks          0
            chunk-size              256k
            persistent-superblock   1
                    device          /dev/hde7
                    raid-disk       0
                    device          /dev/hdg7
                    raid-disk       1
    raiddev /dev/md2
            raid-level              0
            nr-raid-disks           2
            nr-spare-disks          0
            chunk-size              256k
            persistent-superblock   1
                    device          /dev/hde8
                    raid-disk       0
                    device          /dev/hdg8
                    raid-disk       1
    raiddev /dev/md3
            raid-level              0
            nr-raid-disks           2
            nr-spare-disks          0
            chunk-size              256k
            persistent-superblock   1
                    device          /dev/hde9
                    raid-disk       0
                    device          /dev/hdg9
                    raid-disk       1
    raiddev /dev/md4
            raid-level              0
            nr-raid-disks           3
            nr-spare-disks          0
            chunk-size              256k
            persistent-superblock   1
                    device          /dev/hde10
                    raid-disk       0
                    device          /dev/hdg10
                    raid-disk       1
                    device          /dev/hdi1
                    raid-disk       2
    Once /etc/raidtab is in place, create the RAID arrays with the mkraid command like so:
    Code:
    mkraid /dev/md0
    mkraid /dev/md1
    mkraid /dev/md2
    mkraid /dev/md3
    mkraid /dev/md4
    Then format with the filesystem of your choice. I recommend ext3 for its robustness. See my tutorial on filesystems for more information.
    Code:
    mke2fs -j /dev/hde1
    mke2fs -j /dev/md0
    mkesf2 -j /dev/md2
    mkesf2 -j /dev/md3
    mkesf2 -j /dev/md4
    mkswap /dev/hde5
    mkswap /dev/hdg5
    The RAID arrays are now created and ready to mount. From here, proceed with the normal Gentoo installation beginning with Code Listing 17 (mount partitions). The only caveats:

    - You must copy /etc/raidtab to /mnt/gentoo/etc at the same time as /etc/recolv.conf in code listing 19.
    - /etc/fstab should treat the /dev/mdx devices as regular partitions
    - Be sure to compile multi-device and RAID-0 directly into the kernel and not as modules.
    - Be sure to install (emerge) raidtools before finishing the installation.

    Once it's all compiled, configured, and rebooted, a quick benchmark of my new /home partition yields the following results:
    Code:
    No size specified, using 1022 MB
    
    Unit information
    ================
    File size = megabytes
    Blk Size  = bytes
    Rate      = megabytes per second
    CPU%      = percentage of CPU used during the test
    Latency   = milliseconds
    Lat%      = percent of requests that took longer than X seconds
    CPU Eff   = Rate divided by CPU% - throughput per cpu load
    
    Sequential Reads
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19-crypto-r7              1022  4096    1   78.00 34.56%     0.050       92.62   0.00000  0.00000   226
    2.4.19-crypto-r7              1022  4096    2   69.47 30.47%     0.110      551.78   0.00000  0.00000   228
    2.4.19-crypto-r7              1022  4096    4   68.19 30.63%     0.225      669.62   0.00000  0.00000   223
    2.4.19-crypto-r7              1022  4096    8   65.76 30.30%     0.425     1288.70   0.00000  0.00000   217
    
    Random Reads
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19-crypto-r7              1022  4096    1    0.76 0.870%     5.168       22.01   0.00000  0.00000    87
    2.4.19-crypto-r7              1022  4096    2    1.18 0.967%     6.369       33.38   0.00000  0.00000   122
    2.4.19-crypto-r7              1022  4096    4    1.59 1.218%     9.316       65.98   0.00000  0.00000   130
    2.4.19-crypto-r7              1022  4096    8    1.85 1.162%    14.861      117.12   0.00000  0.00000   159
    
    Sequential Writes
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19-crypto-r7              1022  4096    1   45.53 43.53%     0.070     1942.27   0.00000  0.00000   105
    2.4.19-crypto-r7              1022  4096    2   44.88 38.25%     0.131     2072.38   0.00115  0.00000   117
    2.4.19-crypto-r7              1022  4096    4   44.84 40.09%     0.263     4166.85   0.00345  0.00000   112
    2.4.19-crypto-r7              1022  4096    8   44.61 40.54%     0.483     2939.09   0.00577  0.00000   110
    
    Random Writes
                                  File  Blk   Num                   Avg      Maximum      Lat%     Lat%    CPU
    Identifier                    Size  Size  Thr   Rate  (CPU%)  Latency    Latency      >2s      >10s    Eff
    ---------------------------- ------ ----- ---  ------ ------ --------- -----------  -------- -------- -----
    2.4.19-crypto-r7              1022  4096    1    2.61 1.923%     0.023        4.13   0.00000  0.00000   136
    2.4.19-crypto-r7              1022  4096    2    2.67 2.545%     0.026        1.90   0.00000  0.00000   105
    2.4.19-crypto-r7              1022  4096    4    2.90 2.634%     0.026        3.91   0.00000  0.00000   110
    2.4.19-crypto-r7              1022  4096    8    2.99 2.695%     0.026        1.81   0.00000  0.00000   111
    Not too shabby, eh?

    Further resources:
    RAID and Data Protection Solutions for Linux
    The Software-RAID HOWTO
    Linux-RAID FAQ
    Do what you want with the girl, but leave me alone!

  2. #2
    good tutorial, how many have you done? i saw you mentioned a couple in there. do you think you could send some to my email?
    c0rporate_mupp3t@yahoo.com

  3. #3
    Senior Member problemchild's Avatar
    Join Date
    Jul 2002
    Posts
    551
    Thanks, it's nice to know people really read these things. I have links to all of my tutorials in my AO profile, so you can browse them from there.
    Do what you want with the girl, but leave me alone!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •