Menu Close
Red Hat Training
A Red Hat training course is available for RHEL 8
Chapter 8. Configuring RAID logical volumes
You can create, activate, change, remove, display, and use LVM RAID volumes.
8.1. RAID logical volumes
LVM supports RAID levels 0, 1, 4, 5, 6, and 10.
An LVM RAID volume has the following characteristics:
- RAID logical volumes created and managed by LVM leverage the Multiple Devices (MD) kernel drivers.
- You can temporarily split RAID1 images from the array and merge them back into the array later.
- LVM RAID volumes support snapshots.
Clusters
RAID logical volumes are not cluster-aware.
Although you can create and activate RAID logical volumes exclusively on one machine, you cannot activate them simultaneously on more than one machine.
Subvolumes
When you create a RAID logical volume, LVM creates a metadata subvolume that is one extent in size for every data or parity subvolume in the array.
For example, creating a 2-way RAID1 array results in two metadata subvolumes (lv_rmeta_0
and lv_rmeta_1
) and two data subvolumes (lv_rimage_0
and lv_rimage_1
). Similarly, creating a 3-way stripe (plus 1 implicit parity device) RAID4 results in 4 metadata subvolumes (lv_rmeta_0
, lv_rmeta_1
, lv_rmeta_2
, and lv_rmeta_3
) and 4 data subvolumes (lv_rimage_0
, lv_rimage_1
, lv_rimage_2
, and lv_rimage_3
).
Integrity
You can lose data when a RAID device fails or when soft corruption occurs. Soft corruption in data storage implies that the data retrieved from a storage device is different from the data written to that device. Adding integrity to a RAID LV helps mitigate or prevent soft corruption. To learn more about soft corruption and how to add integrity to a RAID LV, see Section 8.6, “Using DM integrity with RAID LV”.
8.2. RAID levels and linear support
RAID supports various configurations, including levels 0, 1, 4, 5, 6, 10, and linear. These RAID types are defined as follows:
- Level 0
RAID level 0, often called striping, is a performance-oriented striped data mapping technique. This means the data being written to the array is broken down into stripes and written across the member disks of the array, allowing high I/O performance at low inherent cost but provides no redundancy.
Many RAID level 0 implementations only stripe the data across the member devices up to the size of the smallest device in the array. This means that if you have multiple devices with slightly different sizes, each device gets treated as though it was the same size as the smallest drive. Therefore, the common storage capacity of a level 0 array is equal to the capacity of the smallest member disk in a Hardware RAID or the capacity of smallest member partition in a Software RAID multiplied by the number of disks or partitions in the array.
- Level 1
RAID level 1, or mirroring, provides redundancy by writing identical data to each member disk of the array, leaving a "mirrored" copy on each disk. Mirroring remains popular due to its simplicity and high level of data availability. Level 1 operates with two or more disks, and provides very good data reliability and improves performance for read-intensive applications but at a relatively high cost.
RAID level 1 comes at a high cost because you write the same information to all of the disks in the array, provides data reliability, but in a much less space-efficient manner than parity based RAID levels such as level 5. However, this space inefficiency comes with a performance benefit: parity-based RAID levels consume considerably more CPU power in order to generate the parity while RAID level 1 simply writes the same data more than once to the multiple RAID members with very little CPU overhead. As such, RAID level 1 can outperform the parity-based RAID levels on machines where software RAID is employed and CPU resources on the machine are consistently taxed with operations other than RAID activities.
The storage capacity of the level 1 array is equal to the capacity of the smallest mirrored hard disk in a Hardware RAID or the smallest mirrored partition in a Software RAID. Level 1 redundancy is the highest possible among all RAID types, with the array being able to operate with only a single disk present.
- Level 4
Level 4 uses parity concentrated on a single disk drive to protect data. Parity information is calculated based on the content of the rest of the member disks in the array. This information can then be used to reconstruct data when one disk in the array fails. The reconstructed data can then be used to satisfy I/O requests to the failed disk before it is replaced and to repopulate the failed disk after it has been replaced.
Because the dedicated parity disk represents an inherent bottleneck on all write transactions to the RAID array, level 4 is seldom used without accompanying technologies such as write-back caching, or in specific circumstances where the system administrator is intentionally designing the software RAID device with this bottleneck in mind (such as an array that will have little to no write transactions once the array is populated with data). RAID level 4 is so rarely used that it is not available as an option in Anaconda. However, it could be created manually by the user if truly needed.
The storage capacity of Hardware RAID level 4 is equal to the capacity of the smallest member partition multiplied by the number of partitions minus one. Performance of a RAID level 4 array is always asymmetrical, meaning reads outperform writes. This is because writes consume extra CPU and main memory bandwidth when generating parity, and then also consume extra bus bandwidth when writing the actual data to disks because you are writing not only the data, but also the parity. Reads need only read the data and not the parity unless the array is in a degraded state. As a result, reads generate less traffic to the drives and across the buses of the computer for the same amount of data transfer under normal operating conditions.
- Level 5
This is the most common type of RAID. By distributing parity across all the member disk drives of an array, RAID level 5 eliminates the write bottleneck inherent in level 4. The only performance bottleneck is the parity calculation process itself. With modern CPUs and Software RAID, that is usually not a bottleneck at all since modern CPUs can generate parity very fast. However, if you have a sufficiently large number of member devices in a software RAID5 array such that the combined aggregate data transfer speed across all devices is high enough, then this bottleneck can start to come into play.
As with level 4, level 5 has asymmetrical performance, and reads substantially outperforming writes. The storage capacity of RAID level 5 is calculated the same way as with level 4.
- Level 6
This is a common level of RAID when data redundancy and preservation, and not performance, are the paramount concerns, but where the space inefficiency of level 1 is not acceptable. Level 6 uses a complex parity scheme to be able to recover from the loss of any two drives in the array. This complex parity scheme creates a significantly higher CPU burden on software RAID devices and also imposes an increased burden during write transactions. As such, level 6 is considerably more asymmetrical in performance than levels 4 and 5.
The total capacity of a RAID level 6 array is calculated similarly to RAID level 5 and 4, except that you must subtract 2 devices (instead of 1) from the device count for the extra parity storage space.
- Level 10
This RAID level attempts to combine the performance advantages of level 0 with the redundancy of level 1. It also helps to alleviate some of the space wasted in level 1 arrays with more than 2 devices. With level 10, it is possible for instance to create a 3-drive array configured to store only 2 copies of each piece of data, which then allows the overall array size to be 1.5 times the size of the smallest devices instead of only equal to the smallest device (like it would be with a 3-device, level 1 array). This avoids CPU process usage to calculate parity like with RAID level 6, but it is less space efficient.
The creation of RAID level 10 is not supported during installation. It is possible to create one manually after installation.
- Linear RAID
Linear RAID is a grouping of drives to create a larger virtual drive.
In linear RAID, the chunks are allocated sequentially from one member drive, going to the next drive only when the first is completely filled. This grouping provides no performance benefit, as it is unlikely that any I/O operations split between member drives. Linear RAID also offers no redundancy and decreases reliability. If any one member drive fails, the entire array cannot be used. The capacity is the total of all member disks.
8.3. LVM RAID segment types
To create a RAID logical volume, you specify a raid type as the --type
argument of the lvcreate
command. The following table describes the possible RAID segment types.
For most users, specifying one of the five available primary types (raid1
, raid4
, raid5
, raid6
, raid10
) should be sufficient.
Table 8.1. LVM RAID segment types
Segment type | Description |
---|---|
|
RAID1 mirroring. This is the default value for the |
| RAID4 dedicated parity disk |
|
Same as |
|
|
|
|
|
|
|
|
|
Same as |
|
|
|
|
|
|
|
|
| Striping. RAID0 spreads logical volume data across multiple data subvolumes in units of stripe size. This is used to increase performance. Logical volume data will be lost if any of the data subvolumes fail. |
8.4. Creating RAID logical volumes
This section provides example commands that create different types of RAID logical volume.
You can create RAID1 arrays with different numbers of copies according to the value you specify for the -m
argument. Similarly, you specify the number of stripes for a RAID 4/5/6 logical volume with the -i argument
. You can also specify the stripe size with the -I
argument.
The following command creates a 2-way RAID1 array named my_lv
in the volume group my_vg
that is one gigabyte in size.
# lvcreate --type raid1 -m 1 -L 1G -n my_lv my_vg
The following command creates a RAID5 array (3 stripes + 1 implicit parity drive) named my_lv
in the volume group my_vg
that is one gigabyte in size. Note that you specify the number of stripes just as you do for an LVM striped volume; the correct number of parity drives is added automatically.
# lvcreate --type raid5 -i 3 -L 1G -n my_lv my_vg
The following command creates a RAID6 array (3 stripes + 2 implicit parity drives) named my_lv
in the volume group my_vg
that is one gigabyte in size.
# lvcreate --type raid6 -i 3 -L 1G -n my_lv my_vg
8.5. Creating a RAID0 (striped) logical volume
A RAID0 logical volume spreads logical volume data across multiple data subvolumes in units of stripe size.
The format for the command to create a RAID0 volume is as follows.
lvcreate --type raid0[_meta] --stripes Stripes --stripesize StripeSize VolumeGroup [PhysicalVolumePath ...]
Table 8.2. RAID0 Command Creation parameters
Parameter | Description |
---|---|
|
Specifying |
| Specifies the number of devices to spread the logical volume across. |
| Specifies the size of each stripe in kilobytes. This is the amount of data that is written to one device before moving to the next device. |
| Specifies the volume group to use. |
| Specifies the devices to use. If this is not specified, LVM will choose the number of devices specified by the Stripes option, one for each stripe. |
This example procedure creates an LVM RAID0 logical volume called mylv
that stripes data across the disks at /dev/sda1
, /dev/sdb1
, and /dev/sdc1
.
Label the disks you will use in the volume group as LVM physical volumes with the
pvcreate
command.WarningThis command destroys any data on
/dev/sda1
,/dev/sdb1
, and/dev/sdc1
.#
pvcreate /dev/sda1 /dev/sdb1 /dev/sdc1
Physical volume "/dev/sda1" successfully created Physical volume "/dev/sdb1" successfully created Physical volume "/dev/sdc1" successfully createdCreate the volume group
myvg
. The following command creates the volume groupmyvg
.#
vgcreate myvg /dev/sda1 /dev/sdb1 /dev/sdc1
Volume group "myvg" successfully createdYou can use the
vgs
command to display the attributes of the new volume group.#
vgs
VG #PV #LV #SN Attr VSize VFree myvg 3 0 0 wz--n- 51.45G 51.45GCreate a RAID0 logical volume from the volume group you have created. The following command creates the RAID0 volume
mylv
from the volume groupmyvg
. This example creates a logical volume that is 2 gigabytes in size, with three stripes and a stripe size of 4 kilobytes.#
lvcreate --type raid0 -L 2G --stripes 3 --stripesize 4 -n mylv myvg
Rounding size 2.00 GiB (512 extents) up to stripe boundary size 2.00 GiB(513 extents). Logical volume "mylv" created.Create a file system on the RAID0 logical volume. The following command creates an
ext4
file system on the logical volume.#
mkfs.ext4 /dev/myvg/mylv
mke2fs 1.44.3 (10-July-2018) Creating filesystem with 525312 4k blocks and 131376 inodes Filesystem UUID: 9d4c0704-6028-450a-8b0a-8875358c0511 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: doneThe following commands mount the logical volume and report the file system disk space usage.
#
mount /dev/myvg/mylv /mnt
#df
Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/myvg-mylv 2002684 6168 1875072 1% /mnt
8.6. Using DM integrity with RAID LV
As a system administrator, you can use device mapper (DM) integrity with a RAID LV to minimize the risk of data loss due to soft corruption or bit rot.
8.6.1. Soft data corruption
Soft corruption in data storage implies that the data retrieved from a storage device is different from the data written to that device. The corrupted data can exist indefinitely on storage devices. You might not discover this corrupted data until you retrieve and attempt to use this data.
Depending on the type of configuration, a Redundant Array of Independent Disks (RAID) LV prevents data loss when a device fails. If a device comprising a RAID array fails, the data can be recovered from other devices that are part of that RAID LV. However, a RAID configuration does not ensure the integrity of the data itself. Soft corruption, silent corruption, soft errors, and silent errors are terms that describe data that has become corrupted, even if the system design and software continues to function as expected.
DM integrity is used with RAID levels 1, 4, 5, 6, and 10 to help mitigate or prevent data loss due to soft corruption. The RAID layer ensures that a non-corrupted copy of the data can fix the soft corruption errors. The integrity layer sits above each RAID image while an extra sub LV stores the integrity metadata (data checksums) for each RAID image. When you retrieve data from an RAID LV with integrity, the integrity data checksums analyze the data for corruption. If corruption is detected, the integrity layer returns an error message, and the RAID layer retrieves a non-corrupted copy of the data from another RAID image. The RAID layer automatically rewrites non-corrupted data over the corrupted data to repair the soft corruption.
When creating a new RAID LV with DM integrity or adding integrity to an existing RAID LV, consider the following:
- The integrity metadata requires additional storage space. For each RAID image, every 500MB data requires 4MB of additional storage space because of the checksums that get added to the data.
- While some RAID configurations are impacted more than others, adding DM integrity impacts performance due to latency when accessing the data. A RAID1 configuration typically offers better performance than RAID5 or its variants.
- The RAID integrity block size also impacts performance. Configuring a larger RAID integrity block size offers better performance. However, a smaller RAID integrity block size offers greater backward compatibility.
- There are two integrity modes available: bitmap or journal. The bitmap integrity mode typically offers better performance than journal mode.
If you experience performance issues, Red Hat recommends that you either use RAID1 with integrity or that you test the performance of a particular RAID configuration to ensure that it meets your requirements.
8.6.2. Creating a RAID LV with DM integrity
When you create an RAID LV, adding DM integrity helps mitigate the risk of losing data due to soft corruption.
Prerequisites
- You must have root access.
Procedure
To create a RAID LV with DM integrity:
# lvcreate --type <raid-level> --raidintegrity y -L <usable-size> -n <logical-volume> <volume-group>
where
<raid-level>
- Specifies the RAID level of the RAID LV that you want to create.
<usable-size>
- Specifies the usable size in MB.
<logical-volume>
- Specifies the name of the LV that you want to create.
<volume-group>
- Specifies the name of the volume group that you want to create the RAID LV under.
In the following example, we create an RAID LV with integrity named test-lv
in the test-vg
volume group, with a usable size of 256M and RAID level 1.
Example RAID LV with integrity
# lvcreate --type raid1 --raidintegrity y -L256M -n test-lv test-vg Creating integrity metadata LV test-lv_rimage_0_imeta with size 8.00 MiB. Logical volume "test-lv_rimage_0_imeta" created. Creating integrity metadata LV test-lv_rimage_1_imeta with size 8.00 MiB. Logical volume "test-lv_rimage_1_imeta" created. Logical volume "test-lv" created.
8.6.3. Adding DM integrity to an existing RAID LV
You can add DM integrity to an existing RAID LV to help mitigate the risk of losing data due to soft corruption.
Prerequisites
- You must have root access.
Procedure
To add DM integrity to an existing RAID LV:
# lvconvert --raidintegrity y <volume-group>/<logical-volume>
where
<volume-group>
- Specifies the name of the volume group that you want to create the RAID LV under.
<logical-volume>
- Specifies the name of the LV that you want to create.
8.6.4. Removing integrity from a RAID LV
Adding integrity to a RAID LV limits the number of operations that you can perform on that RAID LV. Therefore, you must remove the integrity before performing certain operations.
Prerequisites
- You must have root access.
Procedure
To remove integrity from a RAID LV:
# lvconvert --raidintegrity n <volume-group>/<logical-volume>
where
<volume-group>
- Specifies the name of the volume group that you want to create the RAID LV under.
<logical-volume>
- Specifies the name of the LV that you want to create.
8.6.5. Viewing DM integrity information
When you create a RAID LV with integrity or when you add integrity to an existing RAID LV, use the following command to view information about the integrity:
# lvs -a <volume-group>
where <volume-group> is the name of the volume group that contains the RAID LV with integrity.
The following example shows information about the test-lv
RAID LV that was created in the test-vg
volume group.
# lvs -a test-vg LV VG Attr LSize Origin Cpy%Sync test-lv test-vg rwi-a-r--- 256.00m 2.10 [test-lv_rimage_0] test-vg gwi-aor--- 256.00m [test-lv_rimage_0_iorig] 93.75 [test-lv_rimage_0_imeta] test-vg ewi-ao---- 8.00m [test-lv_rimage_0_iorig] test-vg -wi-ao---- 256.00m [test-lv_rimage_1] test-vg gwi-aor--- 256.00m [test-lv_rimage_1_iorig] 85.94 [test-lv_rimage_1_imeta] test-vg ewi-ao---- 8.00m [test-lv_rimage_1_iorig] test-vg -wi-ao---- 256.00m [test-lv_rmeta_0] test-vg ewi-aor--- 4.00m [test-lv_rmeta_1] test-vg ewi-aor--- 4.00m
Synchronization
When you create a RAID LV with integrity or add integrity to an existing RAID LV, we recommend that you wait for the integrity synchronization and the RAID metadata to complete before using the LV. Otherwise, the background initialization might impact the LV’s performance. The Cpy%Sync
column indicates the synchronization progress for both the top level RAID LV and for each RAID image. The RAID image is indicated in the LV column by raid_image_N
. Refer to the LV column to ensure that the synchronization progress displays 100%
for the top level RAID LV and for each RAID image.
RAID images using integrity
The g
attribute in the attributes listed under the Attr
column indicates that the RAID image is using integrity. The integrity checksums are stored in the _imeta
RAID LV.
To display the type for each RAID LV, add the -o+segtype
option to the lvs
command:
# lvs -a my-vg -o+segtype LV VG Attr LSize Origin Cpy%Sync Type test-lv test-vg rwi-a-r--- 256.00m 87.96 raid1 [test-lv_rimage_0] test-vg gwi-aor--- 256.00m [test-lv_rimage_0_iorig] 100.00 integrity [test-lv_rimage_0_imeta] test-vg ewi-ao---- 8.00m linear [test-lv_rimage_0_iorig] test-vg -wi-ao---- 256.00m linear [test-lv_rimage_1] test-vg gwi-aor--- 256.00m [test-lv_rimage_1_iorig] 100.00 integrity [test-lv_rimage_1_imeta] test-vg ewi-ao---- 8.00m linear [test-lv_rimage_1_iorig] test-vg -wi-ao---- 256.00m linear [test-lv_rmeta_0] test-vg ewi-aor--- 4.00m linear [test-lv_rmeta_1] test-vg ewi-aor--- 4.00m linear
Integrity mismatches
There is an incremental counter that counts the number of mismatches detected on each RAID image. To view the data mismatches detected by integrity on a particular RAID image, run the following command:
# lvs -o+integritymismatches <volume-group>/<logical-volume>_raid-image_<n>
where
<volume-group>
- Specifies the name of the volume group that you want to create the RAID LV under.
<logical-volume>
- Specifies the name of the LV that you want to create.
<n>
- Specifies the RAID image that you want to view the integrity mismatch information for.
You must run the command for each RAID image that you want to view. In the following example, we will view the data mismatches from rimage_0
under test-vg/test-lv
.
# lvs -o+integritymismatches test-vg/test-lv_rimage_0 LV VG Attr LSize Origin Cpy%Sync IntegMismatches [test-lv_rimage_0] test-vg gwi-aor--- 256.00m [test-lv_rimage_0_iorig] 100.00 0
We can see that integrity has not detected any data mismatches and thus the IntegMismatches
counter shows zero (0).
Integrity mismatches in kernel message logs
You can also find data integrity information in the kernel message logs, as shown in the following examples.
Example of dm-integrity mismatches from the kernel message logs
device-mapper: integrity: dm-12: Checksum failed at sector 0x24e7
Example of dm-integrity data corrections from the kernel message logs
md/raid1:mdX: read error corrected (8 sectors at 9448 on dm-16)
8.6.6. Additional resources
-
For more information on all the available options, see the
lvmraid
command man page(s).
8.7. Controlling the rate at which RAID volumes are initialized
When you create RAID10 logical volumes, the background I/O required to initialize the logical volumes with a sync
operation can crowd out other I/O operations to LVM devices, such as updates to volume group metadata, particularly when you are creating many RAID logical volumes. This can cause the other LVM operations to slow down.
You can control the rate at which a RAID logical volume is initialized by implementing recovery throttling. You control the rate at which sync
operations are performed by setting the minimum and maximum I/O rate for those operations with the --minrecoveryrate
and --maxrecoveryrate
options of the lvcreate
command. You specify these options as follows.
--maxrecoveryrate Rate[bBsSkKmMgG]
Sets the maximum recovery rate for a RAID logical volume so that it will not crowd out nominal I/O operations. The Rate is specified as an amount per second for each device in the array. If no suffix is given, then kiB/sec/device is assumed. Setting the recovery rate to 0 means it will be unbounded.
--minrecoveryrate Rate[bBsSkKmMgG]
Sets the minimum recovery rate for a RAID logical volume to ensure that I/O for
sync
operations achieves a minimum throughput, even when heavy nominal I/O is present. The Rate is specified as an amount per second for each device in the array. If no suffix is given, then kiB/sec/device is assumed.
The following command creates a 2-way RAID10 array with 3 stripes that is 10 gigabytes in size with a maximum recovery rate of 128 kiB/sec/device. The array is named my_lv
and is in the volume group my_vg
.
# lvcreate --type raid10 -i 2 -m 1 -L 10G --maxrecoveryrate 128 -n my_lv my_vg
You can also specify minimum and maximum recovery rates for a RAID scrubbing operation.
8.8. Converting a Linear device to a RAID device
You can convert an existing linear logical volume to a RAID device by using the --type
argument of the lvconvert
command.
The following command converts the linear logical volume my_lv
in volume group my_vg
to a 2-way RAID1 array.
# lvconvert --type raid1 -m 1 my_vg/my_lv
Since RAID logical volumes are composed of metadata and data subvolume pairs, when you convert a linear device to a RAID1 array, a new metadata subvolume is created and associated with the original logical volume on (one of) the same physical volumes that the linear volume is on. The additional images are added in metadata/data subvolume pairs. For example, if the original device is as follows:
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv /dev/sde1(0)
After conversion to a 2-way RAID1 array the device contains the following data and metadata subvolume pairs:
#lvconvert --type raid1 -m 1 my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 6.25 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sde1(0) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rmeta_0] /dev/sde1(256) [my_lv_rmeta_1] /dev/sdf1(0)
If the metadata image that pairs with the original logical volume cannot be placed on the same physical volume, the lvconvert
will fail.
8.9. Converting an LVM RAID1 logical volume to an LVM linear logical volume
You can convert an existing RAID1 LVM logical volume to an LVM linear logical volume with the lvconvert
command by specifying the -m0
argument. This removes all the RAID data subvolumes and all the RAID metadata subvolumes that make up the RAID array, leaving the top-level RAID1 image as the linear logical volume.
The following example displays an existing LVM RAID1 logical volume.
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0)
[my_lv_rimage_0] /dev/sde1(1)
[my_lv_rimage_1] /dev/sdf1(1)
[my_lv_rmeta_0] /dev/sde1(0)
[my_lv_rmeta_1] /dev/sdf1(0)
The following command converts the LVM RAID1 logical volume my_vg/my_lv
to an LVM linear device.
#lvconvert -m0 my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv /dev/sde1(1)
When you convert an LVM RAID1 logical volume to an LVM linear volume, you can specify which physical volumes to remove. The following example shows the layout of an LVM RAID1 logical volume made up of two images: /dev/sda1
and /dev/sdb1
. In this example, the lvconvert
command specifies that you want to remove /dev/sda1
, leaving /dev/sdb1
as the physical volume that makes up the linear device.
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdb1(0) #lvconvert -m0 my_vg/my_lv /dev/sda1
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv /dev/sdb1(1)
8.10. Converting a mirrored LVM device to a RAID1 device
You can convert an existing mirrored LVM device with a segment type of mirror
to a RAID1 LVM device with the lvconvert
command by specifying the --type raid1
argument. This renames the mirror subvolumes (mimage
) to RAID subvolumes (rimage
). In addition, the mirror log is removed and metadata subvolumes (rmeta
) are created for the data subvolumes on the same physical volumes as the corresponding data subvolumes.
The following example shows the layout of a mirrored logical volume my_vg/my_lv
.
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 15.20 my_lv_mimage_0(0),my_lv_mimage_1(0)
[my_lv_mimage_0] /dev/sde1(0)
[my_lv_mimage_1] /dev/sdf1(0)
[my_lv_mlog] /dev/sdd1(0)
The following command converts the mirrored logical volume my_vg/my_lv
to a RAID1 logical volume.
#lvconvert --type raid1 my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sde1(0) [my_lv_rimage_1] /dev/sdf1(0) [my_lv_rmeta_0] /dev/sde1(125) [my_lv_rmeta_1] /dev/sdf1(125)
8.11. Resizing a RAID logical volume
You can resize a RAID logical volume in the following ways;
-
You can increase the size of a RAID logical volume of any type with the
lvresize
orlvextend
command. This does not change the number of RAID images. For striped RAID logical volumes the same stripe rounding constraints apply as when you create a striped RAID logical volume.
-
You can reduce the size of a RAID logical volume of any type with the
lvresize
orlvreduce
command. This does not change the number of RAID images. As with thelvextend
command, the same stripe rounding constraints apply as when you create a striped RAID logical volume.
-
You can change the number of stripes on a striped RAID logical volume (
raid4/5/6/10
) with the--stripes N
parameter of thelvconvert
command. This increases or reduces the size of the RAID logical volume by the capacity of the stripes added or removed. Note thatraid10
volumes are capable only of adding stripes. This capability is part of the RAID reshaping feature that allows you to change attributes of a RAID logical volume while keeping the same RAID level. For information on RAID reshaping and examples of using thelvconvert
command to reshape a RAID logical volume, see thelvmraid
(7) man page.
8.12. Changing the number of images in an existing RAID1 device
You can change the number of images in an existing RAID1 array just as you can change the number of images in the earlier implementation of LVM mirroring. Use the lvconvert
command to specify the number of additional metadata/data subvolume pairs to add or remove.
When you add images to a RAID1 device with the lvconvert
command, you can specify the total number of images for the resulting device, or you can specify how many images to add to the device. You can also optionally specify on which physical volumes the new metadata/data image pairs will reside.
Metadata subvolumes (named rmeta
) always exist on the same physical devices as their data subvolume counterparts rimage
). The metadata/data subvolume pairs will not be created on the same physical volumes as those from another metadata/data subvolume pair in the RAID array (unless you specify --alloc anywhere
).
The format for the command to add images to a RAID1 volume is as follows:
lvconvert -m new_absolute_count vg/lv [removable_PVs] lvconvert -m +num_additional_images vg/lv [removable_PVs]
For example, the following command displays the LVM device my_vg/my_lv
, which is a 2-way RAID1 array:
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 6.25 my_lv_rimage_0(0),my_lv_rimage_1(0)
[my_lv_rimage_0] /dev/sde1(0)
[my_lv_rimage_1] /dev/sdf1(1)
[my_lv_rmeta_0] /dev/sde1(256)
[my_lv_rmeta_1] /dev/sdf1(0)
The following command converts the 2-way RAID1 device my_vg/my_lv
to a 3-way RAID1 device:
#lvconvert -m 2 my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 6.25 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sde1(0) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rimage_2] /dev/sdg1(1) [my_lv_rmeta_0] /dev/sde1(256) [my_lv_rmeta_1] /dev/sdf1(0) [my_lv_rmeta_2] /dev/sdg1(0)
When you add an image to a RAID1 array, you can specify which physical volumes to use for the image. The following command converts the 2-way RAID1 device my_vg/my_lv
to a 3-way RAID1 device, specifying that the physical volume /dev/sdd1
be used for the array:
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 56.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdb1(0) #lvconvert -m 2 my_vg/my_lv /dev/sdd1
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 28.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rimage_2] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdb1(0) [my_lv_rmeta_2] /dev/sdd1(0)
To remove images from a RAID1 array, use the following command. When you remove images from a RAID1 device with the lvconvert
command, you can specify the total number of images for the resulting device, or you can specify how many images to remove from the device. You can also optionally specify the physical volumes from which to remove the device.
lvconvert -m new_absolute_count vg/lv [removable_PVs] lvconvert -m -num_fewer_images vg/lv [removable_PVs]
Additionally, when an image and its associated metadata subvolume volume are removed, any higher-numbered images will be shifted down to fill the slot. If you remove lv_rimage_1
from a 3-way RAID1 array that consists of lv_rimage_0
, lv_rimage_1
, and lv_rimage_2
, this results in a RAID1 array that consists of lv_rimage_0
and lv_rimage_1
. The subvolume lv_rimage_2
will be renamed and take over the empty slot, becoming lv_rimage_1
.
The following example shows the layout of a 3-way RAID1 logical volume my_vg/my_lv
.
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0)
[my_lv_rimage_0] /dev/sde1(1)
[my_lv_rimage_1] /dev/sdf1(1)
[my_lv_rimage_2] /dev/sdg1(1)
[my_lv_rmeta_0] /dev/sde1(0)
[my_lv_rmeta_1] /dev/sdf1(0)
[my_lv_rmeta_2] /dev/sdg1(0)
The following command converts the 3-way RAID1 logical volume into a 2-way RAID1 logical volume.
#lvconvert -m1 my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdf1(0)
The following command converts the 3-way RAID1 logical volume into a 2-way RAID1 logical volume, specifying the physical volume that contains the image to remove as /dev/sde1
.
#lvconvert -m1 my_vg/my_lv /dev/sde1
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sdf1(1) [my_lv_rimage_1] /dev/sdg1(1) [my_lv_rmeta_0] /dev/sdf1(0) [my_lv_rmeta_1] /dev/sdg1(0)
8.13. Splitting off a RAID image as a separate logical volume
You can split off an image of a RAID logical volume to form a new logical volume.
The format of the command to split off a RAID image is as follows:
lvconvert --splitmirrors count -n splitname vg/lv [removable_PVs]
Just as when you are removing a RAID image from an existing RAID1 logical volume, when you remove a RAID data subvolume (and its associated metadata subvolume) from the middle of the device any higher numbered images will be shifted down to fill the slot. The index numbers on the logical volumes that make up a RAID array will thus be an unbroken sequence of integers.
You cannot split off a RAID image if the RAID1 array is not yet in sync.
The following example splits a 2-way RAID1 logical volume, my_lv
, into two linear logical volumes, my_lv
and new
.
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 12.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdf1(0) #lvconvert --splitmirror 1 -n new my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv /dev/sde1(1) new /dev/sdf1(1)
The following example splits a 3-way RAID1 logical volume, my_lv
, into a 2-way RAID1 logical volume, my_lv
, and a linear logical volume, new
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rimage_2] /dev/sdg1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdf1(0) [my_lv_rmeta_2] /dev/sdg1(0) #lvconvert --splitmirror 1 -n new my_vg/my_lv
#lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdf1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdf1(0) new /dev/sdg1(1)
8.14. Splitting and Merging a RAID Image
You can temporarily split off an image of a RAID1 array for read-only use while keeping track of any changes by using the --trackchanges
argument in conjunction with the --splitmirrors
argument of the lvconvert
command. This allows you to merge the image back into the array at a later time while resyncing only those portions of the array that have changed since the image was split.
The format for the lvconvert
command to split off a RAID image is as follows.
lvconvert --splitmirrors count --trackchanges vg/lv [removable_PVs]
When you split off a RAID image with the --trackchanges
argument, you can specify which image to split but you cannot change the name of the volume being split. In addition, the resulting volumes have the following constraints.
- The new volume you create is read-only.
- You cannot resize the new volume.
- You cannot rename the remaining array.
- You cannot resize the remaining array.
- You can activate the new volume and the remaining array independently.
You can merge an image that was split off with the --trackchanges
argument specified by executing a subsequent lvconvert
command with the --merge
argument. When you merge the image, only the portions of the array that have changed since the image was split are resynced.
The format for the lvconvert
command to merge a RAID image is as follows.
lvconvert --merge raid_image
The following example creates a RAID1 logical volume and then splits off an image from that volume while tracking changes to the remaining array.
#lvcreate --type raid1 -m 2 -L 1G -n my_lv my_vg
Logical volume "my_lv" created #lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sdb1(1) [my_lv_rimage_1] /dev/sdc1(1) [my_lv_rimage_2] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sdb1(0) [my_lv_rmeta_1] /dev/sdc1(0) [my_lv_rmeta_2] /dev/sdd1(0) #lvconvert --splitmirrors 1 --trackchanges my_vg/my_lv
my_lv_rimage_2 split from my_lv for read-only purposes. Use 'lvconvert --merge my_vg/my_lv_rimage_2' to merge back into my_lv #lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sdb1(1) [my_lv_rimage_1] /dev/sdc1(1) my_lv_rimage_2 /dev/sdd1(1) [my_lv_rmeta_0] /dev/sdb1(0) [my_lv_rmeta_1] /dev/sdc1(0) [my_lv_rmeta_2] /dev/sdd1(0)
The following example splits off an image from a RAID1 volume while tracking changes to the remaining array, then merges the volume back into the array.
#lvconvert --splitmirrors 1 --trackchanges my_vg/my_lv
lv_rimage_1 split from my_lv for read-only purposes. Use 'lvconvert --merge my_vg/my_lv_rimage_1' to merge back into my_lv #lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sdc1(1) my_lv_rimage_1 /dev/sdd1(1) [my_lv_rmeta_0] /dev/sdc1(0) [my_lv_rmeta_1] /dev/sdd1(0) #lvconvert --merge my_vg/my_lv_rimage_1
my_vg/my_lv_rimage_1 successfully merged back into my_vg/my_lv #lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sdc1(1) [my_lv_rimage_1] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sdc1(0) [my_lv_rmeta_1] /dev/sdd1(0)
8.15. Setting a RAID fault policy
LVM RAID handles device failures in an automatic fashion based on the preferences defined by the raid_fault_policy
field in the lvm.conf
file.
-
If the
raid_fault_policy
field is set toallocate
, the system will attempt to replace the failed device with a spare device from the volume group. If there is no available spare device, this will be reported to the system log. -
If the
raid_fault_policy
field is set towarn
, the system will produce a warning and the log will indicate that a device has failed. This allows the user to determine the course of action to take.
As long as there are enough devices remaining to support usability, the RAID logical volume will continue to operate.
8.15.1. The allocate RAID Fault Policy
In the following example, the raid_fault_policy
field has been set to allocate
in the lvm.conf
file. The RAID logical volume is laid out as follows.
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0)
[my_lv_rimage_0] /dev/sde1(1)
[my_lv_rimage_1] /dev/sdf1(1)
[my_lv_rimage_2] /dev/sdg1(1)
[my_lv_rmeta_0] /dev/sde1(0)
[my_lv_rmeta_1] /dev/sdf1(0)
[my_lv_rmeta_2] /dev/sdg1(0)
If the /dev/sde
device fails, the system log will display error messages.
# grep lvm /var/log/messages
Jan 17 15:57:18 bp-01 lvm[8599]: Device #0 of raid1 array, my_vg-my_lv, has failed.
Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at
250994294784: Input/output error
Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at
250994376704: Input/output error
Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at 0:
Input/output error
Jan 17 15:57:18 bp-01 lvm[8599]: /dev/sde1: read failed after 0 of 2048 at
4096: Input/output error
Jan 17 15:57:19 bp-01 lvm[8599]: Couldn't find device with uuid
3lugiV-3eSP-AFAR-sdrP-H20O-wM2M-qdMANy.
Jan 17 15:57:27 bp-01 lvm[8599]: raid1 array, my_vg-my_lv, is not in-sync.
Jan 17 15:57:36 bp-01 lvm[8599]: raid1 array, my_vg-my_lv, is now in-sync.
Since the raid_fault_policy
field has been set to allocate
, the failed device is replaced with a new device from the volume group.
# lvs -a -o name,copy_percent,devices vg
Couldn't find device with uuid 3lugiV-3eSP-AFAR-sdrP-H20O-wM2M-qdMANy.
LV Copy% Devices
lv 100.00 lv_rimage_0(0),lv_rimage_1(0),lv_rimage_2(0)
[lv_rimage_0] /dev/sdh1(1)
[lv_rimage_1] /dev/sdf1(1)
[lv_rimage_2] /dev/sdg1(1)
[lv_rmeta_0] /dev/sdh1(0)
[lv_rmeta_1] /dev/sdf1(0)
[lv_rmeta_2] /dev/sdg1(0)
Note that even though the failed device has been replaced, the display still indicates that LVM could not find the failed device. This is because, although the failed device has been removed from the RAID logical volume, the failed device has not yet been removed from the volume group. To remove the failed device from the volume group, you can execute vgreduce --removemissing VG
.
If the raid_fault_policy
has been set to allocate
but there are no spare devices, the allocation will fail, leaving the logical volume as it is. If the allocation fails, you have the option of fixing the drive, then initiating recovery of the failed device with the --refresh
option of the lvchange
command. Alternately, you can replace the failed device.
8.15.2. The warn RAID Fault Policy
In the following example, the raid_fault_policy
field has been set to warn
in the lvm.conf
file. The RAID logical volume is laid out as follows.
# lvs -a -o name,copy_percent,devices my_vg
LV Copy% Devices
my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0)
[my_lv_rimage_0] /dev/sdh1(1)
[my_lv_rimage_1] /dev/sdf1(1)
[my_lv_rimage_2] /dev/sdg1(1)
[my_lv_rmeta_0] /dev/sdh1(0)
[my_lv_rmeta_1] /dev/sdf1(0)
[my_lv_rmeta_2] /dev/sdg1(0)
If the /dev/sdh
device fails, the system log will display error messages. In this case, however, LVM will not automatically attempt to repair the RAID device by replacing one of the images. Instead, if the device has failed you can replace the device with the --repair
argument of the lvconvert
command.
8.16. Replacing a RAID device in a logical volume
You can replace a RAID device in a logical volume.
- If there has been no failure on the RAID device, follow Section 8.16.1, “Replacing a RAID device that has not failed”.
- If the RAID device has failed, follow Section 8.16.4, “Replacing a failed RAID device in a logical volume”.
8.16.1. Replacing a RAID device that has not failed
To replace a RAID device in a logical volume, use the --replace
argument of the lvconvert
command.
Prerequisites
- The RAID device has not failed. The following commands will not work if the RAID device has failed.
Procedure
Replace the RAID device:
# lvconvert --replace dev_to_remove vg/lv possible_replacements
- Replace dev_to_remove with the path to the physical volume that you want to replace.
- Replace vg/lv with the volume group and logical volume name of the RAID array.
- Replace possible_replacements with the path to the physical volume that you want to use as a replacement.
Example 8.1. Replacing a RAID1 device
The following example creates a RAID1 logical volume and then replaces a device in that volume.
Create the RAID1 array:
# lvcreate --type raid1 -m 2 -L 1G -n my_lv my_vg Logical volume "my_lv" created
Examine the RAID1 array:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sdb1(1) [my_lv_rimage_1] /dev/sdb2(1) [my_lv_rimage_2] /dev/sdc1(1) [my_lv_rmeta_0] /dev/sdb1(0) [my_lv_rmeta_1] /dev/sdb2(0) [my_lv_rmeta_2] /dev/sdc1(0)
Replace the
/dev/sdb2
physical volume:# lvconvert --replace /dev/sdb2 my_vg/my_lv
Examine the RAID1 array with the replacement:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 37.50 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sdb1(1) [my_lv_rimage_1] /dev/sdc2(1) [my_lv_rimage_2] /dev/sdc1(1) [my_lv_rmeta_0] /dev/sdb1(0) [my_lv_rmeta_1] /dev/sdc2(0) [my_lv_rmeta_2] /dev/sdc1(0)
Example 8.2. Specifying the replacement physical volume
The following example creates a RAID1 logical volume and then replaces a device in that volume, specifying which physical volume to use for the replacement.
Create the RAID1 array:
# lvcreate --type raid1 -m 1 -L 100 -n my_lv my_vg Logical volume "my_lv" created
Examine the RAID1 array:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdb1(0)
Examine the physical volumes:
# pvs PV VG Fmt Attr PSize PFree /dev/sda1 my_vg lvm2 a-- 1020.00m 916.00m /dev/sdb1 my_vg lvm2 a-- 1020.00m 916.00m /dev/sdc1 my_vg lvm2 a-- 1020.00m 1020.00m /dev/sdd1 my_vg lvm2 a-- 1020.00m 1020.00m
Replace the
/dev/sdb1
physical volume with/dev/sdd1
:# lvconvert --replace /dev/sdb1 my_vg/my_lv /dev/sdd1
Examine the RAID1 array with the replacement:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 28.00 my_lv_rimage_0(0),my_lv_rimage_1(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdd1(0)
Example 8.3. Replacing multiple RAID devices
You can replace more than one RAID device at a time by specifying multiple replace
arguments, as in the following example.
Create a RAID1 array:
# lvcreate --type raid1 -m 2 -L 100 -n my_lv my_vg Logical volume "my_lv" created
Examine the RAID1 array:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rimage_2] /dev/sdc1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdb1(0) [my_lv_rmeta_2] /dev/sdc1(0)
Replace the
/dev/sdb1
and/dev/sdc1
physical volumes:# lvconvert --replace /dev/sdb1 --replace /dev/sdc1 my_vg/my_lv
Examine the RAID1 array with the replacements:
# lvs -a -o name,copy_percent,devices my_vg LV Copy% Devices my_lv 60.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sda1(1) [my_lv_rimage_1] /dev/sdd1(1) [my_lv_rimage_2] /dev/sde1(1) [my_lv_rmeta_0] /dev/sda1(0) [my_lv_rmeta_1] /dev/sdd1(0) [my_lv_rmeta_2] /dev/sde1(0)
8.16.2. Failed devices in LVM RAID
RAID is not like traditional LVM mirroring. LVM mirroring required failed devices to be removed or the mirrored logical volume would hang. RAID arrays can keep on running with failed devices. In fact, for RAID types other than RAID1, removing a device would mean converting to a lower level RAID (for example, from RAID6 to RAID5, or from RAID4 or RAID5 to RAID0).
Therefore, rather than removing a failed device unconditionally and potentially allocating a replacement, LVM allows you to replace a failed device in a RAID volume in a one-step solution by using the --repair
argument of the lvconvert
command.
8.16.3. Recovering a failed RAID device in a logical volume
If the LVM RAID device failure is a transient failure or you are able to repair the device that failed, you can initiate recovery of the failed device.
Prerequisites
- The previously failed device is now working.
Procedure
Refresh the logical volume that contains the RAID device:
# lvchange --refresh my_vg/my_lv
Verification steps
Examine the logical volume with the recovered device:
# lvs --all --options name,devices,lv_attr,lv_health_status my_vg
8.16.4. Replacing a failed RAID device in a logical volume
This procedure replaces a failed device that serves as a physical volume in an LVM RAID logical volume.
Prerequisites
The volume group includes a physical volume that provides enough free capacity to replace the failed device.
If no physical volume with sufficient free extents is available on the volume group, add a new, sufficiently large physical volume using the
vgextend
utility.
Procedure
In the following example, a RAID logical volume is laid out as follows:
# lvs --all --options name,copy_percent,devices my_vg LV Cpy%Sync Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdc1(1) [my_lv_rimage_2] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdc1(0) [my_lv_rmeta_2] /dev/sdd1(0)
If the
/dev/sdc
device fails, the output of thelvs
command is as follows:# lvs --all --options name,copy_percent,devices my_vg /dev/sdc: open failed: No such device or address Couldn't find device with uuid A4kRl2-vIzA-uyCb-cci7-bOod-H5tX-IzH4Ee. WARNING: Couldn't find all devices for LV my_vg/my_lv_rimage_1 while checking used and assumed devices. WARNING: Couldn't find all devices for LV my_vg/my_lv_rmeta_1 while checking used and assumed devices. LV Cpy%Sync Devices my_lv 100.00 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] [unknown](1) [my_lv_rimage_2] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] [unknown](0) [my_lv_rmeta_2] /dev/sdd1(0)
Replace the failed device and display the logical volume:
# lvconvert --repair my_vg/my_lv /dev/sdc: open failed: No such device or address Couldn't find device with uuid A4kRl2-vIzA-uyCb-cci7-bOod-H5tX-IzH4Ee. WARNING: Couldn't find all devices for LV my_vg/my_lv_rimage_1 while checking used and assumed devices. WARNING: Couldn't find all devices for LV my_vg/my_lv_rmeta_1 while checking used and assumed devices. Attempt to replace failed RAID images (requires full device resync)? [y/n]: y Faulty devices in my_vg/my_lv successfully replaced.
Optional: To manually specify the physical volume that replaces the failed device, add the physical volume at the end of the command:
# lvconvert --repair my_vg/my_lv replacement_pv
Examine the logical volume with the replacement:
# lvs --all --options name,copy_percent,devices my_vg /dev/sdc: open failed: No such device or address /dev/sdc1: open failed: No such device or address Couldn't find device with uuid A4kRl2-vIzA-uyCb-cci7-bOod-H5tX-IzH4Ee. LV Cpy%Sync Devices my_lv 43.79 my_lv_rimage_0(0),my_lv_rimage_1(0),my_lv_rimage_2(0) [my_lv_rimage_0] /dev/sde1(1) [my_lv_rimage_1] /dev/sdb1(1) [my_lv_rimage_2] /dev/sdd1(1) [my_lv_rmeta_0] /dev/sde1(0) [my_lv_rmeta_1] /dev/sdb1(0) [my_lv_rmeta_2] /dev/sdd1(0)
Until you remove the failed device from the volume group, LVM utilities still indicate that LVM cannot find the failed device.
Remove the failed device from the volume group:
# vgreduce --removemissing VG
8.17. Checking data coherency in a RAID logical volume (RAID scrubbing)
LVM provides scrubbing support for RAID logical volumes. RAID scrubbing is the process of reading all the data and parity blocks in an array and checking to see whether they are coherent.
Procedure
Optional: Limit the I/O bandwidth that the scrubbing process uses.
When you perform a RAID scrubbing operation, the background I/O required by the
sync
operations can crowd out other I/O to LVM devices, such as updates to volume group metadata. This might cause the other LVM operations to slow down. You can control the rate of the scrubbing operation by implementing recovery throttling.Add the following options to the
lvchange --syncaction
commands in the next steps:--maxrecoveryrate Rate[bBsSkKmMgG]
- Sets the maximum recovery rate so that the operation does crowd out nominal I/O operations. Setting the recovery rate to 0 means that the operation is unbounded.
--minrecoveryrate Rate[bBsSkKmMgG]
-
Sets the minimum recovery rate to ensure that I/O for
sync
operations achieves a minimum throughput, even when heavy nominal I/O is present.
Specify the Rate value as an amount per second for each device in the array. If you provide no suffix, the options assume kiB per second per device.
Display the number of discrepancies in the array, without repairing them:
# lvchange --syncaction check vg/raid_lv
Correct the discrepancies in the array:
# lvchange --syncaction repair vg/raid_lv
NoteThe
lvchange --syncaction repair
operation does not perform the same function as thelvconvert --repair
operation:-
The
lvchange --syncaction repair
operation initiates a background synchronization operation on the array. -
The
lvconvert --repair
operation repairs or replaces failed devices in a mirror or RAID logical volume.
-
The
Optional: Display information about the scrubbing operation:
# lvs -o +raid_sync_action,raid_mismatch_count vg/lv
The
raid_sync_action
field displays the current synchronization operation that the RAID volume is performing. It can be one of the following values:idle
- All sync operations complete (doing nothing)
resync
- Initializing an array or recovering after a machine failure
recover
- Replacing a device in the array
check
- Looking for array inconsistencies
repair
- Looking for and repairing inconsistencies
-
The
raid_mismatch_count
field displays the number of discrepancies found during acheck
operation. -
The
Cpy%Sync
field displays the progress of thesync
operations. The
lv_attr
field provides additional indicators. Bit 9 of this field displays the health of the logical volume, and it supports the following indicators:-
m
(mismatches) indicates that there are discrepancies in a RAID logical volume. This character is shown after a scrubbing operation has detected that portions of the RAID are not coherent. -
r
(refresh) indicates that a device in a RAID array has suffered a failure and the kernel regards it as failed, even though LVM can read the device label and considers the device to be operational. Refresh the logical volume to notify the kernel that the device is now available, or replace the device if you suspect that it failed.
-
Additional resources
-
For more information, see the
lvchange(8)
andlvmraid(7)
man pages.
8.18. Converting a RAID level (RAID takeover)
LVM supports Raid takeover, which means converting a RAID logical volume from one RAID level to another (such as from RAID 5 to RAID 6). Changing the RAID level is usually done to increase or decrease resilience to device failures or to restripe logical volumes. You use the lvconvert
for RAID takeover. For information on RAID takeover and for examples of using the lvconvert
to convert a RAID logical volume, see the lvmraid
(7) man page.
8.19. Changing attributes of a RAID volume (RAID reshape)
RAID reshaping means changing attributes of a RAID logical volume while keeping the same RAID level. Some attributes you can change include RAID layout, stripe size, and number of stripes. For information on RAID reshaping and examples of using the lvconvert
command to reshape a RAID logical volume, see the lvmraid
(7) man page.
8.20. Controlling I/O Operations on a RAID1 logical volume
You can control the I/O operations for a device in a RAID1 logical volume by using the --writemostly
and --writebehind
parameters of the lvchange
command. The format for using these parameters is as follows.
--[raid]writemostly PhysicalVolume[:{t|y|n}]
Marks a device in a RAID1 logical volume as
write-mostly
. All reads to these drives will be avoided unless necessary. Setting this parameter keeps the number of I/O operations to the drive to a minimum. By default, thewrite-mostly
attribute is set to yes for the specified physical volume in the logical volume. It is possible to remove thewrite-mostly
flag by appending:n
to the physical volume or to toggle the value by specifying:t
. The--writemostly
argument can be specified more than one time in a single command, making it possible to toggle the write-mostly attributes for all the physical volumes in a logical volume at once.--[raid]writebehind IOCount
Specifies the maximum number of outstanding writes that are allowed to devices in a RAID1 logical volume that are marked as
write-mostly
. Once this value is exceeded, writes become synchronous, causing all writes to the constituent devices to complete before the array signals the write has completed. Setting the value to zero clears the preference and allows the system to choose the value arbitrarily.
8.21. Changing the region size on a RAID logical volume
When you create a RAID logical volume, the region size for the logical volume will be the value of the raid_region_size
parameter in the /etc/lvm/lvm.conf
file. You can override this default value with the -R
option of the lvcreate
command.
After you have created a RAID logical volume, you can change the region size of the volume with the -R
option of the lvconvert
command. The following example changes the region size of logical volume vg/raidlv
to 4096K. The RAID volume must be synced in order to change the region size.
#lvconvert -R 4096K vg/raid1
Do you really want to change the region_size 512.00 KiB of LV vg/raid1 to 4.00 MiB? [y/n]:y
Changed region size on RAID LV vg/raid1 to 4.00 MiB.