Physically removed a disk before deactivating volume group and can't get LVM to stop printing errors about it

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 5,6,7,8 & 9
  • LVM

Issue

  • Physically removed the lone physical volume backing a volume group. The device nodes for this disk (/dev/sdb & /dev/sdb1) have not disappeared and now we can't get rid of the volume group or the LVs (with vgremove or vgreduce --removemissing). Additionally, every single lvm command returns errors like the following:
[root@localhost ~]# vgdisplay myvg
File descriptor 5 (/dev/pts/2) leaked on vgdisplay invocation. Parent PID 20243: bash
  /dev/myvg/mylv: read failed after 0 of 4096 at 5368643584: Input/output error
  /dev/myvg/mylv: read failed after 0 of 4096 at 5368700928: Input/output error
  /dev/myvg/mylv: read failed after 0 of 4096 at 0: Input/output error
  /dev/myvg/mylv: read failed after 0 of 4096 at 4096: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 0: Input/output error
  /dev/sdb1: read failed after 0 of 2048 at 0: Input/output error
  /dev/sdb: read failed after 0 of 4096 at 268435390464: Input/output error
  /dev/sdb1: read failed after 0 of 512 at 107372675072: Input/output error
  /dev/sdb1: read failed after 0 of 512 at 107372761088: Input/output error
  /dev/sdb1: read failed after 0 of 512 at 0: Input/output error
  /dev/sdb1: read failed after 0 of 512 at 4096: Input/output error
Volume group "myvg" not found
  • The kernel ring buffer, messages, and journalctl logs may indicate errors trying to access the dm device listed in dmsetup for the LV(s), eg:
[25647659.361078] Buffer I/O error on dev dm-5, logical block 15386566, async page read
[25647660.554409] Buffer I/O error on dev dm-5, logical block 0, async page read
[25647660.554688] Buffer I/O error on dev dm-5, logical block 0, async page read
[25647660.554838] Buffer I/O error on dev dm-5, logical block 0, async page read

Resolution

Remove the stale /dev/sdX device node and clean up the stale device-mapper nodes. In the above example, this would be accomplished by either a simple reboot, or by running the following.

# dmsetup remove /dev/myvg/*
# echo 1 > /sys/block/sdb/device/delete

Note: It could be possible that the delete procedure can't be applied due to the no such file or directory message, in that case, you'll have to schedule a maintenance windows to reboot the host to clear the I/O failure messages.

Root Cause

When the disk was physically removed, the /dev/sdb and /dev/sdb1 device nodes weren't automatically removed. The above errors are clearly indicating that /dev/sdb, /dev/sdb1, and /dev/myvg/mylv can no longer be read due to the removal of the disk. In this particular case, the fact that the removed disk's first partition (/dev/sdb1) was the lone PV backing the myvg volume group means that myvg can no longer function.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments