Why do I get I/O errors when my LVM snapshot reaches 100% usage?

Solution Verified - Updated -

Issue

  • IO errors for the VG-LV dm-* enumeration seen in messages, journalctl, kernel ring buffer logs:
Raw
May 21 10:31:20 testbox01 kernel: Buffer I/O error on dev dm-200, logical block 82649537, async page read

[2408452.982341] Buffer I/O error on dev dm-200, logical block 82649537, async page read
  • LVM Snapshot has become invalid with I listed under the Attr column of lvs:
Raw
# lvs
  LV                    VG   Attr    LSize   Origin  Data%
  LogVol00      VolGroup01 owi-ao    7.33T
  LogVol00_snap VolGroup01 Swi-Io  877.00G LogVol00 100.00

  • Other possible symptoms are:

  • LVM Snapshot may report IO errors when LVM commands are run:

Raw
/dev/VolGroup01/LogVol00_snap: read failed after 0 of 4096 at 0: Input/output error
  • LVM is reporting a state change in the logs:
Raw
lvm[13911]: Snapshot Volgroup01-LogVol00_snap changed state to: Invalid
  • LVM snapshot was created, when trying to merge it lvconvert --merge, it fails with unable to merge - invalidated snapshot. How to merge the snapshot and recover from this situation?
  • Unable to run LVM commands, When tries to run any LVM command, it shows I/O error message and goes to hung state. Here are the command outputs, Need to break every time using Ctrl+c:
Raw
 # vgs
  /dev/VolGroup01/LogVol00_snap: read failed after 0 of 4096 at 52613283840: Input/output error
  /dev/VolGroup01/LogVol00_snap: read failed after 0 of 4096 at 52613341184: Input/output error
  /dev/VolGroup01/LogVol00_snap: read failed after 0 of 4096 at 0: Input/output error
  /dev/VolGroup01/LogVol00_snap: read failed after 0 of 4096 at 4096: Input/output error
^C  CTRL-c detected: giving up waiting for lock
  /var/lock/lvm/V_VolGroup01: flock failed: Interrupted system call
  Can't get lock for VolGroup01

Environment

  • Red Hat Enterprise Linux 5, 6, 7, 8, 9
  • LVM

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content