dm-crypt and dm-verity no_workqueue options cause memory corruption on RHEL 8 and 9

Solution Verified - Updated -

Issue

  • System using dm-crypt or dm-verity targets gets crashed with memory corruption errors:

    device-mapper: verity: 7:0: data block 8 is corrupted 
    device-mapper: verity: 7:0: reached maximum errors 
    buffer_io_error: 359 callbacks suppressed 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    Buffer I/O error on dev dm-3, logical block 2, async page read 
    device-mapper: verity: 7:0: reached maximum errors 
    ============================================================================= 
    BUG bio-696 (Not tainted): Poison overwritten 
    ----------------------------------------------------------------------------- 
    
    0x000000007ffb4fc2-0x000000007ffb4fc2 @offset=17600. First byte 0x69 instead of 0x6b 
    Allocated in mempool_alloc+0x67/0x180 age=32 cpu=29 pid=968060 
     mempool_alloc+0x67/0x180 
     bio_alloc_bioset+0x14e/0x220 
     bio_clone_fast+0x19/0x70 
     dm_make_request+0x8d/0x4d0 [dm_mod] 
     generic_make_request_no_check+0x202/0x330 
     submit_bio+0x3c/0x160 
     mpage_readahead+0x14a/0x180 
     read_pages+0x95/0x270 
     page_cache_readahead_unbounded+0x1b8/0x250 
     generic_file_buffered_read+0x561/0xbb0 
     blkdev_read_iter+0x41/0x60 
     new_sync_read+0x10f/0x160 
     vfs_read+0x91/0x150 
     ksys_read+0x4f/0xb0 
     do_syscall_64+0x5b/0x1b0 
     entry_SYSCALL_64_after_hwframe+0x61/0xc6 
    Freed in dm_io_complete+0x4a/0x180 [dm_mod] age=32 cpu=29 pid=191 
     dm_io_complete+0x4a/0x180 [dm_mod] 
     clone_endio+0xb6/0x1c0 [dm_mod] 
     tasklet_action_common.isra.19+0x5a/0x110 
     __do_softirq+0xdc/0x2cf 
     run_ksoftirqd+0x2a/0x40 
     smpboot_thread_fn+0xb5/0x150 
     kthread+0x134/0x150 
     ret_from_fork+0x1f/0x40 
    Slab 0x00000000a0eb8574 objects=30 used=1 fp=0x00000000a1a8a55c flags=0x57ffffc0008100(slab|head|node=1|zone=2|lastcpupid=0x1fffff) 
    Object 0x000000004f9886e4 @offset=17472 fp=0x00000000a1a8a55c
    

Environment

  • Red Hat Enterprise Linux 8.5 to 8.10
  • Red Hat Enterprise Linux 9.0 to 9.5
  • dm-crypt targets created with the no_read_workqueue/no_write_workqueue options
  • dm-verity targets with try_verify_in_tasklet option
  • device-mapper

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content