'attempt to access beyond end of device' errors result in lost reads, access to invalid memory references and crash in Red Hat Enterprise Linux 7

Solution Verified - Updated -

Issue

  • Kernel crashed at "memset" in mounting a truncated iso with following log :
[  246.254952] attempt to access beyond end of device
[  226.254954] loop0: rw=0, want=1067676, limit=1066976
[  226.261562] BUG: unable to handle kernel paging request at ffff9dd42b000000
[  226.261688] IP: [<ffffffffaef865f9>] memset+0x9/0xb0
[  226.261788] PGD 7c252067 PUD 7c253067 PMD 7a3a5063 PTE 800000002b000061
[  226.261888] Oops: 0003 [#1] SMP
[  226.261980] Modules linked in: nls_utf8 isofs loop ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_intel snd_hda_codec lrw gf128mul snd_hda_core snd_hwdep snd_seq glue_helper ppdev ablk_helper cryptd snd_seq_device snd_pcm snd_timer pcspkr snd sg joydev virtio_balloon soundcore parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom
[  226.262721]  virtio_net virtio_blk virtio_console ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel 8139too serio_raw 8139cp mii qxl drm_kms_helper syscopyarea sysfillrect sysimgblt floppy fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
[  226.263070] CPU: 0 PID: 4155 Comm: loop0 Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
[  226.263166] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[  226.263256] task: ffff9dd47a2dc100 ti: ffff9dd478eb8000 task.ti: ffff9dd478eb8000
[  226.263359] RIP: 0010:[<ffffffffaef865f9>]  [<ffffffffaef865f9>] memset+0x9/0xb0
[  226.263482] RSP: 0018:ffff9dd478ebbd90  EFLAGS: 00010286
[  226.263577] RAX: ffff9dd428c4f000 RBX: 000000000000001f RCX: 00000000fdc3a800
[  226.263670] RDX: 00000000fffeb800 RSI: 0000000000000000 RDI: ffff9dd42b000000
[  226.263764] RBP: ffff9dd478ebbda8 R08: 000000005c5cb1df R09: ffff9dd428c4f000
[  226.263857] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9dd477853f00
[  226.263951] R13: ffff9dd477853f00 R14: 0000000000000009 R15: ffff9dd43568c400
[  226.264044] FS:  0000000000000000(0000) GS:ffff9dd47fc00000(0000) knlGS:0000000000000000
[  226.264141] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  226.264233] CR2: ffff9dd42b000000 CR3: 0000000078dd6000 CR4: 00000000001606f0
[  226.264330] Call Trace:
[  226.264427]  [<ffffffffaee7ca5c>] ? zero_fill_bio+0x5c/0x80
[  226.264529]  [<ffffffffc08b5e48>] do_bio_filebacked+0x198/0x320 [loop]
[  226.264641]  [<ffffffffaedb971f>] ? mempool_free+0x4f/0xa0
[  226.264733]  [<ffffffffc08b6521>] loop_thread+0x101/0x2e0 [loop]
[  226.264828]  [<ffffffffaecc2d00>] ? wake_up_atomic_t+0x30/0x30
[  226.264920]  [<ffffffffc08b6420>] ? loop_attr_do_show_backing_file+0xb0/0xb0 [loop]
[  226.265016]  [<ffffffffaecc1c31>] kthread+0xd1/0xe0
[  226.265104]  [<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
[  226.265197]  [<ffffffffaf374c37>] ret_from_fork_nospec_begin+0x21/0x21
[  226.265288]  [<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
[  226.265386] Code: 16 fe 66 44 89 1f 66 44 89 54 17 fe eb 0c 48 83 fa 01 72 06 44 8a 1e 44 88 1f c3 90 90 90 90 90 90 90 49 89 f9 40 88 f0 48 89 d1 <f3> aa 4c 89 c8 c3 0f 1f 84 00 00 00 00 00 0f 1f 84 00 00 00 00
[  226.265670] RIP  [<ffffffffaef865f9>] memset+0x9/0xb0
[  226.265760]  RSP <ffff9dd478ebbd90>
[  226.265843] CR2: ffff9dd42b000000
  • another call trace
[  132.093860] attempt to access beyond end of device
[  132.093862] loop0: rw=0, want=200432, limit=188032
[  132.093918] BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
[  132.093924] IP: [<ffffffffb043c7d6>] bdev_read_page+0x26/0xa0
[  132.093937] PGD 13d067 PUD 23f22067 PMD 0
[  132.093942] Oops: 0000 [#1] SMP
[  132.093946] Modules linked in: sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc nls_utf8 isofs loop uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_generic joydev pcspkr snd_hda_intel virtio_balloon
[  132.093991]  snd_hda_codec sg snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm parport_pc parport snd_timer snd soundcore i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_net virtio_console ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel 8139too serio_raw 8139cp mii qxl drm_kms_helper floppy syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log dm_mod
[  132.094028] CPU: 1 PID: 15787 Comm: cat Not tainted 3.10.0-658.el7.x86_64 #1
[  132.094030] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[  132.094032] task: ffffa006d1642f10 ti: ffffa006fa3d4000 task.ti: ffffa006fa3d4000
[  132.094033] RIP: 0010:[<ffffffffb043c7d6>]  [<ffffffffb043c7d6>] bdev_read_page+0x26/0xa0
[  132.094044] RSP: 0018:ffffa006fa3d7a40  EFLAGS: 00010246
[  132.094045] RAX: 00000000ffffffa1 RBX: ffffa006fcc25040 RCX: ffffa006e2194c00
[  132.094046] RDX: fffffed6015381c0 RSI: 0000000000030ef0 RDI: ffffa006fcc25040
[  132.094048] RBP: ffffa006fa3d7a60 R08: 0000000000019c40 R09: ffffffffb0385307
[  132.094049] R10: ffffa006ffd19c40 R11: 0000000000030ef0 R12: 0000000000000000
[  132.094050] R13: ffffa006fa3d7b70 R14: 000000000000af80 R15: 0000000000000000
[  132.094052] FS:  00007fdcb0fdd740(0000) GS:ffffa006ffd00000(0000) knlGS:0000000000000000
[  132.094053] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  132.094054] CR2: 0000000000000060 CR3: 000000007a78b000 CR4: 00000000001406e0
[  132.094061] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  132.094063] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  132.094064] Stack:
[  132.094065]  0000000000001000 0000000000000002 ffffa006fa3d7b70 000000000000af80
[  132.094069]  ffffa006fa3d7b38 ffffffffb04425ef ffffa006a5584a00 ffffa006fa3d7b60
[  132.094071]  0000000000000002 00000001015381c0 ffffa006fa3d7b68 ffffffffc06a3530
[  132.094074] Call Trace:
[  132.094081]  [<ffffffffb04425ef>] do_mpage_readpage+0x57f/0x740
[  132.094092]  [<ffffffffc06a3530>] ? isofs_get_blocks+0x2a0/0x2a0 [isofs]
[  132.094100]  [<ffffffffb0391b1e>] ? lru_cache_add+0xe/0x10
[  132.094105]  [<ffffffffb044289b>] mpage_readpages+0xeb/0x150
[  132.094112]  [<ffffffffc06a3530>] ? isofs_get_blocks+0x2a0/0x2a0 [isofs]
[  132.094115]  [<ffffffffc06a3530>] ? isofs_get_blocks+0x2a0/0x2a0 [isofs]
[  132.094119]  [<ffffffffc06a261d>] isofs_readpages+0x1d/0x20 [isofs]
[  132.094125]  [<ffffffffb038f9fc>] __do_page_cache_readahead+0x1cc/0x250
[  132.094131]  [<ffffffffb038fbf6>] ondemand_readahead+0x116/0x230
[  132.094135]  [<ffffffffb038fd90>] page_cache_async_readahead+0x80/0xa0
[  132.094141]  [<ffffffffb038209e>] ? __find_get_page+0x1e/0xa0
[  132.094145]  [<ffffffffb03841d8>] generic_file_aio_read+0x558/0x790
[  132.094153]  [<ffffffffb03ffd1d>] do_sync_read+0x8d/0xd0
[  132.094160]  [<ffffffffb040070c>] vfs_read+0x9c/0x170
[  132.094162]  [<ffffffffb04015cf>] SyS_read+0x7f/0xe0
[  132.094170]  [<ffffffffb08b05c9>] system_call_fastpath+0x16/0x1b
[  132.094171] Code: 00 00 00 00 00 0f 1f 44 00 00 55 b8 a1 ff ff ff 48 89 e5 41 56 41 55 41 54 53 48 8b 8f 98 00 00 00 48 89 fb 4c 8b a1 58 03 00 00 <49> 83 7c 24 60 00 74 22 48 83 b9 a0 03 00 00 00 75 18 48 8b bf
[  132.094198] RIP  [<ffffffffb043c7d6>] bdev_read_page+0x26/0xa0
[  132.094208]  RSP <ffffa006fa3d7a40>
[  132.094209] CR2: 0000000000000060

Environment

  • Red Hat Enterprise Linux 7
  • kernel-3.10.0-693 ~ kernel-3.10.0-957.z
  • loop device / truncated iso file

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content