RHEL 7.2 crashed at osq_lock() or dm_bm_prefetch().

Issue

• RHEL - hypervisor Machine crashed during VM migrations:

[6853231.877255] qemu-kvm: sending ioctl 5326 to a partition!
[6853231.877772] qemu-kvm: sending ioctl 80200204 to a partition!
[6853237.589146] BUG: unable to handle kernel paging request at 0000100000000000
[6853237.589747] IP: [<ffffffff810aec7f>] osq_lock+0x3f/0xb0
[6853237.590113] PGD 0
[6853237.590460] Oops: 0002 [#1] SMP
[6853237.590804] Modules linked in: vhost_net vhost macvtap macvlan ebt_arp tun cts binfmt_misc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw openvswitch iptable_filter softdog xprtrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr intel_powerclamp coretemp dm_service_time iTCO_wdt iTCO_vendor_support
[6853237.593398]  intel_rapl kvm_intel kvm mxm_wmi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core pcspkr lpc_ich mfd_core dm_round_robin ipmi_devintf sg ipmi_si ipmi_msghandler wmi shpchp acpi_power_meter dm_multipath 8021q garp mrp bridge stp llc bonding nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs dm_thin_pool dm_persistent_data dm_bufio dm_bio_prison libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common ttm crc32c_intel drm fnic libfcoe libfc i2c_core scsi_transport_fc scsi_tgt enic megaraid_sas dm_mirror dm_region_hash dm_log dm_mod
[6853237.596625] CPU: 4 PID: 29722 Comm: kworker/u96:0 Not tainted 3.10.0-327.36.3.el7.x86_64 #1
[6853237.597212] Hardware name: Cisco Systems Inc UCSB-B200-M4/UCSB-B200-M4, BIOS B200M4.3.1.2a.0.022620161405 02/26/2016
[6853237.597824] Workqueue: dm-thin do_worker [dm_thin_pool]
[6853237.598425] task: ffff88be65f48000 ti: ffff88baa47c8000 task.ti: ffff88baa47c8000
[6853237.599044] RIP: 0010:[<ffffffff810aec7f>]  [<ffffffff810aec7f>] osq_lock+0x3f/0xb0
[6853237.599691] RSP: 0018:ffff88baa47cbcb8  EFLAGS: 00010206
[6853237.600334] RAX: 0000000061b36b60 RBX: 0000100000000000 RCX: ffff88baa47cbfd8
[6853237.600977] RDX: 0000000000000000 RSI: ffff885effd16780 RDI: ffff883e1811b848
[6853237.601628] RBP: ffff88baa47cbcc0 R08: ffff88583d99f8a0 R09: de9e91f05b39f898
[6853237.602300] R10: de9e91f05b39f898 R11: 0000000000000400 R12: ffff883e1811b828
[6853237.602972] R13: ffff88be65f48000 R14: ffff88583d99f800 R15: ffff883e1811b848
[6853237.603633] FS:  0000000000000000(0000) GS:ffff885effd00000(0000) knlGS:0000000000000000
[6853237.604322] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[6853237.604996] CR2: 0000100000000000 CR3: 000000000194a000 CR4: 00000000001427e0
[6853237.605673] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[6853237.606343] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[6853237.607016] Stack:

[6853237.607733]  ffff883e1811b828 ffff88baa47cbd20 ffffffff8163a8ba 0000000000000046
[6853237.608438]  ffff88baa47cbcf8 ffffffff810b5ffd 0000000000ccadee ffff88be675d7300
[6853237.609143]  ffff883e1811b828 ffff883e1811b828 000000331b7c8000 ffff88583d99f800
[6853237.609898] Call Trace:
[6853237.610649]  [<ffffffff8163a8ba>] __mutex_lock_slowpath+0x6a/0x1c0
[6853237.611380]  [<ffffffff810b5ffd>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[6853237.612093]  [<ffffffff81639d7f>] mutex_lock+0x1f/0x2f
[6853237.612848]  [<ffffffffa00bcbb3>] dm_tm_issue_prefetches+0x23/0x60 [dm_persistent_data]
[6853237.613611]  [<ffffffffa00d34d2>] dm_pool_issue_prefetches+0x12/0xb40 [dm_thin_pool]
[6853237.614361]  [<ffffffffa00cfe50>] do_worker+0x60/0x870 [dm_thin_pool]
[6853237.615097]  [<ffffffff8109a84e>] ? move_linked_works+0x5e/0x90
[6853237.615829]  [<ffffffff8109b2a9>] ? pwq_activate_delayed_work+0x39/0x80
[6853237.616571]  [<ffffffff8109d69b>] process_one_work+0x17b/0x470
[6853237.617309]  [<ffffffff8109e46b>] worker_thread+0x11b/0x400
[6853237.618079]  [<ffffffff8109e350>] ? rescuer_thread+0x400/0x400
[6853237.618816]  [<ffffffff810a5b8f>] kthread+0xcf/0xe0
[6853237.619542]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[6853237.620384]  [<ffffffff81646a98>] ret_from_fork+0x58/0x90
[6853237.621115]  [<ffffffff810a5ac0>] ? kthread_create_on_node+0x140/0x140
[6853237.621833] Code: 34 25 e8 ce 00 00 c7 46 10 00 00 00 00 48 c7 06 00 00 00 00 48 89 f3 48 87 1f 48 85 db 48 89 5e 08 75 08 b8 01 00 00 00 5b 5d c3 <48> 89 33 8b 46 10 85 c0 75 ee 65 48 8b 14 25 b8 b7 00 00 eb 0d 
[6853237.623316] RIP  [<ffffffff810aec7f>] osq_lock+0x3f/0xb0
[6853237.624019]  RSP <ffff88baa47cbcb8>
[6853237.624700] CR2: 0000100000000000

• Vmware Virtual Guest :

[3067848.683237] BUG: unable to handle kernel paging request at 0000100000000000
[3067848.683243] IP: [<ffffffff810aebef>] osq_lock+0x3f/0xb0
[3067848.683251] PGD 0 
[3067848.683252] Oops: 0002 [#1] SMP 
[3067848.683254] Modules linked in: binfmt_misc xt_nat xt_addrtype iptable_nat nf_nat_ipv4 iptable_filter ip_tables xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT xt_conntrack nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat nf_conntrack nls_utf8 isofs veth dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop fuse tun bridge stp llc vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon vmw_vmci sg i2c_piix4 pcspkr shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper ttm drm vmxnet3 i2c_core vmw_pvscsi ata_piix libata floppy
[3067848.683281]  dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
[3067848.683285] CPU: 1 PID: 16244 Comm: kworker/u8:0 Not tainted 3.10.0-327.el7.x86_64 #1
[3067848.683286] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[3067848.683290] Workqueue: dm-thin do_worker [dm_thin_pool]
[3067848.683291] task: ffff880eb5100b80 ti: ffff880fc4cb4000 task.ti: ffff880fc4cb4000
[3067848.683292] RIP: 0010:[<ffffffff810aebef>]  [<ffffffff810aebef>] osq_lock+0x3f/0xb0
[3067848.683295] RSP: 0018:ffff880fc4cb7cb8  EFLAGS: 00010206
[3067848.683296] RAX: 00000000385d4ae0 RBX: 0000100000000000 RCX: ffff880fc4cb7fd8
[3067848.683296] RDX: 0000000000000000 RSI: ffff88103fc94740 RDI: ffff880fe65fa848
[3067848.683297] RBP: ffff880fc4cb7cc0 R08: ffff880fb29974a0 R09: dfc0bda9e0397498 
[3067848.683298] R10: dfc0bda9e0397498 R11: 0000000000000001 R12: ffff880fe65fa828
[3067848.683299] R13: ffff880eb5100b80 R14: ffff880fb2997400 R15: ffff880fe65fa848
[3067848.683300] FS:  0000000000000000(0000) GS:ffff88103fc80000(0000) knlGS:0000000000000000
[3067848.683301] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[3067848.683301] CR2: 0000100000000000 CR3: 000000000194a000 CR4: 00000000000407e0
[3067848.683327] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[3067848.683337] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[3067848.683337] Stack:
[3067848.683338]  ffff880fe65fa828 ffff880fc4cb7d20 ffffffff8163968a 0000000000000046
[3067848.683340]  ffff880fc4cb7cf8 ffffffff810b5f5d 0000000037b82deb ffff880faf09f300
[3067848.683341]  ffff880fe65fa828 ffff880fe65fa828 0000000e00068000 ffff880fb2997400
[3067848.683343] Call Trace:
[3067848.683348]  [<ffffffff8163968a>] __mutex_lock_slowpath+0x6a/0x1c0
[3067848.683350]  [<ffffffff810b5f5d>] ? ttwu_do_activate.constprop.84+0x5d/0x70
[3067848.683352]  [<ffffffff81638b4f>] mutex_lock+0x1f/0x2f
[3067848.683357]  [<ffffffffa051ebb3>] dm_tm_issue_prefetches+0x23/0x60 [dm_persistent_data]
[3067848.683360]  [<ffffffffa053b492>] dm_pool_issue_prefetches+0x12/0xb80 [dm_thin_pool]
[3067848.683362]  [<ffffffffa0537e60>] do_worker+0x60/0x870 [dm_thin_pool]
[3067848.683364]  [<ffffffff8109a7ae>] ? move_linked_works+0x5e/0x90
[3067848.683365]  [<ffffffff8109b209>] ? pwq_activate_delayed_work+0x39/0x80
[3067848.683367]  [<ffffffff8109d5fb>] process_one_work+0x17b/0x470
[3067848.683369]  [<ffffffff8109e3cb>] worker_thread+0x11b/0x400
[3067848.683370]  [<ffffffff8109e2b0>] ? rescuer_thread+0x400/0x400
[3067848.683372]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[3067848.683374]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[3067848.683378]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[3067848.683380]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[3067848.683380] Code: 34 25 e8 ce 00 00 c7 46 10 00 00 00 00 48 c7 06 00 00 00 00 48 89 f3 48 87 1f 48 85 db 48 89 5e 08 75 08 b8 01 00 00 00 5b 5d c3 <48> 89 33 8b 46 10 85 c0 75 ee 65 48 8b 14 25 b8 b7 00 00 eb 0d 
[3067848.683394] RIP  [<ffffffff810aebef>] osq_lock+0x3f/0xb0
[3067848.683396]  RSP <ffff880fc4cb7cb8>
[3067848.683397] CR2: 0000100000000000

• Another pattern of crash

[2139073.612969] general protection fault: 0000 [#1] SMP
[2139073.612973] Modules linked in: btrfs zlib_deflate raid6_pq xor vfat msdos fat ext4 mbcache jbd2 veth xt_nat xt_addrtype dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon sg pcspkr i2c_piix4 parport_pc parport shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common
[2139073.613001]  crc32c_intel serio_raw vmwgfx drm_kms_helper ttm vmxnet3 drm ata_piix libata vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
[2139073.613008] CPU: 1 PID: 16786 Comm: kworker/u8:0 Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
[2139073.613009] Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
[2139073.613014] Workqueue: dm-thin do_worker [dm_thin_pool]
[2139073.613015] task: ffff880f1e27d080 ti: ffff880f48b70000 task.ti: ffff880f48b70000
[2139073.613016] RIP: 0010:[<ffffffffa0513d56>]  [<ffffffffa0513d56>] dm_bm_prefetch+0x6/0x20 [dm_persistent_data]
[2139073.613021] RSP: 0018:ffff880f48b73d38  EFLAGS: 00010217
[2139073.613021] RAX: ffff880f1e27d080 RBX: 0000000000000000 RCX: ffff880f48b73fd8
[2139073.613022] RDX: 0000000000000000 RSI: 750a363035303120 RDI: 6e0a333431343132
[2139073.613023] RBP: ffff880f48b73d60 R08: ffff880f811f88a0 R09: dfc183915ebf8898
[2139073.613024] R10: dfc183915ebf8898 R11: 0000000000000400 R12: ffff880f811f5828
[2139073.613024] R13: 6e0a333431343132 R14: ffff880f811f8800 R15: ffff880f811f8898
[2139073.613026] FS:  0000000000000000(0000) GS:ffff880fffc80000(0000) knlGS:0000000000000000
[2139073.613027] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[2139073.613027] CR2: 0000000001028640 CR3: 0000000e3626e000 CR4: 00000000000407e0
[2139073.613031] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[2139073.613059] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[2139073.613060] Stack:
[2139073.613061]  ffff880f48b73d60 ffffffffa0516bcb ffff880f811f8868 ffff880fa4d5fa00
[2139073.613063]  ffff880faa1f6800 ffff880f48b73d70 ffffffffa0533492 ffff880f48b73e18
[2139073.613064]  ffffffffa052fe60 ffff880f811f8898 ffff880f811f8800 ffff880f48b73dc8
[2139073.613066] Call Trace:
[2139073.613070]  [<ffffffffa0516bcb>] ? dm_tm_issue_prefetches+0x3b/0x60 [dm_persistent_data]
[2139073.613073]  [<ffffffffa0533492>] dm_pool_issue_prefetches+0x12/0xb80 [dm_thin_pool]
[2139073.613075]  [<ffffffffa052fe60>] do_worker+0x60/0x870 [dm_thin_pool]
[2139073.613078]  [<ffffffff8109a7ae>] ? move_linked_works+0x5e/0x90
[2139073.613080]  [<ffffffff8109b209>] ? pwq_activate_delayed_work+0x39/0x80
[2139073.613082]  [<ffffffff8109d5fb>] process_one_work+0x17b/0x470
[2139073.613083]  [<ffffffff8109e3cb>] worker_thread+0x11b/0x400
[2139073.613084]  [<ffffffff8109e2b0>] ? rescuer_thread+0x400/0x400
[2139073.613087]  [<ffffffff810a5aef>] kthread+0xcf/0xe0
[2139073.613089]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[2139073.613093]  [<ffffffff81645858>] ret_from_fork+0x58/0x90
[2139073.613094]  [<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[2139073.613095] Code: ff ff ff 4c 89 f6 48 c7 c7 58 a8 51 a0 31 c0 e8 cb ae 11 e1 e9 33 ff ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 <48> 8b 3f ba 01 00 00 00 48 89 e5 e8 fa f9 fd ff 5d c3 0f 1f 84
[2139073.613109] RIP  [<ffffffffa0513d56>] dm_bm_prefetch+0x6/0x20 [dm_persistent_data]
[2139073.613112]  RSP <ffff880f48b73d38>