System not responding in smp_call_function_many in vxfs running system

Issue

• System was not responding and it kept generating below type of messages in the log.

...
[13753504.026002] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [opcmona:5351]
[13753504.026002] Modules linked in: bmhook(OE) tmhook(OE) ip6table_filter ip6_tables rpcsec_gss_krb5 iptable_filter tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner iptable_security xt_conntrack nf_conntrack vxspec(POE) vxio(POE) vxdmp(POE) vxglm(POE) ext4 mbcache jbd2 vxcafs(POE) vxportal(POE) fdd(POE) vxfs(POE) veki(POE) mlx5_ib ib_uverbs ib_core mlx5_core mlxfw devlink joydev pci_hyperv hv_utils sg ptp pps_core hv_balloon nfit libnvdimm iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper i2c_piix4 ablk_helper cryptd pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi hv_storvsc scsi_transport_fc
[13753504.026002]  hv_netvsc hid_hyperv hyperv_keyboard scsi_tgt hyperv_fb ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw hv_vmbus floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: tmhook]
[13753504.026002] CPU: 0 PID: 5351 Comm: opcmona Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.76.1.el7.x86_64 #1
[13753504.026002] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090008  12/07/2018
[13753504.026002] task: ffff90d69326a100 ti: ffff90d6c0978000 task.ti: ffff90d6c0978000
[13753504.026002] RIP: 0010:[<ffffffffb951718e>]  [<ffffffffb951718e>] smp_call_function_many+0x20e/0x270
[13753504.026002] RSP: 0018:ffff90d6c097bb60  EFLAGS: 00000202
[13753504.026002] RAX: 0000000000000003 RBX: ffff90d6c097bb20 RCX: ffff90d6ffd9fc10
[13753504.026002] RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
[13753504.026002] RBP: ffff90d6c097bb98 R08: ffff90d3bfd52000 R09: ffffffffb97869f9
[13753504.026002] R10: ffff90d6ffc1f160 R11: ffffcd58cf4f6c00 R12: ffff90d6c097bb30
[13753504.026002] R13: ffffffffb978694f R14: 0000000000000025 R15: 0000000001320122
[13753504.026002] FS:  00007f606b839700(0000) GS:ffff90d6ffc00000(0000) knlGS:0000000000000000
[13753504.026002] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13753504.026002] CR2: 00007f606b8ba650 CR3: 000000045317a000 CR4: 00000000003606f0
[13753504.026002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[13753504.026002] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[13753504.026002] Call Trace:
[13753504.026002]  [<ffffffffb95c7920>] ? drain_pages+0xb0/0xb0
[13753504.026002]  [<ffffffffb95172d9>] on_each_cpu_mask+0x29/0x70
[13753504.026002]  [<ffffffffb95c4805>] drain_all_pages+0xb5/0xc0
[13753504.026002]  [<ffffffffb95c968b>] __alloc_pages_nodemask+0x8cb/0xbe0
[13753504.026002]  [<ffffffffb94989bd>] copy_process+0x1dd/0x1a80
[13753504.026002]  [<ffffffffb94e283c>] ? set_next_entity+0x3c/0xe0
[13753504.026002]  [<ffffffffb949a411>] do_fork+0x91/0x330
[13753504.026002]  [<ffffffffc15aeb1c>] ? get_next_hook+0x5c/0x80 [tmhook]
[13753504.026002]  [<ffffffffb949a736>] SyS_clone+0x16/0x20
[13753504.026002]  [<ffffffffc15af38e>] tmhook_nonsysentry_handler+0x19e/0x370 [tmhook]
[13753504.026002]  [<ffffffffb9b9a374>] stub_clone+0x44/0x70
[13753504.026002]  [<ffffffffb9b99f92>] ? system_call_fastpath+0x25/0x2a
[13753504.026002] Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
...