Ceph client crashes with a message "kernel BUG at net/ceph/messenger.c:3251!" in ceph_msg_data_add_pages() due to slab OOM

Issue

• Ceph client crashes with a message "kernel BUG at net/ceph/messenger.c:3251!" in ceph_msg_data_add_pages() due to slab OOM

[5053348.557621] xt_physdev: using --physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
[5053378.581656] xt_physdev: using --physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
[5053408.605612] xt_physdev: using --physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
[5053438.629764] xt_physdev: using --physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
[5053468.654522] xt_physdev: using --physdev-out and --physdev-is-out are only supported in the FORWARD and POSTROUTING chains with bridged traffic.
[5053489.224499] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[5053489.225002]   cache: kmalloc-8(594:abb7de1f264158d865752a89aca04ad6821c785ec472c6b1a5c10a72bb5b1e11), object size: 8, buffer size: 8, default order: 0, min order: 0
[5053489.225002]   node 0: slabs: 3, objs: 1536, free: 0
[5053489.255900] SLUB: Unable to allocate memory on node -1 (gfp=0xd0)
[5053489.256002]   cache: kmalloc-8(594:abb7de1f264158d865752a89aca04ad6821c785ec472c6b1a5c10a72bb5b1e11), object size: 8, buffer size: 8, default order: 0, min order: 0
[5053489.256002]   node 0: slabs: 3, objs: 1536, free: 0
[5053491.492272] SLUB: Unable to allocate memory on node -1 (gfp=0x50)
[5053491.493005]   cache: ceph_osd_request(598:150a1d5d938e7a91c055b04b84c47c7a14088ac1e0665308d85e54322dbe96a3), object size: 1056, buffer size: 1056, default order: 3, min order: 0
[5053491.493005]   node 0: slabs: 4, objs: 124, free: 0
[5053491.506192] SLUB: Unable to allocate memory on node -1 (gfp=0x50)
[5053491.507003]   cache: ceph_osd_request(598:150a1d5d938e7a91c055b04b84c47c7a14088ac1e0665308d85e54322dbe96a3), object size: 1056, buffer size: 1056, default order: 3, min order: 0
[5053491.511020]   node 0: slabs: 5, objs: 155, free: 0
[5053491.518304] SLUB: Unable to allocate memory on node -1 (gfp=0x8050)
[5053491.519003]   cache: shared_policy_node(598:150a1d5d938e7a91c055b04b84c47c7a14088ac1e0665308d85e54322dbe96a3), object size: 48, buffer size: 48, default order: 0, min order: 0
[5053491.524013]   node 0: slabs: 7, objs: 595, free: 0
[5053491.528271] ------------[ cut here ]------------
[5053491.529224] kernel BUG at net/ceph/messenger.c:3251!
[5053491.529224] invalid opcode: 0000 [#1] SMP 
[5053491.529224] Modules linked in: ceph ipt_REJECT nf_reject_ipv4 ip6table_nat ip6_tables rbd libceph dns_resolver veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 xt_NFLOG xt_physdev nfnetlink_log xt_statistic xt_nat ip_set_hash_ip xt_set ip_set xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 mbcache jbd2 ppdev iosf_mbi kvm_intel kvm cirrus ttm drm_kms_helper irqbypass crc32_pclmul syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel parport_pc aesni_intel drm parport lrw gf128mul glue_helper ablk_helper joydev cryptd virtio_balloon
[5053491.543770]  pcspkr i2c_piix4 drm_panel_orientation_quirks ip_tables xfs libcrc32c ata_generic pata_acpi virtio_net virtio_blk floppy ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio sunrpc
[5053491.543770] CPU: 8 PID: 7467 Comm: python Kdump: loaded Tainted: G               ------------ T 3.10.0-957.1.3.el7.x86_64 #1
[5053491.543770] Hardware name: OpenStack Foundation OpenStack Nova, BIOS Ubuntu-1.8.2-1ubuntu1~cloud0 04/01/2014
[5053491.543770] task: ffff9178c1f44100 ti: ffff91798cf48000 task.ti: ffff91798cf48000
[5053491.543770] RIP: 0010:[<ffffffffc0781a07>]  [<ffffffffc0781a07>] ceph_msg_data_add_pages+0x67/0x70 [libceph]
[5053491.543770] RSP: 0018:ffff91798cf4b778  EFLAGS: 00010246
[5053491.543770] RAX: 0000000000000000 RBX: ffff9173637620f0 RCX: 0000000000000001
[5053491.543770] RDX: 0000000000000030 RSI: 0000000000000000 RDI: 0000000000000246
[5053491.543770] RBP: ffff91798cf4b798 R08: ffffffff8795b148 R09: ffff917991ab6f00
[5053491.543770] R10: 000000000001a026 R11: 0000000000000000 R12: 0000000000111000
[5053491.543770] R13: ffff9172b3be8000 R14: 0000000000000000 R15: ffff91776eb76300
[5053491.543770] FS:  00007fc678f96340(0000) GS:ffff917997400000(0000) knlGS:0000000000000000
[5053491.543770] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[5053491.543770] CR2: 00007f4960119000 CR3: 000000080320c000 CR4: 00000000001606e0
[5053491.543770] Call Trace:
[5053491.543770]  [<ffffffffc078a895>] ceph_osdc_msg_data_add+0xa5/0xb0 [libceph]
[5053491.543770]  [<ffffffffc079066b>] send_request+0xa4b/0xce0 [libceph]
[5053491.543770]  [<ffffffffc078acf2>] ? __remove_osd_from_lru+0x42/0x90 [libceph]
[5053491.543770]  [<ffffffffc078a973>] ? insert_request+0x73/0x80 [libceph]
[5053491.543770]  [<ffffffffc0790e58>] __submit_request+0x248/0x3c0 [libceph]
[5053491.543770]  [<ffffffffc0790ffb>] submit_request+0x2b/0x30 [libceph]
[5053491.543770]  [<ffffffffc0791815>] ceph_osdc_start_request+0x25/0x40 [libceph]
[5053491.543770]  [<ffffffffc080bdea>] start_read+0x30a/0x5c0 [ceph]
[5053491.543770]  [<ffffffffc080c171>] ceph_readpages+0xd1/0x150 [ceph]
[5053491.543770]  [<ffffffff86dc385f>] __do_page_cache_readahead+0x1cf/0x260
[5053491.543770]  [<ffffffff86dc3a0f>] ondemand_readahead+0x11f/0x240
[5053491.543770]  [<ffffffff86dc3e34>] page_cache_sync_readahead+0x44/0xb0
[5053491.543770]  [<ffffffff86db8082>] generic_file_aio_read+0x2c2/0x790
[5053491.543770]  [<ffffffffc0807c38>] ceph_aio_read+0x208/0xc90 [ceph]
[5053491.543770]  [<ffffffff86de7b3e>] ? do_wp_page+0x19e/0x720
[5053491.543770]  [<ffffffff86e40743>] do_sync_read+0x93/0xe0
[5053491.543770]  [<ffffffff86e4117f>] vfs_read+0x9f/0x170
[5053491.543770]  [<ffffffff86e4203f>] SyS_read+0x7f/0xf0
[5053491.543770]  [<ffffffff87374ddb>] system_call_fastpath+0x22/0x27
[5053491.543770] Code: 4c 89 68 18 4c 89 60 20 44 89 70 28 48 8b 73 78 48 8d 53 70 48 89 c7 e8 78 31 81 c6 4c 01 63 68 5b 41 5c 41 5d 41 5e 5d c3 0f 0b <0f> 0b 0f 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 
[5053491.543770] RIP  [<ffffffffc0781a07>] ceph_msg_data_add_pages+0x67/0x70 [libceph]
[5053491.543770]  RSP <ffff91798cf4b778>