RHEL7: nfs4 client kernel panic in nfs4_callback_recall code path due to invalid nfs4_delegation and nfs_client structures when a callback identifier of 0 is used

Solution Unverified - Updated -

Issue

  • We are experiencing problems when working with the same file from two different sources. We have desktops based on RHEL7 and servers based on RHEL7, all are NFS clients, and NetApp NFS server
  • When accessing a random file that exists on a NFS share from a server and and a desktop at the same time we get a kernel panic on the desktop.
  • This does not occur on all desktops but on some of them we can replicate the crash everytime.
  • This is done by working with the file(opening, writing, saving it) from both sources at the same time.
  • nfsv4.0-svc kernel panic with the following oops
[13063.918409] general protection fault: 0000 [#1] SMP 
[13063.918477] Modules linked in: arc4 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bnep bluetooth fuse xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables sg coretemp kvm snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm mei_me mei hp_wmi iTCO_wdt sparse_keymap iTCO_vendor_support rfkill serio_raw snd_page_alloc snd_timer snd soundcore e1000e ptp pps_core aesni_intel lrw gf128mul glue_helper ablk_helper cryptd tpm_infineon pcspkr lpc_ich mfd_core shpchp wmi acpi_cpufreq mperf nfsd auth_rpcgss nfs_acl lockd uinput sunrpc xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_common i915 ahci libahci
[13063.919285]  i2c_algo_bit drm_kms_helper libata drm i2c_core video
[13063.919349] CPU: 3 PID: 6915 Comm: nfsv4.0-svc Not tainted 3.10.0-123.13.2.el7.x86_64 #1
[13063.919415] Hardware name: Hewlett-Packard HP Compaq 8100 Elite SFF PC/304Ah, BIOS 786H1 v01.02 12/16/2009
[13063.919492] task: ffff8800bb1816c0 ti: ffff880058a7e000 task.ti: ffff880058a7e000
[13063.919552] RIP: 0010:[<ffffffff815e99ce>]  [<ffffffff815e99ce>] _raw_spin_lock+0xe/0x50
[13063.919627] RSP: 0018:ffff880058a7fc70  EFLAGS: 00010207
[13063.919672] RAX: 0000000000020000 RBX: 001fffff00000050 RCX: ffffea00027661f8
[13063.919729] RDX: ffff880058a7fd50 RSI: ffff8800d713c008 RDI: 001fffff00000050
[13063.919785] RBP: ffff880058a7fc70 R08: ffff880096433000 R09: ffff8800d713c026
[13063.919841] R10: 0000000000000001 R11: ffffea000162a480 R12: ffffea0002766740
[13063.919897] R13: ffff8800d713c008 R14: 001fffff00000000 R15: 0000000000000004
[13063.919954] FS:  0000000000000000(0000) GS:ffff880117cc0000(0000) knlGS:0000000000000000
[13063.920019] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[13063.920065] CR2: 000000000272c000 CR3: 00000000018d0000 CR4: 00000000000007e0
[13063.920122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[13063.920178] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[13063.920234] Stack:
[13063.920253]  ffff880058a7fcd8 ffffffffa06d5f3c ffff8800aa789cc0 ffffea00027661f8
[13063.920324]  ffff8800d713c00a ffff880058a7fd70 001fffff00000000 000000004a84a286
[13063.920394]  ffff8800d713c000 0000000000000000 0000000011270000 0000000000000000
[13063.920466] Call Trace:
[13063.920514]  [<ffffffffa06d5f3c>] nfs_delegation_find_inode+0xac/0x150 [nfsv4]
[13063.920590]  [<ffffffffa06d95e2>] nfs4_callback_recall+0x42/0x160 [nfsv4]
[13063.920661]  [<ffffffffa06d85b5>] nfs4_callback_compound+0x485/0x6d0 [nfsv4]
[13063.920745]  [<ffffffffa02ee44a>] ? svcauth_unix_accept+0x1ca/0x290 [sunrpc]
[13063.920823]  [<ffffffffa02e9b09>] svc_process_common+0x5f9/0x6f0 [sunrpc]
[13063.920895]  [<ffffffffa06d7380>] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
[13063.920971]  [<ffffffffa02e9d07>] svc_process+0x107/0x170 [sunrpc]
[13063.921037]  [<ffffffffa06d7380>] ? nfs_callback_authenticate+0x50/0x50 [nfsv4]
[13063.921112]  [<ffffffffa06d73c5>] nfs4_callback_svc+0x45/0x60 [nfsv4]
[13063.921169]  [<ffffffff81085aff>] kthread+0xcf/0xe0
[13063.921212]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[13063.921266]  [<ffffffff815f296c>] ret_from_fork+0x7c/0xb0
[13063.921314]  [<ffffffff81085a30>] ? kthread_create_on_node+0x140/0x140
[13063.921367] Code: 66 83 07 02 f6 47 02 01 74 e5 0f 1f 00 e8 3a 2f ff ff eb db 66 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 b8 00 00 02 00 <f0> 0f c1 07 89 c2 c1 ea 10 66 39 c2 75 02 5d c3 83 e2 fe 0f b7 
[13063.921744] RIP  [<ffffffff815e99ce>] _raw_spin_lock+0xe/0x50
[13063.921794]  RSP <ffff880058a7fc70>

Environment

  • NFS server : NetApp FAS3250 Ontap 8.1.4P6 7-Mode
  • NFS Client : Red Hat Enterprise Linux 7
    • seen on 3.10.0-123.13.2.el7
    • seen on 3.10.0-229.el7
    • seen on 3.10.0-327.el7
  • nfs4

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.