RHEL6.6: pNFS client kernel crash at filelayout_read_pagelist because ds->ds_clp == NULL

Solution In Progress - Updated -

Issue

  • NFS4.1 / pNFS client kernel crash at filelayout_read_pagelist called from nfs_pageio_complete while reconnecting a DS
  • kernel crash with the following messages. Just prior to the crash we see a message indicating a connection error and Deviceid marked out of use:
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: data server ac11500e connection error -110. Deviceid [101018000045404] marked out of use.
BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
IP: [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
PGD 870171067 PUD 871b4a067 PMD 0 
Oops: 0000 [#1] SMP 
last sysfs file: /sys/devices/system/cpu/online
CPU 3 
Modules linked in: bridge stp llc coretemp nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ipv6 microcode iTCO_wdt iTCO_vendor_support enic power_meter acpi_ipmi ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]

Pid: 1573, comm: triplet-specifi Not tainted 2.6.32-504.el6.x86_64 #1 Cisco Systems Inc UCSB-B200-M3/UCSB-B200-M3
RIP: 0010:[<ffffffffa0316aa6>]  [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
RSP: 0018:ffff8808703839d8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880873aa35d8 RCX: 0000000000000000
RDX: ffffffffa0319f80 RSI: ffff880873aa35d8 RDI: ffff8805f9bb5440
RBP: ffff880870383a18 R08: ffff880870382000 R09: 00000000ffffffff
R10: 0003ae8bfee5a774 R11: 0000000000000001 R12: ffff8805f9bb5440
R13: 0000000000000000 R14: ffff880873249600 R15: 0000000000000000
FS:  00007f3d2e720700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000000c8 CR3: 0000000870020000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process triplet-specifi (pid: 1573, threadinfo ffff880870382000, task ffff880871ba9500)
Stack:
 ffff8808703839f8 ffff880800000001 ffff88087433ba00 ffff880873aa35e0
<d> ffff880873aa35d8 ffff88087433b8e0 ffff8805f9b01000 ffff880873aa3560
<d> ffff880870383a88 ffffffffa02d4995 0000000000000000 ffff8805f9bb5440
Call Trace:
 [<ffffffffa02d4995>] pnfs_generic_pg_readpages+0x105/0x290 [nfs]
 [<ffffffffa02a69c2>] nfs_pageio_complete+0x22/0x80 [nfs]
 [<ffffffffa02a8993>] nfs_readpages+0x163/0x210 [nfs]
 [<ffffffff81139e85>] __do_page_cache_readahead+0x185/0x210
 [<ffffffff81139f31>] ra_submit+0x21/0x30
 [<ffffffff8113a2bd>] ondemand_readahead+0x11d/0x250
 [<ffffffff8113a4e3>] page_cache_sync_readahead+0x33/0x50
 [<ffffffff81126008>] generic_file_aio_read+0x558/0x700
 [<ffffffffa029b47a>] nfs_file_read+0xca/0x130 [nfs]
 [<ffffffff8118de9a>] do_sync_read+0xfa/0x140
 [<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8122d446>] ? security_file_permission+0x16/0x20
 [<ffffffff8118e855>] vfs_read+0xb5/0x1a0
 [<ffffffff8118e991>] sys_read+0x51/0x90
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: c0 01 00 00 4c 89 ee 4c 89 e7 e8 76 f5 ff ff 4c 89 ab 58 02 00 00 48 89 83 e8 01 00 00 48 c7 c2 80 9f 31 a0 49 8b 46 18 48 89 de <48> 8b b8 c8 00 00 00 e8 2e 27 f9 ff 85 c0 75 7f 31 d2 e9 42 ff 
RIP  [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
 RSP <ffff8808703839d8>
CR2: 00000000000000c8

Environment

  • Red Hat Enterprise Linux 6.6 (NFS Client)
    • seen on kernel 2.6.32-504.el6
  • seen with NetApp storage system (NFS Server)
    • FAS8060 (ONTAP Version 8.2.2SP1)
  • pNFS / NFS4.1

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.