RHEL6.6: pNFS client kernel crash at filelayout_read_pagelist because ds->ds_clp == NULL
Issue
- NFS4.1 / pNFS client kernel crash at filelayout_read_pagelist called from nfs_pageio_complete while reconnecting a DS
- kernel crash with the following messages. Just prior to the crash we see a message indicating a connection error and Deviceid marked out of use:
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: filelayout_write_pagelist: prepare_ds failed, use MDS
NFS: data server ac11500e connection error -110. Deviceid [101018000045404] marked out of use.
BUG: unable to handle kernel NULL pointer dereference at 00000000000000c8
IP: [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
PGD 870171067 PUD 871b4a067 PMD 0
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/system/cpu/online
CPU 3
Modules linked in: bridge stp llc coretemp nfs_layout_nfsv41_files nfs lockd fscache auth_rpcgss nfs_acl sunrpc cpufreq_ondemand acpi_cpufreq freq_table mperf ipv6 microcode iTCO_wdt iTCO_vendor_support enic power_meter acpi_ipmi ipmi_si ipmi_msghandler sb_edac edac_core lpc_ich mfd_core sg shpchp ext4 jbd2 mbcache sd_mod crc_t10dif wmi megaraid_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Pid: 1573, comm: triplet-specifi Not tainted 2.6.32-504.el6.x86_64 #1 Cisco Systems Inc UCSB-B200-M3/UCSB-B200-M3
RIP: 0010:[<ffffffffa0316aa6>] [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
RSP: 0018:ffff8808703839d8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880873aa35d8 RCX: 0000000000000000
RDX: ffffffffa0319f80 RSI: ffff880873aa35d8 RDI: ffff8805f9bb5440
RBP: ffff880870383a18 R08: ffff880870382000 R09: 00000000ffffffff
R10: 0003ae8bfee5a774 R11: 0000000000000001 R12: ffff8805f9bb5440
R13: 0000000000000000 R14: ffff880873249600 R15: 0000000000000000
FS: 00007f3d2e720700(0000) GS:ffff8800282c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000000000c8 CR3: 0000000870020000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process triplet-specifi (pid: 1573, threadinfo ffff880870382000, task ffff880871ba9500)
Stack:
ffff8808703839f8 ffff880800000001 ffff88087433ba00 ffff880873aa35e0
<d> ffff880873aa35d8 ffff88087433b8e0 ffff8805f9b01000 ffff880873aa3560
<d> ffff880870383a88 ffffffffa02d4995 0000000000000000 ffff8805f9bb5440
Call Trace:
[<ffffffffa02d4995>] pnfs_generic_pg_readpages+0x105/0x290 [nfs]
[<ffffffffa02a69c2>] nfs_pageio_complete+0x22/0x80 [nfs]
[<ffffffffa02a8993>] nfs_readpages+0x163/0x210 [nfs]
[<ffffffff81139e85>] __do_page_cache_readahead+0x185/0x210
[<ffffffff81139f31>] ra_submit+0x21/0x30
[<ffffffff8113a2bd>] ondemand_readahead+0x11d/0x250
[<ffffffff8113a4e3>] page_cache_sync_readahead+0x33/0x50
[<ffffffff81126008>] generic_file_aio_read+0x558/0x700
[<ffffffffa029b47a>] nfs_file_read+0xca/0x130 [nfs]
[<ffffffff8118de9a>] do_sync_read+0xfa/0x140
[<ffffffff8109eb00>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8122d446>] ? security_file_permission+0x16/0x20
[<ffffffff8118e855>] vfs_read+0xb5/0x1a0
[<ffffffff8118e991>] sys_read+0x51/0x90
[<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Code: c0 01 00 00 4c 89 ee 4c 89 e7 e8 76 f5 ff ff 4c 89 ab 58 02 00 00 48 89 83 e8 01 00 00 48 c7 c2 80 9f 31 a0 49 8b 46 18 48 89 de <48> 8b b8 c8 00 00 00 e8 2e 27 f9 ff 85 c0 75 7f 31 d2 e9 42 ff
RIP [<ffffffffa0316aa6>] filelayout_read_pagelist+0xf6/0x190 [nfs_layout_nfsv41_files]
RSP <ffff8808703839d8>
CR2: 00000000000000c8
Environment
- Red Hat Enterprise Linux 6.6 (NFS Client)
- seen on kernel 2.6.32-504.el6
- seen with NetApp storage system (NFS Server)
- FAS8060 (ONTAP Version 8.2.2SP1)
- pNFS / NFS4.1
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.