System crashed at oracleacfs code path at function "ofs_getattr".

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux 7.
  • Oracleacfs.

Issue

  • The server rebooted unexpectedly.
  • The server crashed in a third-party module [oracleacfs] code path.

Resolution

Root Cause

  • The system crashed in the code path of the third-party module oracleacfs.
  • Since panic occurred in the function of Oracle module, the exact root cause cannot be identified as oracleacfs is a proprietary module.

Diagnostic Steps

  • The kernel ring buffer shows call traces of the panic task which indicates a panic when operating in the code section of the third-party module oracleacfs.
crash> log | less
.....
[1106507.081266] BUG: unable to handle kernel NULL pointer dereference at           (null)
[1106507.081405] IP: [<ffffffffc13a8b8b>] ofs_getattr+0xfb/0xb20 [oracleacfs]
[1106507.081576] PGD 0 
[1106507.081725] Oops: 0000 [#1] SMP 
[1106507.081883] Modules linked in: nfsd oracleacfs(POE) oracleadvm(POE) oracleoks(POE) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache 8021q garp mrp stp llc bonding oraclea
sm(O) iTCO_wdt iTCO_vendor_support vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd 
pcspkr cdc_ether usbnet mii lpc_ich i2c_i801 ioatdma dca rpcrdma sunrpc ib_isert iscsi_target_mod ipmi_ssif ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib rdma_ucm ib_uverbs wmi ib_umad ocrdma(T) ipmi_si sg ipmi_devintf rdma_cm ipmi_msghandler ib_cm iw_cm ib_core binfmt_misc ip_tables ext4 mbcache jbd2 dm_round_robin sd_mod lpfc mgag200 i2c_algo_bit
[1106507.083725]  drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nvmet_fc mpt2sas nvmet crc_t10dif crct10dif_generic nvme_fc nvme_fabrics crct10dif_pclmul crc32c_intel nvme_core raid_class be2net scsi_transport_fc scsi_transport_sas dm_multipath drm_panel_orientation_quirks scsi_tgt crct10dif_common dm_mirror dm_region_hash dm_log dm_mod
[1106507.085530] CPU: 7 PID: 4963 Comm: nfsd Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
[1106507.086599] Hardware name: IBM Flex System x240 Compute Node -[873715X]-/00AE643, BIOS -[B2E172BUS-2.70]- 06/22/2020
[1106507.087197] task: ffff943270388000 ti: ffff943718310000 task.ti: ffff943718310000
[1106507.087812] RIP: 0010:[<ffffffffc13a8b8b>]  [<ffffffffc13a8b8b>] ofs_getattr+0xfb/0xb20 [oracleacfs]
[1106507.088480] RSP: 0018:ffff943718313a70  EFLAGS: 00010246
[1106507.089147] RAX: 0000000000000000 RBX: ffff943718313ba0 RCX: 0000000000000000
[1106507.089823] RDX: ffff942b1b744ab0 RSI: 0000000000000000 RDI: ffff9439d8d80bc8
[1106507.090552] RBP: ffff943718313ae0 R08: 0000000000000644 R09: ffff9432965c5a78
[1106507.091276] R10: ffff9439d8d80bb8 R11: ffff94323f448560 R12: ffff942b1b744fd8
[1106507.091992] R13: ffff94394df6db30 R14: ffff9432965c5a40 R15: 0000000000000000
[1106507.092724] FS:  0000000000000000(0000) GS:ffff944fbf1c0000(0000) knlGS:0000000000000000
[1106507.093477] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1106507.094235] CR2: 0000000000000000 CR3: 0000000b72610000 CR4: 00000000001607e0
[1106507.095015] Call Trace:
[1106507.095804]  [<ffffffffba4617a9>] vfs_getattr+0x49/0x80
[1106507.096615]  [<ffffffffc0cf4a2c>] nfsd4_encode_fattr+0x18c/0x1a50 [nfsd]
[1106507.097474]  [<ffffffffc0ada827>] ? KsTSDLookup+0x2a7/0x3b0 [oracleoks]
[1106507.098319]  [<ffffffffc0ad7b14>] ? KsLHInfoReleCb+0x174/0x240 [oracleoks]
[1106507.099188]  [<ffffffffc0adafa9>] ? KsReleaseRwLockRecursive_OSD+0x129/0x1f0 [oracleoks]
[1106507.100057]  [<ffffffffba435b05>] ? kmem_cache_alloc+0x35/0x1f0
[1106507.100949]  [<ffffffffba2d2686>] ? prepare_creds+0x26/0x1c0
[1106507.101838]  [<ffffffffba2d3a87>] ? set_groups+0x37/0x50
[1106507.102749]  [<ffffffffc0ce16bf>] ? nfsd_setuser+0x12f/0x2f0 [nfsd]
[1106507.103663]  [<ffffffffc0cd9a92>] ? nfsd_setuser_and_check_port+0x62/0xb0 [nfsd]
[1106507.104608]  [<ffffffffba51ada2>] ? security_inode_permission+0x22/0x30
[1106507.105546]  [<ffffffffba467c42>] ? __inode_permission+0x52/0xe0
[1106507.106513]  [<ffffffffc058afab>] ? write_bytes_to_xdr_buf+0xcb/0xe0 [sunrpc]
[1106507.107485]  [<ffffffffc0cda4eb>] ? fh_verify+0x16b/0x600 [nfsd]
[1106507.108470]  [<ffffffffc0cf6339>] nfsd4_encode_getattr+0x49/0x50 [nfsd]
[1106507.109517]  [<ffffffffc0cf68f1>] nfsd4_encode_operation+0x81/0x1b0 [nfsd]
[1106507.109522]  [<ffffffffc0cebd90>] nfsd4_proc_compound+0x240/0x780 [nfsd]
[1106507.109525]  [<ffffffffc0cd68c0>] nfsd_dispatch+0xe0/0x290 [nfsd]
[1106507.109537]  [<ffffffffc0583250>] svc_process_common+0x410/0x7e0 [sunrpc]
[1106507.109545]  [<ffffffffc058371b>] svc_process+0xfb/0x110 [sunrpc]
[1106507.109548]  [<ffffffffc0cd61ff>] nfsd+0xdf/0x150 [nfsd]
[1106507.109552]  [<ffffffffc0cd6120>] ? nfsd_destroy+0x80/0x80 [nfsd]
[1106507.109554]  [<ffffffffba2cb621>] kthread+0xd1/0xe0
[1106507.109556]  [<ffffffffba2cb550>] ? insert_kthread_work+0x40/0x40
[1106507.109559]  [<ffffffffba9c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[1106507.109560]  [<ffffffffba2cb550>] ? insert_kthread_work+0x40/0x40
[1106507.109577] Code: 94 c7 48 8b 55 c0 48 8b 42 10 f6 40 38 10 74 14 48 b8 1f f2 35 00 12 06 10 c4 48 85 42 18 0f 84 ec 02 00 00 48 8b 82 48 03 00 00 <48> 8b 00 48 85 c0 0f 84 89 08 00 00 0f be 58 60 8d 4b 01 80 f9 
[1106507.109601] RIP  [<ffffffffc13a8b8b>] ofs_getattr+0xfb/0xb20 [oracleacfs]
[1106507.109601]  RSP <ffff943718313a70>
[1106507.109602] CR2: 0000000000000000
  • Backtrace of the panic task:
crash> bt
PID: 4963     TASK: ffff943270388000  CPU: 7    COMMAND: "nfsd"
 #0 [ffff943718313700] machine_kexec at ffffffffba269514
 #1 [ffff943718313760] __crash_kexec at ffffffffba329e82
 #2 [ffff943718313830] crash_kexec at ffffffffba329f78
 #3 [ffff943718313848] oops_end at ffffffffba9bc818
 #4 [ffff943718313870] no_context at ffffffffba27974c
 #5 [ffff9437183138c0] __bad_area_nosemaphore at ffffffffba279a2a
 #6 [ffff943718313910] bad_area_nosemaphore at ffffffffba279b54
 #7 [ffff943718313920] __do_page_fault at ffffffffba9bf8d0
 #8 [ffff943718313990] do_page_fault at ffffffffba9bfb05
 #9 [ffff9437183139c0] page_fault at ffffffffba9bb7b8
    [exception RIP: ofs_getattr+251]
    RIP: ffffffffc13a8b8b  RSP: ffff943718313a70  RFLAGS: 00010246
    RAX: 0000000000000000  RBX: ffff943718313ba0  RCX: 0000000000000000
    RDX: ffff942b1b744ab0  RSI: 0000000000000000  RDI: ffff9439d8d80bc8
    RBP: ffff943718313ae0   R8: 0000000000000644   R9: ffff9432965c5a78
    R10: ffff9439d8d80bb8  R11: ffff94323f448560  R12: ffff942b1b744fd8
    R13: ffff94394df6db30  R14: ffff9432965c5a40  R15: 0000000000000000
    ORIG_RAX: ffffffffffffffff  CS: 0010  SS: 0018
#10 [ffff943718313ae8] vfs_getattr at ffffffffba4617a9
#11 [ffff943718313b18] nfsd4_encode_fattr at ffffffffc0cf4a2c [nfsd]
#12 [ffff943718313d40] nfsd4_encode_getattr at ffffffffc0cf6339 [nfsd]
#13 [ffff943718313d58] nfsd4_encode_operation at ffffffffc0cf68f1 [nfsd]
#14 [ffff943718313d90] nfsd4_proc_compound at ffffffffc0cebd90 [nfsd]
#15 [ffff943718313de0] nfsd_dispatch at ffffffffc0cd68c0 [nfsd]
#16 [ffff943718313e18] svc_process_common at ffffffffc0583250 [sunrpc]
#17 [ffff943718313e80] svc_process at ffffffffc058371b [sunrpc]
#18 [ffff943718313ea0] nfsd at ffffffffc0cd61ff [nfsd]
#19 [ffff943718313ec8] kthread at ffffffffba2cb621
crash> 
  • Function ofs_getattr is from the oracleacfs module.
crash> sym ofs_getattr
ffffffffc13a8a90 (t) ofs_getattr [oracleacfs] 
                        ^                ^
                        |                |
                  [Function Name]   [Module Name]
  • oracleacfs is a third-party module.
crash> mod -t
NAME        TAINTS
ocrdma      T
oracleasm   O
oracleoks   POE
oracleadvm  POE
oracleacfs  POE    <<-----
crash> 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments