RHEL5: Kernel crash in shrink_dcache_for_unmount while unmounting 3rd party file system

Solution Verified - Updated -

Issue

  • The first kernel errata for RHEL 5.8 (2.6.18-308.1.1) seems to have introduced an issue where a third party file system (PVFS / OrangeFS) can no longer be unmounted without triggering a kernel panic.
  • The initial 5.8 release (2.6.18-308) and previous RHEL releases did not produce this issue, but each errata release in the 5.8 sequence - through 2.6.18-308.24.1 - does produce a panic.
  • The panic can be triggered by mounting a PVFS file system, executing a dd that moves data onto the file system (50MB from /dev/urandom in my case), and then unmounting the file system.
  • Kernel crashes with a message similar to the following, which shows that the kernel is tainted, and a 3rd party fileystem is being used
pvfs2: pvfs2_init called with debug mask: "none" (0x00000000)
pvfs2: module version 2.8.6-orangefs-2012-12-17-194734 loaded
PVFS: kernel debug mask has been modified to "none" (0x00000000)
PVFS: client debug mask has been modified to "none" (0x00000000)
pvfs2: module version 2.8.2 loaded
BUG: Dentry ffff8100205b7c48{i=100002,n=batayl} still in use (-1) [unmount of pvfs28 pvfs28]
----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at fs/dcache.c:612
invalid opcode: 0000 [1] SMP 
last sysfs file: /devices/pci0000:00/0000:00:00.0/irq
CPU 1 
Modules linked in: nfs nfs_acl pvfs28(U) orangefs(U) autofs4 hidp l2cap bluetooth lockd sunrpc be2iscsi ib_iser rdma_cm ib_cm iw_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp bnx2i cnic ipv6 xfrm_nalgo crypto_api uio cxgb3i libcxgbi cxgb3 8021q libiscsi_tcp libiscsi2 scsi_transport_iscsi2 scsi_transport_iscsi vsock(U) vmmemctl(U) acpiphp dm_mirror dm_multipath scsi_dh video backlight sbs power_meter hwmon i2c_ec dell_wmi wmi button battery asus_acpi acpi_memhotplug ac mptctl lp vmci(U) sg shpchp floppy ide_cd i2c_piix4 i2c_core cdrom parport_pc parport tpm_tis tpm tpm_bios pcspkr serio_raw vmxnet(U) pvscsi(U) vmxnet3(U) dm_raid45 dm_message dm_region_hash dm_log dm_mod dm_mem_cache ata_piix libata mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
Pid: 4582, comm: umount Tainted: G     ---- 2.6.18-308.24.1.el5 #1
RIP: 0010:[<ffffffff800effd3>]  [<ffffffff800effd3>] shrink_dcache_for_umount_subtree+0x177/0x222
RSP: 0018:ffff810020607e08  EFLAGS: 00010286
RAX: 0000000000000060 RBX: ffff8100205b7c48 RCX: ffffffff80324028
RDX: ffffffff80324028 RSI: 0000000000000000 RDI: ffffffff80324020
RBP: ffff810020808238 R08: ffffffff80324028 R09: 000000000000003f
R10: ffff810020607aa8 R11: 0000000000000080 R12: ffff8100205b7ca8
R13: 0000000000000000 R14: 0000000000000000 R15: 00000000072f9580
FS:  00002b0211862780(0000) GS:ffff81003fe1c7c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000003d860e7940 CR3: 000000002061a000 CR4: 00000000000006a0
Process umount (pid: 4582, threadinfo ffff810020606000, task ffff81003c8be040)
Stack:  ffff81003ee04220 0000000000000206 ffff81003ee04000 ffffffff88716280
 ffff81003ee040d0 ffffffff800f04ef ffff81003ee04000 ffffffff800e7850
 000000000000001c ffff81003ee04000 ffff81003ee040d0 ffffffff800e797e
Call Trace:
 [<ffffffff800f04ef>] shrink_dcache_for_umount+0x37/0x45
 [<ffffffff800e7850>] generic_shutdown_super+0x1b/0xfb
 [<ffffffff800e797e>] kill_anon_super+0x9/0x35
 [<ffffffff88704476>] :pvfs28:pvfs2_kill_sb+0xe9/0x173
 [<ffffffff800e7a2f>] deactivate_super+0x6a/0x82
 [<ffffffff800f28f7>] sys_umount+0x245/0x27b
 [<ffffffff800237fa>] sys_newstat+0x19/0x31
 [<ffffffff8009b025>] recalc_sigpending+0xe/0x25
 [<ffffffff8001de35>] sigprocmask+0xb7/0xdb
 [<ffffffff80030239>] sys_rt_sigprocmask+0x50/0xd9
 [<ffffffff8005d116>] system_call+0x7e/0x83


Code: 0f 0b 68 27 2a 2c 80 c2 64 02 5f 41 58 eb fe 48 8b 6b 28 48 
RIP  [<ffffffff800effd3>] shrink_dcache_for_umount_subtree+0x177/0x222
 RSP <ffff810020607e08>

Environment

  • Red Hat Enterprise Linux 5
    • Kernels 2.6.18-308.1.1 or above
  • 3rd party filesystem such as PVFS, VxFS, or Orangefs

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content