kernel BUG at fs/inode.c:1254!

Latest response

Dear all,

I have a kernel freeze on a server and I don't know why.

Can you help me?

Trace:

2016-08-24T16:00:10+00:00 server kernel[]:[warning] : ------------[ cut here ]------------
2016-08-24T16:00:10+00:00 server kernel[]:[crit] : kernel BUG at fs/inode.c:1254!
2016-08-24T16:00:10+00:00 server mogilefsd[14506]:[debug] [queryworker(14703)] node_timeout: 15 (elapsed: 15.0158669948578): HEAD http://127.0.0.1:7500/dev1/0/973/852/0973852432.fid
2016-08-24T16:00:10+00:00 server mogilefsd[14506]:[debug] [queryworker(14703)] Failed HEAD check for http://127.0.0.1:7500/dev1/0/973/852/0973852432.fid (500): node_timeout: 15 (elapsed: 15.0158669948578)
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : invalid opcode: 0000 [#1] SMP 
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : last sysfs file: /sys/devices/virtual/net/bond0/statistics/tx_bytes
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : CPU 1 
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : Modules linked in: xfs exportfs cpufreq_ondemand acpi_cpufreq freq_table mperf bonding ipv6 vfat fat ipmi_devintf ipmi_si ipmi_msghandler microcode iTCO_wdt iTCO_vendor_support ics932s401 cdc_ether usbnet mii joydev i2c_i801 lpc_ich mfd_core shpchp igb i2c_algo_bit i2c_core ptp pps_core ioatdma dca be2net ext4 jbd2 mbcache sr_mod cdrom lpfc scsi_transport_fc scsi_tgt ahci wmi megaraid_sas mppVhba(U) dm_mirror dm_region_hash dm_log dm_mod mppUpper(U) sg sd_mod crc_t10dif [last unloaded: ip_tables]
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : 
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : Pid: 14551, comm: nginx Not tainted 2.6.32-573.12.1.el6.x86_64 #1 IBM IBM System x3550 M4: -[7914M6C]-/00AM544
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : RIP: 0010:[<ffffffff811b019f>]  [<ffffffff811b019f>] generic_delete_inode+0x1bf/0x1d0
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : RSP: 0018:ffff8804917e3e08  EFLAGS: 00010283
2016-08-24T16:00:10+00:00 server kernel[]:[warning] : RAX: ffff880028033b20 RBX: ffff880151179988 RCX: 0000000000000034
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : RDX: 0000000000000007 RSI: ffff880151179ba0 RDI: ffff880028033b18
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : RBP: ffff8804917e3e28 R08: 4038000000000000 R09: fe90e8b8a5cae807
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81fe5920
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : R13: 0000000000000000 R14: 00000000ffffff9c R15: 0000000000000000
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : FS:  00007fc23cbea7e0(0000) GS:ffff880028220000(0000) knlGS:0000000000000000
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : CR2: 0000000000000001 CR3: 000000048ff4a000 CR4: 00000000000407e0
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : Process nginx (pid: 14551, threadinfo ffff8804917e0000, task ffff880876506040)
2016-08-24T16:00:11+00:00 server kernel[]:[warning] : Stack:
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : 00000000ffffff9c ffff880151179988 0000000000000000 0000000001998034
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : <d> ffff8804917e3e48 ffffffff811b0215 ffff8804917e3e58 ffff880151179988
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : <d> ffff8804917e3e68 ffffffff811af062 ffff8804917e3e68 ffff8804917e3e98
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : Call Trace:
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff811b0215>] generic_drop_inode+0x65/0x80
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff811af062>] iput+0x62/0x70
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff811a3a79>] do_unlinkat+0x1a9/0x260
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff81197946>] ? sys_newlstat+0x36/0x50
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff811a3b46>] sys_unlink+0x16/0x20
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : [<ffffffff8100b0d2>] system_call_fastpath+0x16/0x1b
2016-08-24T16:00:12+00:00 server kernel[]:[warning] : Code: ff ff e9 43 ff ff ff 0f 0b 0f 1f 00 eb fb be ce 04 00 00 48 c7 c7 35 2c 7d 81 e8 1d 73 ec ff 48 8b 83 18 02 00 00 e9 a6 fe ff ff <0f> 0b eb fe 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 
2016-08-24T16:00:12+00:00 server kernel[]:[alert] : RIP  [<ffffffff811b019f>] generic_delete_inode+0x1bf/0x1d0
2016-08-24T16:00:13+00:00 server kernel[]:[warning] : RSP <ffff8804917e3e08> 

Thanks

Responses

nginx is trying to delete a file. The error encountered at fs/inode.c:1254 is BUG_ON(inode->i_state != I_CLEAR); and I_CLEAR is a flag denoting an inode is clean and can be destroyed. This sounds like either a filesystem bug or a race condition.

We have seen this before with proprietary storage drivers writing to block zero of their block device, triggering a partition table rescan which then triggers a race condition, and it can also be reproduced with a certain ioctl to a block device containing a mounted filesystem which causes the same race:

You are using Dell MPP by the look of the drivers mppVhba(U) mppUpper(U). I would be inclined to ask Dell about this, specifically if their driver writes to block zero when it shouldn't. That would be a bug for them to address in MPP. I think we could prove such a thing with blktrace so you're welcome to open a support case if you'd like an in-depth investigation.

Or, if you're running something to rescan partition tables (like blockdev --rereadpt) of devices with mounted filesystems then don't do that, it's not correct. Unmount the filesystem first.

Hi Jamie,

Thank you very much for your help!

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.