Why Glusterfs crashes on client after updating from Red Hat Enterprise Linux 7.1 to Red Hat Enterprise Linux 7.2 ?

Solution Verified - Updated -

Issue

  • Why Glusterfs crashes on client after updating from Red Hat Enterprise Linux 7.1 to Red Hat Enterprise Linux 7.2 ?

  • The backup software stores its deduplicated data on a GlusterFS volume but after updating the backup server from RHEL 7.1 to 7.2, glusterfs crashes on client from time to time.

~snippet from dmesg:

[901920.810095] INFO: task glusterfs:1475 blocked for more than 120 seconds.
[901920.810127] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[901920.810150] glusterfs       D ffff8810521f3ad0     0  1475      1 0x00000080
[901920.810153]  ffff8810521f3970 0000000000000086 ffff8810515ce780 ffff8810521f3fd8
[901920.810156]  ffff8810521f3fd8 ffff8810521f3fd8 ffff8810515ce780 ffff88085fd14780
[901920.810158]  0000000000000000 7fffffffffffffff ffffffff81168860 ffff8810521f3ad0
[901920.810159] Call Trace:
[901920.810168]  [<ffffffff81168860>] ? wait_on_page_read+0x60/0x60
[901920.810173]  [<ffffffff8163a889>] schedule+0x29/0x70
[901920.810174]  [<ffffffff81638579>] schedule_timeout+0x209/0x2d0
[901920.810176]  [<ffffffff81168860>] ? wait_on_page_read+0x60/0x60
[901920.810177]  [<ffffffff81639ebe>] io_schedule_timeout+0xae/0x130
[901920.810179]  [<ffffffff81639f58>] io_schedule+0x18/0x20
[901920.810180]  [<ffffffff8116886e>] sleep_on_page+0xe/0x20
[901920.810181]  [<ffffffff8163883b>] __wait_on_bit_lock+0x5b/0xc0
[901920.810183]  [<ffffffff81168988>] __lock_page+0x78/0xa0
[901920.810186]  [<ffffffff810a6b60>] ? wake_atomic_t_function+0x40/0x40
[901920.810188]  [<ffffffff81178a50>] invalidate_inode_pages2_range+0x3f0/0x430
[901920.810194]  [<ffffffffa0542bb6>] fuse_reverse_inval_inode+0x66/0x90 [fuse]
[901920.810196]  [<ffffffffa05369d8>] fuse_notify+0x158/0x7f0 [fuse]
[901920.810198]  [<ffffffffa0535816>] ? fuse_copy_do+0x46/0x90 [fuse]
[901920.810201]  [<ffffffffa0538023>] fuse_dev_do_write+0x153/0x3c0 [fuse]
[901920.810202]  [<ffffffffa0538619>] fuse_dev_write+0x79/0xa0 [fuse]
[901920.810206]  [<ffffffff811ddccd>] do_sync_write+0x8d/0xd0
[901920.810208]  [<ffffffff811de4ed>] vfs_write+0xbd/0x1e0
[901920.810209]  [<ffffffff811def8f>] SyS_write+0x7f/0xe0
[901920.810213]  [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b

Environment

  • CentOS 7.2 (Storage node)

    • glusterfs-3.7.6-1.el7.x86_64 Installed via GlusterFS (community) yum repository
  • RHEL 7.2 (Gluster client where crash happen)

    • glusterfs-3.7.1-16.el7.x86_64 Installed via RHN repository

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.