Why did gfs2_grow cause the system to hang on RHEL 5 and RHEL 6?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux Server 5 (with the High Availability and Resilient Storage Add Ons)
  • Red Hat Enterprise Linux Server 6 (with the High Availability and Resilient Storage Add Ons)
  • Global Filesystem 2(GFS2)

Issue

When running the gfs2_grow command the system hangs and the command prompt does not return. This can also happen if invalid or incorrect parameters are passed to the gfs2_grow command. The errors in the log will look similar to this.

INFO: task gfs2_grow:9101 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
gfs2_grow     D ffff81084c122420     0  9101  26340                     (NOTLB)
 ffff810ad12abac8 0000000000000086 ffff81102deae080 ffff81084c1238a0
 0000000000000096 0000000000000007 ffff810ae79d07e0 ffff81011cb337e0
 00075800f3e4c9d3 0000000000000f3f ffff810ae79d09c8 00000001f3e3975f
Call Trace:
 [<ffffffff886ea083>] :gfs2:just_schedule+0x0/0xd
 [<ffffffff886ea08c>] :gfs2:just_schedule+0x9/0xd
 [<ffffffff80063a0a>] __wait_on_bit+0x40/0x6e
 [<ffffffff886ea083>] :gfs2:just_schedule+0x0/0xd
 [<ffffffff80063aa4>] out_of_line_wait_on_bit+0x6c/0x78
 [<ffffffff800a34d5>] wake_bit_function+0x0/0x23
 [<ffffffff886e9f92>] :gfs2:gfs2_glock_schedule_for_reclaim+0x20/0x99
 [<ffffffff886ea042>] :gfs2:gfs2_glock_wait+0x37/0x78
 [<ffffffff886f3633>] :gfs2:gfs2_write_begin+0x69/0x370
 [<ffffffff886f4ef6>] :gfs2:gfs2_file_buffered_write+0xf3/0x26c
 [<ffffffff8820dc91>] :dm_mod:dm_table_unplug_all+0x38/0x83
 [<ffffffff886f531c>] :gfs2:__gfs2_file_aio_write_nolock+0x2ad/0x2e4
 [<ffffffff800a54e6>] ktime_get_ts+0x1a/0x4e
 [<ffffffff800c1025>] delayacct_end+0x5d/0x86
 [<ffffffff886f54c0>] :gfs2:gfs2_file_write_nolock+0xaa/0x10f
 [<ffffffff8002d079>] wake_up_bit+0x11/0x22
 [<ffffffff886eb2f7>] :gfs2:run_queue+0xca/0x1bb
 [<ffffffff800a34a7>] autoremove_wake_function+0x0/0x2e
 [<ffffffff886e9ee4>] :gfs2:do_promote+0x107/0x195
 [<ffffffff886eb2f7>] :gfs2:run_queue+0xca/0x1bb
 [<ffffffff886e9f92>] :gfs2:gfs2_glock_schedule_for_reclaim+0x20/0x99
 [<ffffffff886f5610>] :gfs2:gfs2_file_write+0x49/0xa7
 [<ffffffff80016b29>] vfs_write+0xce/0x174
 [<ffffffff800173f2>] sys_write+0x45/0x6e
 [<ffffffff8005d28d>] tracesys+0xd5/0xe0

Resolution

WARNING: Do not kill the gfs2_grow process or reboot the node running the gfs2_grow command. Interrupting the gfs2_grow command will corrupt the filesystem and metadata if the gfs2_grow is writing the changes to the device. This action will render the data non-recoverable.

In order to resolve this issue and release the lock you have to unmount the GFS2 file system from the OTHER nodes in the cluster. If umount fails on the OTHER nodes, you will have to edit the /etc/fstab and comment out the GFS2 filesystem then reboot. If reboot fails on the OTHER nodes, you will have to manually fence the OTHER node(s) then reboot

Root Cause

Another system has a GFS2 lock on the filesystem and will not allow the gfs2_grow command to finish and exit. Please note that the changes are written to disk in the last step as noted in the man page for `gfs2_grow.

 # man gfs2_grow
      The gfs2_grow tool tries to prevent you from corrupting your filesystem by checking as many of the likely problems
      as it can.  When expanding  a filesystem, only the last step of updating the resource index affects the currently
      mounted filesystem and so failure part way through the expansion process should leave your filesystem in its
      original unexpanded state.

Diagnostic Steps

If the gfs2_grow hangs on repeated attempts to grow the GFS2 filesystem then capturing the glocks and DLM dump may aid in narrowing down the issue. Use the script gfs2_lockcapture to capture the debug data on all the cluster nodes for only that particular GFS2 filesystem that is being expanded by gfs2_grow.

For example:
Make the script executable and review the options:

# chmod 700 gfs2_lockcapture
# ./gfs2_lockcapture --help

Get a listing of all your GFS2 filesystems

# ./gfs2_lockcapture -i
List of all the mounted GFS2 filesystems that can have their lockdump data captured:
cs5cluster:cs-rh5-1-clust.rhexample.com(id:1)
    cs5cluster:gfs2_01124894 --> /dev/mapper/shared_vg-01124894_lv on /mnt/1124894 type gfs2 (rw,hostdata=jid=0:id=327681:first=1)

Then when the gfs2_grow appears to hang after you have let it run for some period of time, then run a command similar to the following on ALL your cluster nodes. I would enable the debug option like I do so that you will know what information it is grabbing. You do not have to grab the process information if you do not want to cause in some instances can cause problems and most likely not needed for this issue. The command wil grab the information every 60 seconds 3 times for 1 particular GFS2 filesystem.

# ./gfs2_lockcapture -r 3 -s 60 -n gfs2_01124894 -d

NOTE: I would test this procedure before the issue occurs. The keyboard shortcut control-c will abruptly exit the application and any files created will be in /tmp directory. To verify that it is capturing the data you can always do a du -sh on the dlm or gfs2 directory for that particular run.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments