2.9.3. Troubleshooting GFS2 Performance with the GFS2 Lock Dump
debugfsfile which can be found at the following path name, assuming that
debugfsis mounted on
debugfsfile is to use the
catcommand to take a copy of the complete content of the file (it might take a long time if you have a large amount of RAM and a lot of cached inodes) while the application is experiencing problems, and then looking through the resulting data at a later date.
debugfsfile, one a few seconds or even a minute or two after the other. By comparing the holder information in the two traces relating to the same glock number, you can tell whether the workload is making progress (that is, it is just slow) or whether it has become stuck (which is always a bug and should be reported to Red Hat support immediately).
debugfsfile starting with H: (holders) represent lock requests either granted or waiting to be granted. The flags field on the holders line f: shows which: The 'W' flag refers to a waiting request, the 'H' flag refers to a granted request. The glocks which have large numbers of waiting requests are likely to be those which are experiencing particular contention.
Table 2.1. Glock flags
|b||Blocking||Valid when the locked flag is set, and indicates that the operation that has been requested from the DLM may block. This flag is cleared for demotion operations and for "try" locks. The purpose of this flag is to allow gathering of stats of the DLM response time independent from the time taken by other nodes to demote locks.|
|d||Pending demote||A deferred (remote) demote request|
|D||Demote||A demote request (local or remote)|
|f||Log flush||The log needs to be committed before releasing this glock|
|F||Frozen||Replies from remote nodes ignored - recovery is in progress. This flag is not related to file system freeze, which uses a different mechanism, but is used only in recovery.|
|i||Invalidate in progress||In the process of invalidating pages under this glock|
|I||Initial||Set when DLM lock is associated with this glock|
|l||Locked||The glock is in the process of changing state|
|L||LRU||Set when the glock is on the LRU list|
|o||Object||Set when the glock is associated with an object (that is, an inode for type 2 glocks, and a resource group for type 3 glocks)|
|p||Demote in progress||The glock is in the process of responding to a demote request|
|q||Queued||Set when a holder is queued to a glock, and cleared when the glock is held, but there are no remaining holders. Used as part of the algorithm the calculates the minimum hold time for a glock.|
|r||Reply pending||Reply received from remote node is awaiting processing|
|y||Dirty||Data needs flushing to disk before releasing this glock|
Table 2.2. Glock holder flags
|a||Async||Do not wait for glock result (will poll for result later)|
|A||Any||Any compatible lock mode is acceptable|
|c||No cache||When unlocked, demote DLM lock immediately|
|e||No expire||Ignore subsequent lock cancel requests|
|E||exact||Must have exact lock mode|
|F||First||Set when holder is the first to be granted for this lock|
|H||Holder||Indicates that requested lock is granted|
|p||Priority||Enqueue holder at the head of the queue|
|t||Try||A "try" lock|
|T||Try 1CB||A "try" lock that sends a callback|
|W||Wait||Set while waiting for request to complete|
find -inum numberwhere number is the inode number converted from the hex format in the glocks file into decimal.
findon a file system when it is experiencing lock contention, you are likely to make the problem worse. It is a good idea to stop the application before running the
findwhen you are looking for contended inodes.
Table 2.3. Glock types
|Type number||Lock type||Use|
|2||Inode||Inode metadata and data|
|3||Rgrp||Resource group metadata|
|5||Iopen||Inode last closer detection|
gfs2_growcommand to expand the file system.