Chapter 13. Monitoring your Red Hat Storage Workload

Monitoring volumes helps in capacity planning and performance tuning tasks of the Red Hat Storage volume. You can monitor the Red Hat Storage volumes on different parameters and use those system outputs to identify and troubleshoot issues.
You can use Volume Top and Profile commands to view the performance and identify bottlenecks of each brick of a volume. This helps system administrators to get vital performance information whenever performance needs to be probed.
You can also perform statedump of the brick processes and NFS server process of a volume, and also view volume status and volume information.

Note

If you restart the server process, the existing Profile and Top information will be reset.

13.1. Running Volume Profile Command

Red Hat Storage Volume Profile command provides an interface to get the per-brick or NFS server I/O information for each File Operation (FOP) of a volume. This information helps in identifying bottlenecks in the storage system.
This section describes how to run Red Hat Storage Volume Profile command by performing the following operations:

13.1.1. Start Profiling

You must start the profiling to view the File Operation information for each brick.
To start profiling:
  • Start profiling using the following command:
# gluster volume profile VOLNAME start
For example, to start profiling on test-volume:
# gluster volume profile test-volume start
Profiling started on test-volume
When profiling on the volume is started, the following additional options are displayed in the Volume Info:
diagnostics.count-fop-hits: on

diagnostics.latency-measurement: on

13.1.2. Displaying the I/O Information

You can view the I/O information of each brick.
To display I/O information:
  • Display the I/O information of the bricks of a volume using the following command:
    # gluster volume profile VOLNAME info
    For example, to see the I/O information on test-volume:
    # gluster volume profile test-volume info
    Brick: Test:/export/2
    Cumulative Stats:
    
    Block                     1b+           32b+           64b+
    Size:
           Read:                0              0              0
           Write:             908             28              8
    
    Block                   128b+           256b+         512b+
    Size:
           Read:                0               6             4
           Write:               5              23            16
    
    Block                  1024b+          2048b+        4096b+
    Size:
           Read:                 0              52           17
           Write:               15             120          846
    
    Block                   8192b+         16384b+      32768b+
    Size:
           Read:                52               8           34
           Write:              234             134          286
    
    Block                                  65536b+     131072b+
    Size:
           Read:                               118          622
           Write:                             1341          594
    
    
    %-latency  Avg-      Min-       Max-       calls     Fop
              latency   Latency    Latency  
    ___________________________________________________________
    4.82      1132.28   21.00      800970.00   4575    WRITE
    5.70       156.47    9.00      665085.00   39163   READDIRP
    11.35      315.02    9.00     1433947.00   38698   LOOKUP
    11.88     1729.34   21.00     2569638.00    7382   FXATTROP
    47.35   104235.02 2485.00     7789367.00     488   FSYNC
    
    ------------------
    
    ------------------
    
    Duration     : 335
    
    BytesRead    : 94505058
    
    BytesWritten : 195571980
  • Display the I/O information of a NFS server of the specified volume using the following command:
    # gluster volume profile VOLNAME info nfs
    For example, to see the I/O information of NFS server:
    # gluster volume profile test-volume info nfs
    NFS Server : localhost
    ----------------------
    Cumulative Stats:
       Block Size:              32768b+               65536b+ 
     No. of Reads:                    0                     0 
    No. of Writes:                 1000                  1000 
     %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
     ---------   -----------   -----------   -----------   ------------        ----
          0.01     410.33 us     194.00 us     641.00 us              3      STATFS
          0.60     465.44 us     346.00 us     867.00 us            147       FSTAT
          1.63     187.21 us      67.00 us    6081.00 us           1000     SETATTR
          1.94     221.40 us      58.00 us   55399.00 us           1002      ACCESS
          2.55     301.39 us      52.00 us   75922.00 us            968        STAT
          2.85     326.18 us      88.00 us   66184.00 us           1000    TRUNCATE
          4.47     511.89 us      60.00 us  101282.00 us           1000       FLUSH
          5.02    3907.40 us    1723.00 us   19508.00 us            147    READDIRP
         25.42    2876.37 us     101.00 us  843209.00 us           1012      LOOKUP
         55.52    3179.16 us     124.00 us  121158.00 us           2000       WRITE
     
        Duration: 7074 seconds
       Data Read: 0 bytes
    Data Written: 102400000 bytes
     
    Interval 1 Stats:
       Block Size:              32768b+               65536b+ 
     No. of Reads:                    0                     0 
    No. of Writes:                 1000                  1000 
     %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
     ---------   -----------   -----------   -----------   ------------        ----
          0.01     410.33 us     194.00 us     641.00 us              3      STATFS
          0.60     465.44 us     346.00 us     867.00 us            147       FSTAT
          1.63     187.21 us      67.00 us    6081.00 us           1000     SETATTR
          1.94     221.40 us      58.00 us   55399.00 us           1002      ACCESS
          2.55     301.39 us      52.00 us   75922.00 us            968        STAT
          2.85     326.18 us      88.00 us   66184.00 us           1000    TRUNCATE
          4.47     511.89 us      60.00 us  101282.00 us           1000       FLUSH
          5.02    3907.40 us    1723.00 us   19508.00 us            147    READDIRP
         25.41    2878.07 us     101.00 us  843209.00 us           1011      LOOKUP
         55.53    3179.16 us     124.00 us  121158.00 us           2000       WRITE
     
        Duration: 330 seconds
       Data Read: 0 bytes
    Data Written: 102400000 bytes

13.1.3. Stop Profiling

You can stop profiling the volume, if you do not need profiling information anymore.
To stop profiling
  • Stop profiling using the following command:
    # gluster volume profile VOLNAME stop
    For example, to stop profiling on test-volume:
    # gluster volume profile test-volume stop
    Profiling stopped on test-volume