- 13.1. Running Volume Profile Command
- 13.2. Running Volume Top Command
- 13.2.1. Viewing Open File Descriptor Count and Maximum File Descriptor Count
- 13.2.2. Viewing Highest File Read Calls
- 13.2.3. Viewing Highest File Write Calls
- 13.2.4. Viewing Highest Open Calls on Directory
- 13.2.5. Viewing Highest Read Calls on Directory
- 13.2.6. Viewing List of Read Performance
- 13.2.7. Viewing List of Write Performance
- 13.3. Listing Volumes
- 13.4. Displaying Volume Information
- 13.5. Performing Statedump on a Volume
- 13.6. Displaying Volume Status
Monitoring volumes helps in capacity planning and performance tuning tasks of the Red Hat Storage volume. You can monitor the Red Hat Storage volumes on different parameters and use those system outputs to identify and troubleshoot issues.
You can use Volume
Top and Profile commands to view the performance and identify bottlenecks of each brick of a volume. This helps system administrators to get vital performance information whenever performance needs to be probed.
You can also perform statedump of the brick processes and NFS server process of a volume, and also view volume status and volume information.
Note
If you restart the server process, the existing
Profile and Top information will be reset.
Red Hat Storage Volume
Profile command provides an interface to get the per-brick or NFS server I/O information for each File Operation (FOP) of a volume. This information helps in identifying bottlenecks in the storage system.
This section describes how to run Red Hat Storage Volume
Profile command by performing the following operations:
You must start the profiling to view the File Operation information for each brick.
To start profiling:
- Start profiling using the following command:
# gluster volume profile VOLNAME start
For example, to start profiling on test-volume:
# gluster volume profile test-volume start Profiling started on test-volume
When profiling on the volume is started, the following additional options are displayed in the Volume Info:
diagnostics.count-fop-hits: on diagnostics.latency-measurement: on
You can view the I/O information of each brick.
To display I/O information:
- Display the I/O information of the bricks of a volume using the following command:
# gluster volume profileVOLNAMEinfoFor example, to see the I/O information on test-volume:# gluster volume profile test-volume info Brick: Test:/export/2 Cumulative Stats: Block 1b+ 32b+ 64b+ Size: Read: 0 0 0 Write: 908 28 8 Block 128b+ 256b+ 512b+ Size: Read: 0 6 4 Write: 5 23 16 Block 1024b+ 2048b+ 4096b+ Size: Read: 0 52 17 Write: 15 120 846 Block 8192b+ 16384b+ 32768b+ Size: Read: 52 8 34 Write: 234 134 286 Block 65536b+ 131072b+ Size: Read: 118 622 Write: 1341 594 %-latency Avg- Min- Max- calls Fop latency Latency Latency ___________________________________________________________ 4.82 1132.28 21.00 800970.00 4575 WRITE 5.70 156.47 9.00 665085.00 39163 READDIRP 11.35 315.02 9.00 1433947.00 38698 LOOKUP 11.88 1729.34 21.00 2569638.00 7382 FXATTROP 47.35 104235.02 2485.00 7789367.00 488 FSYNC ------------------ ------------------ Duration : 335 BytesRead : 94505058 BytesWritten : 195571980 - Display the I/O information of a NFS server of the specified volume using the following command:
# gluster volume profileVOLNAMEinfo nfsFor example, to see the I/O information of NFS server:# gluster volume profile test-volume info nfs NFS Server : localhost ---------------------- Cumulative Stats: Block Size: 32768b+ 65536b+ No. of Reads: 0 0 No. of Writes: 1000 1000 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.01 410.33 us 194.00 us 641.00 us 3 STATFS 0.60 465.44 us 346.00 us 867.00 us 147 FSTAT 1.63 187.21 us 67.00 us 6081.00 us 1000 SETATTR 1.94 221.40 us 58.00 us 55399.00 us 1002 ACCESS 2.55 301.39 us 52.00 us 75922.00 us 968 STAT 2.85 326.18 us 88.00 us 66184.00 us 1000 TRUNCATE 4.47 511.89 us 60.00 us 101282.00 us 1000 FLUSH 5.02 3907.40 us 1723.00 us 19508.00 us 147 READDIRP 25.42 2876.37 us 101.00 us 843209.00 us 1012 LOOKUP 55.52 3179.16 us 124.00 us 121158.00 us 2000 WRITE Duration: 7074 seconds Data Read: 0 bytes Data Written: 102400000 bytes Interval 1 Stats: Block Size: 32768b+ 65536b+ No. of Reads: 0 0 No. of Writes: 1000 1000 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.01 410.33 us 194.00 us 641.00 us 3 STATFS 0.60 465.44 us 346.00 us 867.00 us 147 FSTAT 1.63 187.21 us 67.00 us 6081.00 us 1000 SETATTR 1.94 221.40 us 58.00 us 55399.00 us 1002 ACCESS 2.55 301.39 us 52.00 us 75922.00 us 968 STAT 2.85 326.18 us 88.00 us 66184.00 us 1000 TRUNCATE 4.47 511.89 us 60.00 us 101282.00 us 1000 FLUSH 5.02 3907.40 us 1723.00 us 19508.00 us 147 READDIRP 25.41 2878.07 us 101.00 us 843209.00 us 1011 LOOKUP 55.53 3179.16 us 124.00 us 121158.00 us 2000 WRITE Duration: 330 seconds Data Read: 0 bytes Data Written: 102400000 bytes
You can stop profiling the volume, if you do not need profiling information anymore.
To stop profiling
- Stop profiling using the following command:
# gluster volume profileVOLNAMEstopFor example, to stop profiling on test-volume:# gluster volume profiletest-volumestopProfiling stopped on test-volume