'Fuser' command's help page in the crash utility has limited information.

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • Crash Utility

Issue

  • Fuser command can accept a variety of inputs including dentry, inodes and full path among others.
  • Limited information is available on the fuser help page under crash utility.
  • The help/man page of the 'fuser' command suggests that the argument can be a full pathname or inode address.
  crash>  help fuser | grep -e SYNOPSIS -e pathname -A 1 
  SYNOPSIS
  fuser [pathname | inode]

  pathname: the full pathname of the file.
  inode:    the hexadecimal inode address for the file.

Resolution

  • RHEL 8: Fixed in version crash-7.3.2-8.el8.x86_64 via errata RHBA-2023:6947

  • RHEL 9: Fixed in version crash-8.0.3-1.el9.x86_64 via errata RHBA-2023:6391

Root Cause

  • The crash utility section of the fuser help page has limited information.

Diagnostic Steps

Let's see how the fuser command responds to different inputs.

  • The fuser command accepts the full pathname (PATH: /proc/kcore).
crash> set 40533
    PID: 40533
COMMAND: "crash"
   TASK: ffff9a36c6c65080  [THREAD_INFO: ffff9a36c6c65080]
    CPU: 0
  STATE: TASK_RUNNING (ACTIVE)

crash>  files -R /proc/kcore
PID: 40533  TASK: ffff9a36c6c65080  CPU: 0   COMMAND: "crash"
ROOT: /    CWD: /root 
 FD       FILE            DENTRY           INODE       TYPE PATH
  4 ffff9a36c44b7b00 ffff9a36c2281240 ffff9a36c141f650 REG  /proc/kcore
                                                                 '
                                                                 '
                                                                 v 
crash> fuser /proc/kcore  <<< - - - - - - - - - - - - - - - - - -'
 PID         TASK        COMM             USAGE
39361  ffff9a373e0ea840  "crash"          fd 
40533  ffff9a36c6c65080  "crash"          fd
  • The struct inode address (INODE: ffff9a36c141f650) can be passed through the fuser command.
crash> set 40533
    PID: 40533
COMMAND: "crash"
   TASK: ffff9a36c6c65080  [THREAD_INFO: ffff9a36c6c65080]
    CPU: 0
  STATE: TASK_RUNNING (ACTIVE)

crash>  files -R ffff9a36c141f650
PID: 40533  TASK: ffff9a36c6c65080  CPU: 0   COMMAND: "crash"
ROOT: /    CWD: /root 
 FD       FILE            DENTRY           INODE       TYPE PATH
  4 ffff9a36c44b7b00 ffff9a36c2281240 ffff9a36c141f650 REG  /proc/kcore

                                             v
                                             |
crash>  fuser ffff9a36c141f650 <<< - - - - - ' 
 PID         TASK        COMM             USAGE
39361  ffff9a373e0ea840  "crash"          fd 
40533  ffff9a36c6c65080  "crash"          fd
  • Fuser can display the task information by using the struct dentry address (DENTRY: ffff9a36c2281240).
crash> set 40533
    PID: 40533
COMMAND: "crash"
   TASK: ffff9a36c6c65080  [THREAD_INFO: ffff9a36c6c65080]
    CPU: 0
  STATE: TASK_RUNNING (ACTIVE)

crash> files -R ffff9a36c2281240
PID: 40533  TASK: ffff9a36c6c65080  CPU: 1   COMMAND: "crash"
ROOT: /    CWD: /root 
 FD       FILE            DENTRY           INODE       TYPE PATH
  4 ffff9a36c44b7b00 ffff9a36c2281240 ffff9a36c141f650 REG  /proc/kcore
                           v
                           |
                           v        
crash> fuser ffff9a36c2281240
 PID         TASK        COMM             USAGE
39361  ffff9a373e0ea840  "crash"          fd 
40533  ffff9a36c6c65080  "crash"          fd
  • Fuser command accepts the struct file address (FILE: ffff9a36c44b7b00) address in the crash utility.
crash> set 40533
    PID: 40533
COMMAND: "crash"
   TASK: ffff9a36c6c65080  [THREAD_INFO: ffff9a36c6c65080]
    CPU: 0
  STATE: TASK_RUNNING (ACTIVE)

crash> files -R ffff9a36c44b7b00
PID: 40533  TASK: ffff9a36c6c65080  CPU: 1   COMMAND: "crash"
ROOT: /    CWD: /root 
 FD       FILE            DENTRY           INODE       TYPE PATH
  4 ffff9a36c44b7b00 ffff9a36c2281240 ffff9a36c141f650 REG  /proc/kcore
          v
          '- - - - - - - . 
                         v
crash> fuser ffff9a36c44b7b00
 PID         TASK        COMM             USAGE
 5683  ffff949dc396d580  "crash"          fd 
 5690  ffff949dc55a9c80  "gdb worker"     fd 
 5691  ffff949dc0f38000  "gdb worker"     fd 
 5692  ffff949dc2a98000  "gdb worker"     fd 
 5693  ffff949dd535b900  "gdb worker"     fd 
 5694  ffff949dd5359c80  "gdb worker"     fd 
 5695  ffff949dc0de3900  "gdb worker"     fd 
 5696  ffff949dc0923900  "gdb worker"     fd 
 5697  ffff949dc4840000  "gdb worker"     fd 
 5698  ffff949dd4755580  "gdb worker"     fd 
 5699  ffff949dd475b900  "gdb worker"     fd 
 5700  ffff949dc0fb5580  "gdb worker"     fd 
 5701  ffff949dc5e98000  "gdb worker"     fd 
 5743  ffff949dd14c8000  "less"           fd

We can pass the different struct vm_area_struct.vm_start,vm_end,vm_flags to the fuser.
- Testing with struct vm_area_struct (VMA: ffff9a3731fa2658) address.

crash> set 40533
    PID: 40533
COMMAND: "crash"
   TASK: ffff9a36c6c65080  [THREAD_INFO: ffff9a36c6c65080]
    CPU: 0
  STATE: TASK_RUNNING (ACTIVE)

crash> vm -R ffff9a3731fa2658
PID: 40533  TASK: ffff9a36c6c65080  CPU: 0   COMMAND: "crash"
       MM               PGD          RSS    TOTAL_VM
ffff9a3731fdb600  ffff9a36c328c000  359664k  787876k 
      VMA           START       END     FLAGS FILE
ffff9a3731fa2658 5621467b0000 5621467d6000 8100873 /usr/bin/crash
       v
       '- - - - - .
                  v
crash> fuser ffff9a3731fa2658
 PID         TASK        COMM             USAGE
40533  ffff9a36c6c65080  "crash"          mmap
  • Testing with struct vm_area_struct.vm_start (vm_start = 0x55c944e89000) value.
crash> struct vm_area_struct.vm_start 0xffff9a36f38a2488
  vm_start = 0x55c944e89000

crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a36c1892840  [THREAD_INFO: ffff9a36c1892840]
    CPU: 1
  STATE: TASK_INTERRUPTIBLE

crash> vm -R 55c944e89000
PID: 1      TASK: ffff9a36c1892840  CPU: 1   COMMAND: "systemd"
       MM               PGD          RSS    TOTAL_VM
ffff9a36c24c1800  ffff9a36c24f6000  6316k   249592k 
      VMA           START       END     FLAGS FILE
ffff9a36f38a2488 55c944e89000 55c944e8a000 8100873 /usr/lib/systemd/systemd
VIRTUAL     PHYSICAL    V 
55c944e89000  5f6ec000  |
                        |
                        v  
crash> fuser 55c944e89000
 PID         TASK        COMM             USAGE
    1  ffff9a36c1892840  "systemd"        mmap 
 6708  ffff9a370531d080  "(sd-pam)"       mmap
  • Testing with vm_area_struct.vm_end (vm_end = 0xffff9a36f38a2488) value.
  • Ideally we should have obtained the VMA address, 0x55c944e8a000, and succeeded in dereferencing that VMA's ->vm_end.
crash>  vm_area_struct.vm_end 0xffff9a36f38a2488
  vm_end = 0x55c944e8a000

crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a36c1892840  [THREAD_INFO: ffff9a36c1892840]
    CPU: 1
  STATE: TASK_INTERRUPTIBLE

crash> vm -R 55c944e8a000
PID: 1      TASK: ffff9a36c1892840  CPU: 1   COMMAND: "systemd"
       MM               PGD          RSS    TOTAL_VM
ffff9a36c24c1800  ffff9a36c24f6000  6316k   249592k 
      VMA           START       END     FLAGS FILE
ffff9a36f38a2488 55c944e89000 55c944e8a000 8100873 /usr/lib/systemd/systemd
VIRTUAL     PHYSICAL               V
55c944e89000  5f6ec000             |
                                   |
                                   |
crash>  fuser 55c944e8a000<<< - - -'
 PID         TASK        COMM             USAGE
    1  ffff9a36c1892840  "systemd"        mmap 
 6708  ffff9a370531d080  "(sd-pam)"       mmap
  • Testing with struct vm_area_struct.vm_flags (vm_flags = 0x8000075) value.
crash> vm_area_struct.vm_flags 0xffff9a36f38a2488
  vm_flags = 0x8000075,

crash> set 1
    PID: 1
COMMAND: "systemd"
   TASK: ffff9a36c1892840  [THREAD_INFO: ffff9a36c1892840]
    CPU: 1
  STATE: TASK_INTERRUPTIBLE 

crash> vm -R 8000075
PID: 1      TASK: ffff9a36c1892840  CPU: 1   COMMAND: "systemd"
       MM               PGD          RSS    TOTAL_VM
ffff9a36c24c1800  ffff9a36c24f6000  6316k   249592k 
      VMA           START       END     FLAGS FILE
ffff9a36f4250658 7f23747de000 7f237495f000 8000075 /usr/lib64/libm-2.28.so
ffff9a36f4250570 7f2374b60000 7f2374bf2000 8000075 /usr/lib64/libudev.so.1.6.11
ffff9a36f4250828 7f2374df7000 7f2374ea4000 8000075 /usr/lib64/libsepol.so.1
ffff9a36ff7b0cb0 7f23750a8000 7f2375225000 8000075 /usr/lib64/libunistring.so.2.1.0
ffff9a36fdddc570 7f2375429000 7f2375448000 8000075 /usr/lib64/libgpg-error.so.0.24.2
ffff9a36fdddce80 7f237564a000 7f2375658000 8000075 /usr/lib64/libjson-c.so.4.0.0
ffff9a36fdddc658 7f237585a000 7f23758af000 8000075 /usr/lib64/libdevmapper.so.1.02
ffff9a36fdddcd98 7f2375ab4000 7f2375ab8000 8000075 /usr/lib64/libattr.so.1.1.2448
ffff9a36fddd7000 7f2375cba000 7f2375f6e000 8000075 /usr/lib64/libcrypto.so.1.1.1k
ffff9a36fddd7740 7f23761a3000 7f237622a000 8000075 /usr/lib64/libssl.so.1.1.1k
ffff9a36fddd73a0 7f2376437000 7f237644d000 8000075 /usr/lib64/libz.so.1.2.11
ffff9a36fddd71d0 7f237664f000 7f2376654000 8000075 /usr/lib64/libcap-ng.so.0.0.0
ffff9a36fddd7cb0 7f2376855000 7f237685b000 8000075 /usr/lib64/libuuid.so.1.3.0
ffff9a36fdde22b8 7f2376a5d000 7f2376a60000 8000075 /usr/lib64/libdl-2.28.so
ffff9a36fdde2d98 7f2376c61000 7f2376ce4000 8000075 /usr/lib64/libpcre2-8.so.0.7.1
ffff9a36fdde2000 7f2376ee5000 7f2376f32000 8000075 /usr/lib64/libblkid.so.1.1.0
ffff9a36fdde2ae0 7f2377138000 7f2377154000 8000075 /usr/lib64/liblz4.so.1.8.3
ffff9a36fe797000 7f2377355000 7f237737a000 8000075 /usr/lib64/liblzma.so.5.2.4
ffff9a36fe7979f8 7f237757c000 7f2377598000 8000075 /usr/lib64/libidn2.so.0.3.6
ffff9a36fe797d98 7f237779a000 7f23778b2000 8000075 /usr/lib64/libgcrypt.so.20.2.5
ffff9a36fe7970e8 7f2377ab8000 7f2377b32000 8000075 /usr/lib64/libcryptsetup.so.12.6.0
ffff9a36fe797570 7f2377d36000 7f2377d3e000 8000075 /usr/lib64/libacl.so.1.1.2253
ffff9a36f494e000 7f2377f3f000 7f2377f46000 8000075 /usr/lib64/libcap.so.2.48
ffff9a36f496a9f8 7f2378147000 7f2378303000 8000075 /usr/lib64/libc-2.28.so
ffff9a36f496a910 7f237850d000 7f2378528000 8000075 /usr/lib64/libpthread-2.28.so
ffff9a36f496a1d0 7f237872d000 7f2378744000 8000075 /usr/lib64/libgcc_s-8-20210514.so.1
ffff9a36f496a3a0 7f2378945000 7f237895c000 8000075 /usr/lib64/libkmod.so.2.3.3
ffff9a36f494e740 7f2378b5d000 7f2378b7c000 8000075 /usr/lib64/libaudit.so.1.0.0
ffff9a36f494ebc8 7f2378d8e000 7f2378d9c000 8000075 /usr/lib64/libpam.so.0.84.2
ffff9a36f494e658 7f2378f9e000 7f2378ff4000 8000075 /usr/lib64/libmount.so.1.1.0
ffff9a36f494e0e8 7f23791f8000 7f237921f000 8000075 /usr/lib64/libselinux.so.1
ffff9a36f494e2b8 7f2379423000 7f2379440000 8000075 /usr/lib64/libseccomp.so.2.5.2
ffff9a36f3881570 7f2379642000 7f2379649000 8000075 /usr/lib64/librt-2.28.so
ffff9a36f38819f8 7f237984a000 7f2379a54000 8000075 /usr/lib/systemd/libsystemd-shared-239.so
                                             V
crash> fuser 8000075   <<< - - - - - - - - - ' 
 PID         TASK        COMM             USAGE
    1  ffff9a36c1892840  "systemd"        mmap 
  717  ffff9a36f3bdd080  "systemd-journa  mmap 
  765  ffff9a36f33ad080  "systemd-udevd"  mmap 
  864  ffff9a36c30a8000  "rpcbind"        mmap 
  868  ffff9a36c30ad080  "auditd"         mmap 
  869  ffff9a36c2ec0000  "auditd"         mmap 
  870  ffff9a36c2ec2840  "sedispatch"     mmap 
  871  ffff9a36f4955080  "auditd"         mmap 
  901  ffff9a36c5a6a840  "smartd"         mmap 
  902  ffff9a36c2ec5080  "lsmd"           mmap 
  903  ffff9a36c30aa840  "sssd"           mmap 
  904  ffff9a36c398d080  "qemu-ga"        mmap 
  905  ffff9a36c3988000  "dbus-daemon"    mmap 
  908  ffff9a36f33a8000  "systemd-machin  mmap 
  910  ffff9a36f4938000  "alsactl"        mmap 
  914  ffff9a36c5a40000  "chronyd"        mmap 
  915  ffff9a36c2e20000  "udisksd"        mmap 
  916  ffff9a36f3bf2840  "mcelog"         mmap 
  919  ffff9a36c4690000  "irqbalance"     mmap 
  920  ffff9a36c5c60000  "polkitd"        mmap 
  921  ffff9a36c5c62840  "rtkit-daemon"   mmap 
  923  ffff9a36c3a42840  "avahi-daemon"   mmap 
  933  ffff9a36c5c65080  "qemu-ga"        mmap 
  937  ffff9a36c6768000  "ksmtuned"       mmap 
  943  ffff9a36c676d080  "gmain"          mmap 
  944  ffff9a36c4ff5080  "gmain"          mmap 
  950  ffff9a36c6680000  "dbus-daemon"    mmap 
  956  ffff9a36c3a40000  "avahi-daemon"   mmap 
  962  ffff9a36f4cea840  "gmain"          mmap 
  963  ffff9a36c5a42840  "rtkit-daemon"   mmap 
  964  ffff9a36c2e22840  "rtkit-daemon"   mmap 
  965  ffff9a36c6685080  "gdbus"          mmap 
  966  ffff9a36f4388000  "sssd_be"        mmap 
  969  ffff9a36c5a68000  "gdbus"          mmap 
  970  ffff9a36c4ff2840  "JS Helper"      mmap 
  971  ffff9a36c6045080  "JS Helper"      mmap 
  977  ffff9a36c3e95080  "sssd_nss"       mmap 
  978  ffff9a36c6042840  "polkitd"        mmap 
  980  ffff9a36c6040000  "firewalld"      mmap 
  981  ffff9a36f33aa840  "ModemManager"   mmap 
  982  ffff9a36c4692840  "probing-thread  mmap 
  983  ffff9a36c3e90000  "gmain"          mmap 
  991  ffff9a36c6445080  "gdbus"          mmap 
 1033  ffff9a36c589a840  "cleanup"        mmap 
 1051  ffff9a36c57f8000  "systemd-logind  mmap 
 1052  ffff9a36c57fa840  "accounts-daemo  mmap 
 1053  ffff9a36c57fd080  "gmain"          mmap 
 1055  ffff9a36c5898000  "gdbus"          mmap 
 1078  ffff9a36c5818000  "NetworkManager  mmap 
 1091  ffff9a36c5968000  "gmain"          mmap 
 1095  ffff9a36c596a840  "gdbus"          mmap 
 1097  ffff9a36f2d75080  "cupsd"          mmap 
 1098  ffff9a36c84da840  "tuned"          mmap 
 1104  ffff9a36c4b55080  "rhsmcertd"      mmap 
 1109  ffff9a36c4b52840  "gssproxy"       mmap 
 1111  ffff9a36f3068000  "gssproxy"       mmap 
 1112  ffff9a36f306a840  "gssproxy"       mmap 
 1113  ffff9a36f306d080  "gssproxy"       mmap 
 1114  ffff9a36f3070000  "gssproxy"       mmap 
 1115  ffff9a36f3072840  "gssproxy"       mmap 
 1245  ffff9a36c37b2840  "sshd"           mmap 
 1360  ffff9a36c43ed080  "tuned"          mmap 
 1374  ffff9a36c3672840  "tuned"          mmap 
 1375  ffff9a36c3670000  "tuned"          mmap 
 1376  ffff9a36c3675080  "gmain"          mmap 
 1377  ffff9a36c43e8000  "tuned"          mmap 
 1395  ffff9a36c3642840  "rsyslogd"       mmap 
 1405  ffff9a36c6440000  "crond"          mmap 
 1407  ffff9a36c4b6d080  "atd"            mmap 
 1409  ffff9a36c43ea840  "in:imjournal"   mmap 
 1410  ffff9a36c4305080  "rs:main Q:Reg"  mmap 
 1855  ffff9a36c5a6d080  "dnsmasq"        mmap 
 1856  ffff9a36c3640000  "dnsmasq"        mmap 
 6293  ffff9a36c3e92840  "gdm"            mmap 
 6294  ffff9a36c6de5080  "gmain"          mmap 
 6295  ffff9a36c6de0000  "gdbus"          mmap 
 6445  ffff9a36c4695080  "upowerd"        mmap 
 6446  ffff9a36c6d85080  "gmain"          mmap 
 6447  ffff9a36c6d80000  "gdbus"          mmap 
 6465  ffff9a36c6d82840  "packagekitd"    mmap 
 6467  ffff9a37051fa840  "wpa_supplicant  mmap 
 6469  ffff9a36f2f80000  "gmain"          mmap 
 6470  ffff9a36c6c4d080  "gdbus"          mmap 
 6541  ffff9a3706eda840  "colord"         mmap 
 6547  ffff9a3706fc0000  "gmain"          mmap 
 6558  ffff9a3706e4a840  "gdbus"          mmap 
 6612  ffff9a3706ffa840  "spice-vdagentd  mmap 
 6613  ffff9a3706e40000  "gmain"          mmap 
 6688  ffff9a3705200000  "gdm-session-wo  mmap 
 6689  ffff9a3705202840  "gmain"          mmap 
 6691  ffff9a36c398a840  "gdbus"          mmap 
 6704  ffff9a3718e7d080  "systemd"        mmap 
 6708  ffff9a370531d080  "(sd-pam)"       mmap 
 6716  ffff9a36c6c62840  "pulseaudio"     mmap 
 6722  ffff9a36c547a840  "gnome-keyring-  mmap 
 6723  ffff9a3705272840  "gmain"          mmap

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments