LSOF shows different number of open files from that in '/proc/<PID>/fd'

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux
    • 5
    • 6
    • 7

Issue

We are trying to understand the below output.

The number of open files is drastically larger than the number of file descriptors for a given process:

[root@srv01 ~]# lsof -p 51960 | wc -l
50286

[root@hostname fd]# pwd
/proc/51960/fd
[root@hostname fd]# ll | wc -l
3379

How are the files reported by lsof remaining open?

Resolution

With -p option, lsof lists all open files belonging to the process which has the pid specified with -p. These include the open files having the normal FD numbers along with linked libraries, executables and so on. This is the reason why the number of open files is drastically larger than that in /proc/<pid>/fd. It's possible to have the same result by excluding the latters with -d option. It would be an exclusion list if all entries of the set begin with "^".

LSOF(8)

       FD         is the File Descriptor number of the file or:

                       cwd  current working directory;
                       Lnn  library references (AIX);
                       err  FD information error (see NAME column);
                       jld  jail directory (FreeBSD);
                       ltx  shared library text (code and data);
                       Mxx  hex memory-mapped type number xx.
                       m86  DOS Merge mapped file;
                       mem  memory-mapped file;
                       mmap memory-mapped device;
                       pd   parent directory;
                       rtd  root directory;
                       tr   kernel trace file (OpenBSD);
                       txt  program text (code and data);
                       v86  VP/ix mapped file;

Example:

# lsof -p 20818
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME
httpd   20818 apache  cwd    DIR  253,0     4096      2 /
httpd   20818 apache  rtd    DIR  253,0     4096      2 /
httpd   20818 apache  txt    REG  253,0   354816 659107 /usr/sbin/httpd
httpd   20818 apache  mem    REG  253,0    31992   6284 /lib64/libnss_sss.so.2
httpd   20818 apache  mem    REG  253,0     9488 673904 /usr/lib64/apr-util-1/apr_ldap-1.so
httpd   20818 apache  mem    REG  253,0    27424     42 /lib64/libnss_dns-2.12.so
httpd   20818 apache  mem    REG  253,0    65928     44 /lib64/libnss_files-2.12.so
httpd   20818 apache  mem    REG  253,0    10192   1183 /lib64/libkeyutils.so.1.3
httpd   20818 apache  mem    REG  253,0    43728   1195 /lib64/libkrb5support.so.0.1
httpd   20818 apache  mem    REG  253,0   174840   1191 /lib64/libk5crypto.so.3.1
httpd   20818 apache  mem    REG  253,0    14664     90 /lib64/libcom_err.so.2.1
httpd   20818 apache  mem    REG  253,0   941920   1193 /lib64/libkrb5.so.3.3
httpd   20818 apache  mem    REG  253,0   277704   1187 /lib64/libgssapi_krb5.so.2.2
httpd   20818 apache  mem    REG  253,0  1946880 655817 /usr/lib64/libcrypto.so.1.0.1e
httpd   20818 apache  mem    REG  253,0   436984 655819 /usr/lib64/libssl.so.1.0.1e
httpd   20818 apache  mem    REG  253,0   177808 921107 /usr/lib64/httpd/modules/mod_ssl.so
httpd   20818 apache  mem    REG  253,0    10416 920837 /usr/lib64/httpd/modules/mod_version.so
httpd   20818 apache  mem    REG  253,0    27312 920749 /usr/lib64/httpd/modules/mod_cgi.so
httpd   20818 apache  mem    REG  253,0    22992 920778 /usr/lib64/httpd/modules/mod_disk_cache.so
httpd   20818 apache  mem    REG  253,0    10384 920833 /usr/lib64/httpd/modules/mod_suexec.so
httpd   20818 apache  mem    REG  253,0    39664 920743 /usr/lib64/httpd/modules/mod_cache.so
httpd   20818 apache  mem    REG  253,0    14648 920823 /usr/lib64/httpd/modules/mod_proxy_connect.so
httpd   20818 apache  mem    REG  253,0    39632 920808 /usr/lib64/httpd/modules/mod_proxy_ajp.so
httpd   20818 apache  mem    REG  253,0    31472 920825 /usr/lib64/httpd/modules/mod_proxy_http.so
httpd   20818 apache  mem    REG  253,0    35600 920824 /usr/lib64/httpd/modules/mod_proxy_ftp.so
httpd   20818 apache  mem    REG  253,0    27160 920809 /usr/lib64/httpd/modules/mod_proxy_balancer.so
httpd   20818 apache  mem    REG  253,0    85328 920803 /usr/lib64/httpd/modules/mod_proxy.so
httpd   20818 apache  mem    REG  253,0    60464 920828 /usr/lib64/httpd/modules/mod_rewrite.so
httpd   20818 apache  mem    REG  253,0    14608 920832 /usr/lib64/httpd/modules/mod_substitute.so
httpd   20818 apache  mem    REG  253,0    14608 920693 /usr/lib64/httpd/modules/mod_alias.so
httpd   20818 apache  mem    REG  253,0    10448 920835 /usr/lib64/httpd/modules/mod_userdir.so
httpd   20818 apache  mem    REG  253,0    14608 920830 /usr/lib64/httpd/modules/mod_speling.so
httpd   20818 apache  mem    REG  253,0    10416 920685 /usr/lib64/httpd/modules/mod_actions.so
httpd   20818 apache  mem    REG  253,0    10416 920777 /usr/lib64/httpd/modules/mod_dir.so
httpd   20818 apache  mem    REG  253,0    31384 920802 /usr/lib64/httpd/modules/mod_negotiation.so
httpd   20818 apache  mem    REG  253,0    10424 920838 /usr/lib64/httpd/modules/mod_vhost_alias.so
httpd   20818 apache  mem    REG  253,0    52016 920758 /usr/lib64/httpd/modules/mod_dav_fs.so
httpd   20818 apache  mem    REG  253,0    19032 920795 /usr/lib64/httpd/modules/mod_info.so
httpd   20818 apache  mem    REG  253,0    35472 920740 /usr/lib64/httpd/modules/mod_autoindex.so
httpd   20818 apache  mem    REG  253,0    22864 920831 /usr/lib64/httpd/modules/mod_status.so
httpd   20818 apache  mem    REG  253,0    93328 920755 /usr/lib64/httpd/modules/mod_dav.so
httpd   20818 apache  mem    REG  253,0    18800 920800 /usr/lib64/httpd/modules/mod_mime.so
httpd   20818 apache  mem    REG  253,0    14576 920829 /usr/lib64/httpd/modules/mod_setenvif.so
httpd   20818 apache  mem    REG  253,0    14608 920836 /usr/lib64/httpd/modules/mod_usertrack.so
httpd   20818 apache  mem    REG  253,0    18896 920792 /usr/lib64/httpd/modules/mod_headers.so
httpd   20818 apache  mem    REG  253,0    22960 920776 /usr/lib64/httpd/modules/mod_deflate.so
httpd   20818 apache  mem    REG  253,0    10512 920789 /usr/lib64/httpd/modules/mod_expires.so
httpd   20818 apache  mem    REG  253,0    27672 920801 /usr/lib64/httpd/modules/mod_mime_magic.so
httpd   20818 apache  mem    REG  253,0    23120 920790 /usr/lib64/httpd/modules/mod_ext_filter.so
httpd   20818 apache  mem    REG  253,0    10416 920788 /usr/lib64/httpd/modules/mod_env.so
httpd   20818 apache  mem    REG  253,0    10440 920799 /usr/lib64/httpd/modules/mod_logio.so
httpd   20818 apache  mem    REG  253,0    27168 920797 /usr/lib64/httpd/modules/mod_log_config.so
httpd   20818 apache  mem    REG  253,0    43600 920794 /usr/lib64/httpd/modules/mod_include.so
httpd   20818 apache  mem    REG  253,0    31192 920711 /usr/lib64/httpd/modules/mod_authnz_ldap.so
httpd   20818 apache  mem    REG  253,0    43832     56 /lib64/librt-2.12.so
httpd   20818 apache  mem    REG  253,0    88600     80 /lib64/libz.so.1.2.3
httpd   20818 apache  mem    REG  253,0   106160 654714 /usr/lib64/libsasl2.so.2.0.23
httpd   20818 apache  mem    REG  253,0   240560     92 /lib64/libnspr4.so
httpd   20818 apache  mem    REG  253,0    18720     93 /lib64/libplc4.so
httpd   20818 apache  mem    REG  253,0    14560     94 /lib64/libplds4.so
httpd   20818 apache  mem    REG  253,0   179416 654497 /usr/lib64/libnssutil3.so
httpd   20818 apache  mem    REG  253,0  1296368 655752 /usr/lib64/libnss3.so
httpd   20818 apache  mem    REG  253,0   181168 655754 /usr/lib64/libsmime3.so
httpd   20818 apache  mem    REG  253,0   246584 655755 /usr/lib64/libssl3.so
httpd   20818 apache  mem    REG  253,0   110960     54 /lib64/libresolv-2.12.so
httpd   20818 apache  mem    REG  253,0    60512   1153 /lib64/liblber-2.4.so.2.5.6
httpd   20818 apache  mem    REG  253,0   314240   1155 /lib64/libldap-2.4.so.2.5.6
httpd   20818 apache  mem    REG  253,0    56144 920796 /usr/lib64/httpd/modules/mod_ldap.so
httpd   20818 apache  mem    REG  253,0     6264 920713 /usr/lib64/httpd/modules/mod_authz_default.so
httpd   20818 apache  mem    REG  253,0    10448 920712 /usr/lib64/httpd/modules/mod_authz_dbm.so
httpd   20818 apache  mem    REG  253,0    10520 920716 /usr/lib64/httpd/modules/mod_authz_groupfile.so
httpd   20818 apache  mem    REG  253,0    10424 920722 /usr/lib64/httpd/modules/mod_authz_owner.so
httpd   20818 apache  mem    REG  253,0    10384 920737 /usr/lib64/httpd/modules/mod_authz_user.so
httpd   20818 apache  mem    REG  253,0    10448 920719 /usr/lib64/httpd/modules/mod_authz_host.so
httpd   20818 apache  mem    REG  253,0     6264 920709 /usr/lib64/httpd/modules/mod_authn_default.so
httpd   20818 apache  mem    REG  253,0    10416 920708 /usr/lib64/httpd/modules/mod_authn_dbm.so
httpd   20818 apache  mem    REG  253,0    10352 920706 /usr/lib64/httpd/modules/mod_authn_anon.so
httpd   20818 apache  mem    REG  253,0    10456 920701 /usr/lib64/httpd/modules/mod_authn_alias.so
httpd   20818 apache  mem    REG  253,0    10416 920710 /usr/lib64/httpd/modules/mod_authn_file.so
httpd   20818 apache  mem    REG  253,0    31256 920700 /usr/lib64/httpd/modules/mod_auth_digest.so
httpd   20818 apache  mem    REG  253,0    10448 920697 /usr/lib64/httpd/modules/mod_auth_basic.so
httpd   20818 apache  mem    REG  253,0   469528     20 /lib64/libfreebl3.so
httpd   20818 apache  mem    REG  253,0    16304    118 /lib64/libuuid.so.1.3.0
httpd   20818 apache  mem    REG  253,0    19536     34 /lib64/libdl-2.12.so
httpd   20818 apache  mem    REG  253,0  1921216     28 /lib64/libc-2.12.so
httpd   20818 apache  mem    REG  253,0   142640     52 /lib64/libpthread-2.12.so
httpd   20818 apache  mem    REG  253,0   181856 663742 /usr/lib64/libapr-1.so.0.3.9
httpd   20818 apache  mem    REG  253,0  1522808     91 /lib64/libdb-4.7.so
httpd   20818 apache  mem    REG  253,0   165264    122 /lib64/libexpat.so.1.5.2
httpd   20818 apache  mem    REG  253,0    40400     32 /lib64/libcrypt-2.12.so
httpd   20818 apache  mem    REG  253,0   146080 663765 /usr/lib64/libaprutil-1.so.0.3.9
httpd   20818 apache  mem    REG  253,0   122040     96 /lib64/libselinux.so.1
httpd   20818 apache  mem    REG  253,0   181432    111 /lib64/libpcre.so.0.0.1
httpd   20818 apache  mem    REG  253,0   596264     36 /lib64/libm-2.12.so
httpd   20818 apache  mem    REG  253,0   154520     21 /lib64/ld-2.12.so
httpd   20818 apache  DEL    REG    0,4           42657 /dev/zero
httpd   20818 apache  DEL    REG    0,4          155021 /dev/zero
httpd   20818 apache  DEL    REG    0,4          155016 /dev/zero
httpd   20818 apache    0r   CHR    1,3      0t0   3800 /dev/null
httpd   20818 apache    1w   CHR    1,3      0t0   3800 /dev/null
httpd   20818 apache    2w   REG  253,0      280 924961 /var/log/httpd/error_log
httpd   20818 apache    3u  sock    0,6      0t0  42619 can't identify protocol
httpd   20818 apache    4u  IPv6  42620      0t0    TCP *:http (LISTEN)
httpd   20818 apache    5u  sock    0,6      0t0  42623 can't identify protocol
httpd   20818 apache    6u  IPv6  42624      0t0    TCP *:https (LISTEN)
httpd   20818 apache    7r  FIFO    0,8      0t0 155015 pipe
httpd   20818 apache    8w  FIFO    0,8      0t0 155015 pipe
httpd   20818 apache    9w   REG  253,0      242 925009 /var/log/httpd/ssl_error_log
httpd   20818 apache   10w   REG  253,0        0 924955 /var/log/httpd/access_log
httpd   20818 apache   11w   REG  253,0        0 921120 /var/log/httpd/ssl_access_log
httpd   20818 apache   12w   REG  253,0        0 921124 /var/log/httpd/ssl_request_log
httpd   20818 apache   13r   CHR    1,9      0t0   3805 /dev/urandom
httpd   20818 apache   14u   REG    0,9        0   3798 [eventpoll]

# ll /proc/20818/fd
total 0
lr-x------. 1 root root 64 Sep 25 11:43 0 -> /dev/null
l-wx------. 1 root root 64 Sep 25 11:43 1 -> /dev/null
l-wx------. 1 root root 64 Sep 25 11:43 10 -> /var/log/httpd/access_log
l-wx------. 1 root root 64 Sep 25 11:43 11 -> /var/log/httpd/ssl_access_log
l-wx------. 1 root root 64 Sep 25 11:43 12 -> /var/log/httpd/ssl_request_log
lr-x------. 1 root root 64 Sep 25 11:43 13 -> /dev/urandom
lrwx------. 1 root root 64 Sep 25 11:43 14 -> [eventpoll]
l-wx------. 1 root root 64 Sep 25 11:43 2 -> /var/log/httpd/error_log
lrwx------. 1 root root 64 Sep 25 11:43 3 -> socket:[42619]
lrwx------. 1 root root 64 Sep 25 11:43 4 -> socket:[42620]
lrwx------. 1 root root 64 Sep 25 11:43 5 -> socket:[42623]
lrwx------. 1 root root 64 Sep 25 11:43 6 -> socket:[42624]
lr-x------. 1 root root 64 Sep 25 11:43 7 -> pipe:[155015]
l-wx------. 1 root root 64 Sep 25 11:43 8 -> pipe:[155015]
l-wx------. 1 root root 64 Sep 25 11:43 9 -> /var/log/httpd/ssl_error_log

# lsof -a -p 20818 -d ^mem -d ^cwd -d ^rtd -d ^txt -d ^DEL | wc -l
16

# ll /proc/20818/fd | wc -l
16

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments