rpcbind fails to start with IPv6 disabled

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.3 and later
  • net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf

Issue

  • On RHEL7.3 and later with IPv6 disabled, rpcbind fails to start

    # systemctl status rpcbind.socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)
           0.0.0.0:111 (Datagram)
           [::]:111 (Stream)
           [::]:111 (Datagram)

[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.

  • On RHEL7.3 and later with IPv6 disabled, an AVC is seen regarding rpc.statd

    type=AVC msg=audit(XXX): avc:  denied  { write } for  pid=19644 comm="rpc.statd" path="/run/rpc.statd.lock" dev="tmpfs" ino=45143 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=0

Resolution

  • Verify that dracut is not configured to produce a generic initramfs, but a host-only initramfs

    # grep -rw ^hostonly /usr/lib/dracut/dracut.conf.d /etc/dracut.conf /etc/dracut.conf.d
/usr/lib/dracut/dracut.conf.d/01-dist.conf:hostonly="yes"
/usr/lib/dracut/dracut.conf.d/02-generic-image.conf:hostonly="no"

    In the example above, the file /usr/lib/dracut/dracut.conf.d/02-generic-image.conf (shipped by the dracut-config-generic package) is setting hostonly="no", which will produce a generic initramfs. You hence need to remove this package from the system:

    # yum -y remove dracut-config-generic

  • Rebuild the initramfs to incorporate the changes made to /etc/sysctl.conf into the initramfs

    # dracut -v -f

    It is necessary to rebuild the initramfs when ever there are modifications made to /etc/sysctl.conf.

  • Verify that the initramfs contains the /etc/sysctl.conf file and matches the one on the system

    # diff -u /etc/sysctl.conf <(lsinitrd -f /etc/sysctl.conf)

    In the example above, the file was included properly since the diff command didn't return anything.

  • Reboot the system

  • For additional details please refer to When is it necessary to rebuild the initramfs?

Workaround:

Workaround is to create a systemd override file (Managing Services with systemd Unit File Modify) for rpcbind.socket containing

 [Socket]
 ListenStream=/var/run/rpcbind.sock
 ListenStream=0.0.0.0:111

Red Hat recommends to rebuild the initramfs than the workaround.

Root Cause

  • This problem occurs when omitting the step to update the initramfs per How do I disable or enable the IPv6 protocol in Red Hat Enterprise Linux?

  • Without systemd being aware that IPv6 is disabled in the early boot process, it will attempt to make a connection on port 111 on any IPv6 address, [::]:111, due to updates made to rpcbind per BZ 1359592.

    # rpm -q rpcbind --changelog | head
* Tue Aug 02 2016 Steve Dickson <steved@redhat.com> - 0.2.0-38
- Removing the braces from the ${RPCBIND_ARGS} in rpcbind.service (bz 1362232)

* Fri Jul 29 2016 Steve Dickson <steved@redhat.com> - 0.2.0-37
- Make sure rpcbind.socket listens for remote IPv6 connections (bz 1359592)

* Thu Jul 21 2016 Steve Dickson <steved@redhat.com> - 0.2.0-36
- Added upstream debugging (bz 1358890)

* Sat Apr 09 2016 Steve Dickson <steved@redhat.com> - 0.2.0-35

  • rpcbind.socket service file for rpcbind-0.2.0-38.el7.x86_64 released with Red Hat Enterprise Linux Server release 7.3 and later

    # systemctl cat rpcbind.socket
# /usr/lib/systemd/system/rpcbind.socket
[Unit]
Description=RPCbind Server Activation Socket

[Socket]
ListenStream=/var/run/rpcbind.sock
ListenStream=[::]:111
ListenStream=0.0.0.0:111
BindIPv6Only=ipv6-only

[Install]
WantedBy=sockets.target

  • rpcbind.socket service file for rpcbind-0.2.0-32.el7.x86_64 released with Red Hat Enterprise Linux Server release 7.2

    # systemctl cat rpcbind.socket
# /usr/lib/systemd/system/rpcbind.socket
[Unit]
Description=RPCbind Server Activation Socket

[Socket]
ListenStream=/var/run/rpcbind.sock

[Install]
WantedBy=sockets.target

Diagnostic Steps

  • rpcbind.socket status before disabling IPv6

    # systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since [...] ago
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: Listening on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket

  • After adding net.ipv6.conf.all.disable_ipv6 = 1 to /etc/sysctl.conf and issuing rebooting

    # systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.
[...] systemd[1]: Unit rpcbind.socket entered failed state.
[...] systemd[1]: Starting RPCbind Server Activation Socket.
[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket.
...

  • After rebuilding the initramfs and rebooting

    ● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since [...] ago
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: Listening on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

7 Comments

Log in to comment
NB Newbie 5 points
14 December 2016 2:51 PM

Hi,

Shouldn't -H also be passed to dracut so /etc/sysctl.conf is copied into the initial RAM filesystem?

# uname -r
3.10.0-514.2.2.el7.x86_64
# dracut -v -f > /dev/null 2>&1
# lsinitrd /boot/initramfs-3.10.0-514.2.2.el7.x86_64.img -f /etc/sysctl.conf
# dracut -v -f -H > /dev/null 2>&1
# lsinitrd /boot/initramfs-3.10.0-514.2.2.el7.x86_64.img -f /etc/sysctl.conf
# System default settings live in /usr/lib/sysctl.d/00-system.conf.
# To override those settings, enter new settings here, or in an /etc/sysctl.d/<name>.conf file
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv6.conf.all.disable_ipv6 = 1
JS Red Hat Active Contributor 120 points
2 March 2017 1:03 AM

It is not required to use the -H flag to incorporate the changes into the initramfs.

Matt Millard's picture Active Contributor 100 points
18 May 2017 2:19 PM

Will there be a fix for this behavior? Having to update initramfs for a setting change in sysctl.conf doesn't make much sense. Why does rpcbind assume IPV6 is enabled?

JS Red Hat Active Contributor 120 points
18 June 2017 5:03 PM

Hello Matt,

"Will there be a fix for this behavior?" o The behavior is expected behavior where IPv6 is not properly disabled either via a grub boot parameter or disabling the protocol via a sysctl.

"Having to update initramfs for a setting change in sysctl.conf doesn't make much sense. " o /etc/sysctl.conf && /etc/sysctl.d are read during the early boot process. If IPv6 is disabled via a sysctl the system has to be informed of such disabling during the early boot process particularly due to SystemD starting much earlier that previous init systems.

"Why does rpcbind assume IPV6 is enabled?" o As stated above, the change in behavior is due to an update in rpcbind * Fri Jul 29 2016 Steve Dickson steved@redhat.com - 0.2.0-37 - Make sure rpcbind.socket listens for remote IPv6 connections (bz 1359592)

Matthew Galloway's picture Community Member 21 points
14 June 2018 7:56 PM

6/14/2018 - We currently are running RHEL 7.4. The rebuilding of initramfs doesn't seem to solve the issue. Disabling via grub does seem to do the trick though.

UPDATE-We had rebuilt the initramfs SO many times but /usr/lib/systemd/system/rpcbind.socket still contained the IPv6 streams. Once those streams were removed from /usr/lib/systemd/system/rpcbind.socket, systemd daemon-reload. NFS mounts work as expected and it's persistent across reboots. Highly recommend you copying rpcbind.socket to /etc/systemd/system THEN edit the file..reload...etc...etc.

RS Community Member 35 points
15 February 2019 12:15 PM 
# uname -r
3.10.0-957.1.3.el7.x86_64

# lsinitrd initramfs-3.10.0-957.1.3.el7.x86_64.img | grep sysctl
drwxr-xr-x   2 root     root            0 Feb 15 11:46 usr/lib/sysctl.d
-rw-r--r--   1 root     root          293 Aug 24 12:36 usr/lib/sysctl.d/00-system.conf
-rw-r--r--   1 root     root         1810 Jun 20  2018 usr/lib/sysctl.d/10-default-yama-scope.conf
-rw-r--r--   1 root     root         1205 Sep  7 10:00 usr/lib/sysctl.d/50-default.conf
-rwxr-xr-x   1 root     root        70600 Sep  7 10:01 usr/lib/systemd/systemd-sysctl
lrwxrwxrwx   1 root     root           25 Feb 15 11:46 usr/lib/systemd/system/sysinit.target.wants/systemd-sysctl.service -> ../systemd-sysctl.service
-rw-r--r--   1 root     root          711 Sep  7 10:00 usr/lib/systemd/system/systemd-sysctl.service

# cat /etc/sysctl.d/10-ipv6.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

# dracut -f

# lsinitrd initramfs-3.10.0-957.1.3.el7.x86_64.img | grep sysctl
drwxr-xr-x   2 root     root            0 Feb 15 13:04 usr/lib/sysctl.d
-rw-r--r--   1 root     root          293 Aug 24 12:36 usr/lib/sysctl.d/00-system.conf
-rw-r--r--   1 root     root         1810 Jun 20  2018 usr/lib/sysctl.d/10-default-yama-scope.conf
-rw-r--r--   1 root     root         1205 Sep  7 10:00 usr/lib/sysctl.d/50-default.conf
-rwxr-xr-x   1 root     root        70600 Sep  7 10:01 usr/lib/systemd/systemd-sysctl
lrwxrwxrwx   1 root     root           25 Feb 15 13:04 usr/lib/systemd/system/sysinit.target.wants/systemd-sysctl.service -> ../systemd-sysctl.service
-rw-r--r--   1 root     root          711 Sep  7 10:00 usr/lib/systemd/system/systemd-sysctl.service

# ln -s /etc/sysctl.d/10-ipv6.conf /usr/lib/sysctl.d/10-ipv6.conf

# dracut -f

# lsinitrd initramfs-3.10.0-957.1.3.el7.x86_64.img | grep sysctl
drwxr-xr-x   2 root     root            0 Feb 15 13:07 etc/sysctl.d
-rw-r--r--   1 root     root          159 Oct 30 09:05 **etc/sysctl.d/10-ipv6.conf**
drwxr-xr-x   2 root     root            0 Feb 15 13:07 usr/lib/sysctl.d
-rw-r--r--   1 root     root          293 Aug 24 12:36 usr/lib/sysctl.d/00-system.conf
-rw-r--r--   1 root     root         1810 Jun 20  2018 usr/lib/sysctl.d/10-default-yama-scope.conf
lrwxrwxrwx   1 root     root           34 Feb 15 13:07 usr/lib/sysctl.d/**10-ipv6.conf -> ../../../etc/sysctl.d/10-ipv6.conf**
-rw-r--r--   1 root     root         1205 Sep  7 10:00 usr/lib/sysctl.d/50-default.conf
-rwxr-xr-x   1 root     root        70600 Sep  7 10:01 usr/lib/systemd/systemd-sysctl
lrwxrwxrwx   1 root     root           25 Feb 15 13:07 usr/lib/systemd/system/sysinit.target.wants/systemd-sysctl.service -> ../systemd-sysctl.service
-rw-r--r--   1 root     root          711 Sep  7 10:00 usr/lib/systemd/system/systemd-sysctl.service
VB Community Member 72 points
22 October 2019 6:07 PM

This solved the problem for us.

sudo systemctl enable rpcbind.socket sudo systemctl restart rpcbind.socket