rpcbind fails to start with IPv6 disabled

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux 7.3 and later
  • net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf

Issue

  • On RHEL7.3 and later with IPv6 disabled, rpcbind fails to start

    # systemctl status rpcbind.socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)
           0.0.0.0:111 (Datagram)
           [::]:111 (Stream)
           [::]:111 (Datagram)

[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.

  • On RHEL7.3 and later with IPv6 disabled, an AVC is seen regarding rpc.statd

    type=AVC msg=audit(XXX): avc:  denied  { write } for  pid=19644 comm="rpc.statd" path="/run/rpc.statd.lock" dev="tmpfs" ino=45143 scontext=system_u:system_r:rpcd_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=0

Resolution

  • Verify that dracut is not configured to produce a generic initramfs, but a host-only initramfs

    # grep -rw ^hostonly /usr/lib/dracut/dracut.conf.d /etc/dracut.conf /etc/dracut.conf.d
/usr/lib/dracut/dracut.conf.d/01-dist.conf:hostonly="yes"
/usr/lib/dracut/dracut.conf.d/02-generic-image.conf:hostonly="no"

    In the example above, the file /usr/lib/dracut/dracut.conf.d/02-generic-image.conf (shipped by the dracut-config-generic package) is setting hostonly="no", which will produce a generic initramfs. You hence need to remove this package from the system:

    # yum -y remove dracut-config-generic

  • Rebuild the initramfs to incorporate the changes made to /etc/sysctl.conf into the initramfs

    # dracut -v -f

    It is necessary to rebuild the initramfs when ever there are modifications made to /etc/sysctl.conf.

  • Verify that the initramfs contains the /etc/sysctl.conf file and matches the one on the system

    # diff -u /etc/sysctl.conf <(lsinitrd -f /etc/sysctl.conf)

    In the example above, the file was included properly since the diff command didn't return anything.

  • Reboot the system

  • For additional details please refer to When is it necessary to rebuild the initramfs?

Workaround:

Workaround is to create a systemd override file (Managing Services with systemd Unit File Modify) for rpcbind.socket containing

 [Socket]
 ListenStream=/var/run/rpcbind.sock
 ListenStream=0.0.0.0:111

Red Hat recommends to rebuild the initramfs than the workaround.

Root Cause

  • This problem occurs when omitting the step to update the initramfs per How do I disable or enable the IPv6 protocol in Red Hat Enterprise Linux?

  • Without systemd being aware that IPv6 is disabled in the early boot process, it will attempt to make a connection on port 111 on any IPv6 address, [::]:111, due to updates made to rpcbind per BZ 1359592.

    # rpm -q rpcbind --changelog | head
* Tue Aug 02 2016 Steve Dickson <steved@redhat.com> - 0.2.0-38
- Removing the braces from the ${RPCBIND_ARGS} in rpcbind.service (bz 1362232)

* Fri Jul 29 2016 Steve Dickson <steved@redhat.com> - 0.2.0-37
- Make sure rpcbind.socket listens for remote IPv6 connections (bz 1359592)

* Thu Jul 21 2016 Steve Dickson <steved@redhat.com> - 0.2.0-36
- Added upstream debugging (bz 1358890)

* Sat Apr 09 2016 Steve Dickson <steved@redhat.com> - 0.2.0-35

  • rpcbind.socket service file for rpcbind-0.2.0-38.el7.x86_64 released with Red Hat Enterprise Linux Server release 7.3 and later

    # systemctl cat rpcbind.socket
# /usr/lib/systemd/system/rpcbind.socket
[Unit]
Description=RPCbind Server Activation Socket

[Socket]
ListenStream=/var/run/rpcbind.sock
ListenStream=[::]:111
ListenStream=0.0.0.0:111
BindIPv6Only=ipv6-only

[Install]
WantedBy=sockets.target

  • rpcbind.socket service file for rpcbind-0.2.0-32.el7.x86_64 released with Red Hat Enterprise Linux Server release 7.2

    # systemctl cat rpcbind.socket
# /usr/lib/systemd/system/rpcbind.socket
[Unit]
Description=RPCbind Server Activation Socket

[Socket]
ListenStream=/var/run/rpcbind.sock

[Install]
WantedBy=sockets.target

Diagnostic Steps

  • rpcbind.socket status before disabling IPv6

    # systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since [...] ago
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: Listening on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket

  • After adding net.ipv6.conf.all.disable_ipv6 = 1 to /etc/sysctl.conf and issuing rebooting

    # systemctl status rpcbind.socket
● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: failed (Result: resources)
   Listen: /var/run/rpcbind.sock (Stream)
           [::]:111 (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.
[...] systemd[1]: Unit rpcbind.socket entered failed state.
[...] systemd[1]: Starting RPCbind Server Activation Socket.
[...] systemd[1]: rpcbind.socket failed to listen on sockets: Address family not supported by protocol
[...] systemd[1]: Failed to listen on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket.
...

  • After rebuilding the initramfs and rebooting

    ● rpcbind.socket - RPCbind Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/rpcbind.socket; enabled; vendor preset: enabled)
   Active: active (listening) since [...] ago
   Listen: /var/run/rpcbind.sock (Stream)
           0.0.0.0:111 (Stream)

[...] systemd[1]: Listening on RPCbind Server Activation Socket.
[...] systemd[1]: Starting RPCbind Server Activation Socket.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments