/sbin/rpcbind crashes after failed mount

Solution In Progress - Updated -

Environment

  • Red Hat Enterprise Linux 6.4

Issue

RCPBIND stopped working after a failed mount request:

Dec 17 21:39:16 hostname1 rpc.statd[2584]: nsm_parse_reply: can't decode RPC reply
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for / (/): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for / (/): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for / (/): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for / (/): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for /NoExist (/NoExist): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for /NoExist (/NoExist): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for /NoExist (/NoExist): unmatched host
Dec 17 21:39:42 hostname1 rpc.mountd[6575]: refused mount request from 10.140.xxx.x for /NoExist (/NoExist): unmatched host
Dec 17 21:39:53 hostname1 abrt[17255]: Saved core dump of pid 2566 (/sbin/rpcbind) to /var/spool/abrt/ccpp-2013-12-17-21:39:53-2566 (598016 bytes)

The mounts appear to come from our Qualys server, but this runs weekly, and normally does not cause a crash.

Resolution

Root Cause

  • Due to buffer overruns in libtrpc, the rpcbind utility sometimes terminated
    unexpectedly with a segmentation fault. With this update, buffer is allocated by
    the svcauth_gss_validate() call, which avoids the buffer overruns and thus
    prevents the rpcbind crashes. (BZ#1056809)

Diagnostic Steps

  • Capture a core file and review for a stack trace similar to the following.
Core was generated by `rpcbind'.
Program terminated with signal 11, Segmentation fault.
#0  svc_vc_recv (xprt=0x7f4e07ca2850, msg=0x7fff823d1750) at svc_vc.c:613
613             if (cd->nonblock) {
(gdb) backtrace
#0  svc_vc_recv (xprt=0x7f4e07ca2850, msg=0x7fff823d1750) at svc_vc.c:613
#1  0x00007f4e067251cc in svc_getreq_common (fd=<value optimized out>) at svc.c:650
#2  0x00007f4e06725411 in svc_getreq_poll (pfdp=<value optimized out>, pollretval=1) at svc.c:761
#3  0x00007f4e06d6ab3e in my_svc_run () at rpcb_svc_com.c:1166
#4  0x00007f4e06d69998 in main (argc=<value optimized out>, argv=<value optimized out>) at rpcbind.c:257

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

6 Comments

Log in to comment
PG Community Member 56 points
2 September 2015 11:36 PM

I've seen a problem similar to this twice now on RHEL 6.4 and RHEL 6.7. Qualys scan crashes rpcbind.

I don't have a core file, but this is in my logs:

Aug 27 00:50:07 nrs-labs rpc.mountd[2425]: refused mount request from for / (/): not exported
Aug 27 00:50:07 nrs-labs rpc.mountd[2425]: refused mount request from for / (/): not exported
Aug 27 00:50:08 nrs-labs rpc.mountd[2425]: refused mount request from for /QNotExistQ (/): not exported
Aug 27 00:50:08 nrs-labs rpc.mountd[2425]: refused mount request from for /QNotExistQ (/): not exported
Aug 27 00:50:08 nrs-labs smbd[5306]: [2015/08/27 00:50:08.190966, 0] smbd/negprot.c:561(reply_negprot)
Aug 27 00:50:08 nrs-labs smbd[5306]: negprot protocols not 0-terminated
Aug 27 00:50:08 nrs-labs rpc.mountd[2425]: refused mount request from for / (/): not exported
Aug 27 00:50:08 nrs-labs rpc.mountd[2425]: refused mount request from for /QNotExistQ (/): not exported
Aug 27 00:50:09 nrs-labs kernel: svc: , port=51301: unknown version (-1 for prog 100003, nfsd)
Aug 27 00:50:11 nrs-labs xinetd[7742]: START: bpd pid=5309 from=::ffff:
Aug 27 00:50:14 nrs-labs kernel: svc: , port=41028: unknown version (-1 for prog 100003, nfsd)
Aug 27 00:50:14 nrs-labs kernel: svc:, port=41028: unknown version (-1 for prog 100227, nfsacl)
Aug 27 00:50:22 nrs-labs kernel: rpcbind[2101] general protection ip:7f76d8e36f72 sp:7ffcf63ffad0 error:0 in libc-2.12.so (deleted)[7f76d8dc1000+18a000]

BH Red Hat Pro 596 points
2 September 2015 11:57 PM

Hello Patrick,

I'd suggest opening a support case so your issue can be fully investigated.

KU Community Member 44 points
12 April 2016 10:52 AM

Hello Redhat, could you please provide any update for this issue ? Solution is in progress since 26 June 2015, any new outcomes ? Thanks. Karol Dabrowski

BH Red Hat Pro 596 points
13 April 2016 2:39 AM

Hello Karol,

This specific issue should be resolved by libtirpc-0.2.1-10.el6.x86_64.rpm which was released as part of the https://rhn.redhat.com/errata/RHBA-2014-1419.html errata.

KU Community Member 44 points
13 April 2016 7:34 AM

Hello Brad,

Thanks for quick answer, though - any newer patch fixes implemented since libtirpc-0.2.1-10.el6.x86_64 ? Entire fleet is up-to-date with this one, and still experiencing the issue ...

BH Red Hat Pro 596 points
13 April 2016 8:26 AM

Hi Karol,

Looks like libtirpc-0.2.1-11.el6.x86_64 is the latest so you could give that a try. I'd say it is unlikely, although certainly not impossible, that you are being affected by this exact issue. I'd suggest opening a support case so we can determine the exact nature of the issue and get it resolved for you.