Issue

• Periodically rpc.statd spawns additional processes with the --no-notify flag, so many of these processes bind to existing sockets that it causes new mounts to fail when they too attempt to bind to a socket. The mount.nfs command fails with:

mount.nfs: mount(2): Input/output error
mount.nfs: mount system call failed

• When filer is being taken over, I see rpc.statd spawned off with --no-notify option.

Eg.
...
rpcuser    312  0.0  0.0  30524  2556 ?        Ss   Aug19   0:00 rpc.statd --no-notify
rpcuser    340  0.0  0.0  30524  2556 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser    646  0.0  0.0  30524  2552 ?        Ss   Aug21   0:00 rpc.statd --no-notify
rpcuser    750  0.0  0.0  30524  2556 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser    856  0.0  0.0  30524  2548 ?        Ss   Aug21   0:00 rpc.statd --no-notify
rpcuser    903  0.0  0.0  30524  2552 ?        Ss   Aug22   0:00 rpc.statd --no-notify
rpcuser    950  0.0  0.0  30524  2556 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser   1044  0.0  0.0  30524  2544 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser   1172  0.0  0.0  30524  2548 ?        Ss   Aug21   0:00 rpc.statd --no-notify
rpcuser   1182  0.0  0.0  30524  2548 ?        Ss   Aug21   0:00 rpc.statd --no-notify
rpcuser   1270  0.0  0.0  30524  2552 ?        Ss   Aug21   0:00 rpc.statd --no-notify
rpcuser   1291  0.0  0.0  30524  2552 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser   1298  0.0  0.0  30524  2556 ?        Ss   Aug22   0:00 rpc.statd --no-notify
rpcuser   1345  0.0  0.0  30524  2560 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser   1349  0.0  0.0  30524  2544 ?        Ss   Aug20   0:00 rpc.statd --no-notify
rpcuser   1402  0.0  0.0  30524  2556 ?        Ss   Aug22   0:00 rpc.statd --no-notify

Once you have many runaway rpc.statd processes, the new mount attempt will fail. (I saw many bind() calls trying to acquire new address but complain about address in use.