rhn_check failing with "up2date_client.up2dateErrors.NetworkError: Network error: Connection reset by peer"

Issue

• rhn_check runs in a script from a cron job on weekly basis
• The script runs on 300+ servers but failed on few servers with this error message:

[Sun Feb  9 06:30:02 2014] up2date logging into up2date server
[Sun Feb  9 06:30:03 2014] up2date successfully retrieved authentication token from up2date server
[Sun Feb  9 06:30:33 2014] up2date
Traceback (most recent call last):
  File "/usr/sbin/rhn_check", line 367, in ?
    cli.run()
  File "/usr/share/rhn/up2date_client/rhncli.py", line 66, in run
    sys.exit(self.main() or 0)
  File "/usr/sbin/rhn_check", line 68, in main
    if s.capabilities.hasCapability('staging_content', 1) and cfg['stagingContent'] != 0:
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 159, in __get_capabilities
    self.registration.welcome_message()
  File "/usr/share/rhn/up2date_client/rhnserver.py", line 69, in __call__
    raise up2dateErrors.NetworkError(message)
up2date_client.up2dateErrors.NetworkError: Network error: Connection reset by peer

• The script is running at the same time on all servers
• The part of the script executing rhn_check looks like this:

echo "************************************************************************"
echo ""
echo "[DEBUG] Checking RHN Connectivity"
/usr/sbin/rhn_check -v > "$PDIR"/rhn_chk.err
...