reboot in remote execution job on RHEL6 clients cause job to fail

Latest response

I have an issue on RHEL6 hosts with all jobs that do a reboot in a post script.
Seems like the job task is waiting for all spawned processes to exit completely before it exits the SSH session.

If I do a reboot it means the job with fail and output:

 231:Broadcast message from someone@somewhere
 232:(/dev/pts/0) at 5:04 ...
 233:
 234:The system is going down for reboot NOW!
 235:Failed to refresh the connector
 236:Net::SSH::Disconnect connection closed by remote host
 237:Exit status: EXCEPTION

Same issue if a do a "/sbin/shutdown -r 1" or a "nohup /sbin/shutdown -r 1 &".
This problem does not occurr on RHEL7 hosts.

Any idea how to do a reboot without causing the job to fail?

Responses

Are you able to provide more information about the job? i.e. is it a script?

Sure, here is a preview of the job template for a RHEL6 host.

# Helper function that exits with a particular message and code.
#
# Usage:
#   exit_with_message "Could not do a thing" 2
#
function exit_with_message() {
  echo "${1}, exiting..."
  exit $2
}

repos="rhel-6-server-rpms, rhel-6-server-satellite-tools-6.*-rpms, rhel-server-rhscl-6-rpms, rhel-6-server-optional-rpms, PM_EPEL_epel-6-x86_64, PM_VMware_rhel-6-vmware-tools-x86_64-rpms, PM_EMC_Networker_emc-networker-*-x86_64"

echo "Updating packages in repos: $repos"
today=$(date +%Y%m%d)
output=/var/log/yum-update.$today.log
echo "Local logfile: $output"
yum -y --disablerepo=* --enablerepo="$repos" update | tee $output
RETVAL=${PIPESTATUS[0]}
[ $RETVAL -eq 0 ] || exit_with_message "YUM update failed" $RETVAL

if grep -qi "no packages marked for update" $output ; then
  echo "Will not reboot since no packages marked for update ..."
else
  echo "Reboot in 1 minute ..."
  /sbin/shutdown -r 1
  RETVAL=$?
  [ $RETVAL -eq 0 ] || exit_with_message "Reboot failed" $RETVAL
fi

and here is output from the job.

 1:
Updating packages in repos: rhel-6-server-rpms, rhel-6-server-satellite-tools-6.*-rpms, rhel-server-rhscl-6-rpms, rhel-6-server-optional-rpms, PM_EPEL_epel-6-x86_64, PM_VMware_rhel-6-vmware-tools-x86_64-rpms, PM_EMC_Networker_emc-networker-*-x86_64
   2:
Local logfile: /var/log/yum-update.20170306.log
   3:
Loaded plugins: package_upload, product-id, search-disabled-repos, security,
   4:
: subscription-manager, versionlock
   5:
Setting up Update Process
Resolving Dependencies
   7:
--> Running transaction check
   8:
---> Package vmware-tools-core.x86_64 0:10.0.6-1.el6 will be updated
   9:
---> Package vmware-tools-core.x86_64 0:10.0.9-1.el6 will be an update
  10:
---> Package vmware-tools-esx-nox.x86_64 0:10.0.6-1.el6 will be updated
  11:
---> Package vmware-tools-esx-nox.x86_64 0:10.0.9-1.el6 will be an update
  12:
---> Package vmware-tools-foundation.x86_64 0:10.0.6-1.el6 will be updated
SNIP
221:
vmware-tools-services.x86_64 0:10.0.9-1.el6
 222:
vmware-tools-vgauth.x86_64 0:10.0.9-1.el6
 223:
Complete!
 224:
Reboot in 1 minute ...
 225:

 226:
Broadcast message from satellite-exec@myhost
 227:
(/dev/pts/0) at 5:03 ...
 228:
 229:
The system is going down for reboot in 1 minute!
 230:

 231:
Broadcast message from satellite-exec@myhost
 232:
(/dev/pts/0) at 5:04 ...
 233:
 234:
The system is going down for reboot NOW!
 235:
Failed to refresh the connector
 236:
Net::SSH::Disconnect connection closed by remote host
 237:
Exit status: EXCEPTION

All RHEL7 hosts exits the SSH session (row 226) with status OK but RHEL6 hosts waits as you can see and the SSH session exits with a disconnect error.

No ideas...?

Have you considered scheduling a job using at instead of running the shutdown -r 1 command?

at now +1 minutes <<< "/sbin/shutdown -r 1"

When running the script with at command it completes successfully.

Considered something like that but was hoping for a solution. But I will try that as a workaround until something better comes up, thanks for the suggestion.

I might be late, fyi, this one works for me :

nohup shutdown -r +2 &
exit_with_message "Your message here" 0
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.