Jobs remain Running in Ansible Automation Platform after execution node reboot
Issue
-
After an execution node reboot, Automation Controller jobs remain in
runningstatus with no finish time. Lab example (job id 12):"status": "running", "finished": null, "execution_node": "10.0.x.x", "work_unit_id": "100108119KoOuCIUb", "job_explanation": ""Note:
/api/controller/v2/jobs/<job-id>/ -
The execution node returns to
readyand continues to report consumed capacity for the stuck job. Lab example:"node_type": "execution", "node_state": "ready", "jobs_running": 1, "consumed_capacity": 2, "errors": ""Note:
/api/controller/v2/instances/?hostname=<execution-node> -
On the execution node, no ansible-runner / command-runner worker process remains for the job (
psshows no matches), but Receptor still stores the work unit as Running with a stale PID. Lab example from the Receptor status file:{"State":1,"Detail":"Running: PID 23","StdoutSize":4733,"WorkType":"ansible-runner","ExtraData":{"Pid":12,"Params":"worker --private-data-dir=/home/cloud-user/aap/controller/data/job_execution/awx_12_dg33gsth --delete"}}Note:
/var/lib/receptor/<execution-node>/<work-unit-id>/status -
Jobs do not automatically fail, time out, or get reaped after the reboot. Worker capacity stays consumed until an administrator cancels the job and/or cleans up job execution data on the node.
Environment
- Red Hat Ansible Automation Platform 2.7
- Containerized deployment with Receptor mesh and dedicated execution nodes
- Trigger: reboot or OS patching of an execution node while jobs are running; node recovers quickly enough that it is not treated as permanently lost
Subscriber exclusive content
A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.