Jobs remain Running in Ansible Automation Platform after execution node reboot

Solution Verified - Updated -

Issue

  • After an execution node reboot, Automation Controller jobs remain in running status with no finish time. Lab example (job id 12):

    "status": "running",
    "finished": null,
    "execution_node": "10.0.x.x",
    "work_unit_id": "100108119KoOuCIUb",
    "job_explanation": ""
    

    Note: /api/controller/v2/jobs/<job-id>/

  • The execution node returns to ready and continues to report consumed capacity for the stuck job. Lab example:

    "node_type": "execution",
    "node_state": "ready",
    "jobs_running": 1,
    "consumed_capacity": 2,
    "errors": ""
    

    Note: /api/controller/v2/instances/?hostname=<execution-node>

  • On the execution node, no ansible-runner / command-runner worker process remains for the job (ps shows no matches), but Receptor still stores the work unit as Running with a stale PID. Lab example from the Receptor status file:

    {"State":1,"Detail":"Running: PID 23","StdoutSize":4733,"WorkType":"ansible-runner","ExtraData":{"Pid":12,"Params":"worker --private-data-dir=/home/cloud-user/aap/controller/data/job_execution/awx_12_dg33gsth --delete"}}
    

    Note: /var/lib/receptor/<execution-node>/<work-unit-id>/status

  • Jobs do not automatically fail, time out, or get reaped after the reboot. Worker capacity stays consumed until an administrator cancels the job and/or cleans up job execution data on the node.

Environment

  • Red Hat Ansible Automation Platform 2.7
  • Containerized deployment with Receptor mesh and dedicated execution nodes
  • Trigger: reboot or OS patching of an execution node while jobs are running; node recovers quickly enough that it is not treated as permanently lost

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content