Upgrade from OpenShift 3.7 to 3.9 Fails at TASK openshift_web_console with Message: Console install failed.

Issue

I am trying to upgrade my OpenShift Container Platform cluster from 3.7 to 3.9, and my installation fails with an error similar to the following:

TASK [openshift_web_console : Verify that the web console is running] *****************************
FAILED - RETRYING: Verify that the web console is running (60 retries left).
FAILED - RETRYING: Verify that the web console is running (59 retries left).
FAILED - RETRYING: Verify that the web console is running (58 retries left).
FAILED - RETRYING: Verify that the web console is running (57 retries left).
FAILED - RETRYING: Verify that the web console is running (56 retries left).
FAILED - RETRYING: Verify that the web console is running (55 retries left).
FAILED - RETRYING: Verify that the web console is running (54 retries left).
FAILED - RETRYING: Verify that the web console is running (53 retries left).
FAILED - RETRYING: Verify that the web console is running (52 retries left).
FAILED - RETRYING: Verify that the web console is running (51 retries left).
FAILED - RETRYING: Verify that the web console is running (50 retries left).
FAILED - RETRYING: Verify that the web console is running (49 retries left).
FAILED - RETRYING: Verify that the web console is running (48 retries left).
FAILED - RETRYING: Verify that the web console is running (47 retries left).
FAILED - RETRYING: Verify that the web console is running (46 retries left).
FAILED - RETRYING: Verify that the web console is running (45 retries left).
FAILED - RETRYING: Verify that the web console is running (44 retries left).
FAILED - RETRYING: Verify that the web console is running (43 retries left).
FAILED - RETRYING: Verify that the web console is running (42 retries left).
FAILED - RETRYING: Verify that the web console is running (41 retries left).
FAILED - RETRYING: Verify that the web console is running (40 retries left).
FAILED - RETRYING: Verify that the web console is running (39 retries left).
FAILED - RETRYING: Verify that the web console is running (38 retries left).
FAILED - RETRYING: Verify that the web console is running (37 retries left).
FAILED - RETRYING: Verify that the web console is running (36 retries left).
FAILED - RETRYING: Verify that the web console is running (35 retries left).
FAILED - RETRYING: Verify that the web console is running (34 retries left).
FAILED - RETRYING: Verify that the web console is running (33 retries left).
FAILED - RETRYING: Verify that the web console is running (32 retries left).
FAILED - RETRYING: Verify that the web console is running (31 retries left).
FAILED - RETRYING: Verify that the web console is running (30 retries left).
FAILED - RETRYING: Verify that the web console is running (29 retries left).
FAILED - RETRYING: Verify that the web console is running (28 retries left).
FAILED - RETRYING: Verify that the web console is running (27 retries left).
FAILED - RETRYING: Verify that the web console is running (26 retries left).
FAILED - RETRYING: Verify that the web console is running (25 retries left).
FAILED - RETRYING: Verify that the web console is running (24 retries left).
FAILED - RETRYING: Verify that the web console is running (23 retries left).
FAILED - RETRYING: Verify that the web console is running (22 retries left).
FAILED - RETRYING: Verify that the web console is running (21 retries left).
FAILED - RETRYING: Verify that the web console is running (20 retries left).
FAILED - RETRYING: Verify that the web console is running (19 retries left).
FAILED - RETRYING: Verify that the web console is running (18 retries left).
FAILED - RETRYING: Verify that the web console is running (17 retries left).
FAILED - RETRYING: Verify that the web console is running (16 retries left).
FAILED - RETRYING: Verify that the web console is running (15 retries left).
FAILED - RETRYING: Verify that the web console is running (14 retries left).
FAILED - RETRYING: Verify that the web console is running (13 retries left).
FAILED - RETRYING: Verify that the web console is running (12 retries left).
FAILED - RETRYING: Verify that the web console is running (11 retries left).
FAILED - RETRYING: Verify that the web console is running (10 retries left).
FAILED - RETRYING: Verify that the web console is running (9 retries left).
FAILED - RETRYING: Verify that the web console is running (8 retries left).
FAILED - RETRYING: Verify that the web console is running (7 retries left).
FAILED - RETRYING: Verify that the web console is running (6 retries left).
FAILED - RETRYING: Verify that the web console is running (5 retries left).
FAILED - RETRYING: Verify that the web console is running (4 retries left).
FAILED - RETRYING: Verify that the web console is running (3 retries left).
FAILED - RETRYING: Verify that the web console is running (2 retries left).
FAILED - RETRYING: Verify that the web console is running (1 retries left).
fatal: [example.cluster.lan]: FAILED! => {"attempts": 60, "changed": false, "cmd": ["curl", "-k", "https://webconsole.openshift-web-console.svc/healthz"], "delta": "0:00:01.013555", "end": "2018-04-09 17:47:24.207603", "msg": "non-zero return code", "rc": 7, "start": "2018-04-09 17:47:23.194048", "stderr": "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                 Dload  Upload   Total   Spent    Left  Speed\n\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused", "stderr_lines": ["  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current", "                                 Dload  Upload   Total   Spent    Left  Speed", "", "  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0", "  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0curl: (7) Failed connect to webconsole.openshift-web-console.svc:443; Connection refused"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [openshift_web_console : Check status in the openshift-web-console namespace] ****************
changed: [example.cluster.lan]

TASK [openshift_web_console : debug] **************************************************************
ok: [example.cluster.lan] => {
    "msg": [
        "In project openshift-web-console on server https://example.cluster.lan:8443", 
        "", 
        "svc/webconsole - 172.X.X.X:443 -> 8443", 
        "  deployment/webconsole deploys registry.access.redhat.com/openshift3/ose-web-console:v3.9.14", 
        "    deployment #1 running for about an hour - 0/1 pods", 
        "", 
        "View details with 'oc describe <resource>/<name>' or list everything with 'oc get all'."
    ]
}

TASK [openshift_web_console : Get pods in the openshift-web-console namespace] ********************
changed: [example.cluster.lan]

TASK [openshift_web_console : debug] **************************************************************
ok: [example.cluster.lan] => {
    "msg": [
        "NAME                          READY     STATUS    RESTARTS   AGE       IP        NODE", 
        "webconsole-56c6745c85-tt8dp   0/1       Pending   0          54m       <none>    <none>"
    ]
}

TASK [openshift_web_console : Get events in the openshift-web-console namespace] ******************
changed: [example.cluster.lan]

TASK [openshift_web_console : debug] **************************************************************
ok: [example.cluster.lan] => {
    "msg": [
        "LAST SEEN   FIRST SEEN   COUNT     NAME                                           KIND         SUBOBJECT   TYPE      REASON              SOURCE                  MESSAGE", 
        "14m         54m          141       webconsole-56c6745c85-tt8dp.1523dfd0cb7b26af   Pod                      Warning   FailedScheduling    default-scheduler       0/3 nodes are available: 1 NodeUnschedulable, 2 MatchNodeSelector.", 
        "12m         13m          7         webconsole-56c6745c85-tt8dp.1523e20fdb005ba1   Pod                      Warning   FailedScheduling    default-scheduler       0/3 nodes are available: 1 NodeUnschedulable, 2 MatchNodeSelector.", 
        "1m          11m          36        webconsole-56c6745c85-tt8dp.1523e224c847702c   Pod                      Warning   FailedScheduling    default-scheduler       0/3 nodes are available: 1 NodeUnschedulable, 2 MatchNodeSelector.", 
        "54m         54m          1         webconsole-56c6745c85.1523dfd0cb68adb0         ReplicaSet               Normal    SuccessfulCreate    replicaset-controller   Created pod: webconsole-56c6745c85-tt8dp", 
        "54m         54m          1         webconsole.1523dfd0c2c1bff1                    Deployment               Normal    ScalingReplicaSet   deployment-controller   Scaled up replica set webconsole-56c6745c85 to 1"
    ]
}

TASK [openshift_web_console : Get console pod logs] ***********************************************
changed: [example.cluster.lan]

TASK [openshift_web_console : debug] **************************************************************
ok: [example.cluster.lan] => {
    "msg": []
}

TASK [openshift_web_console : Remove temp directory] **********************************************
ok: [example.cluster.lan]

TASK [openshift_web_console : Report console errors] **********************************************
fatal: [example.cluster.lan]: FAILED! => {"changed": false, "msg": "Console install failed."}
    to retry, use: --limit @/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade.retry

PLAY RECAP ****************************************************************************************
localhost                  : ok=25   changed=0    unreachable=0    failed=0   
example.cluster.lan        : ok=311  changed=40   unreachable=0    failed=1   



Failure summary:


  1. Hosts:    example.cluster.lan
     Play:     Upgrade web console
     Task:     Report console errors
     Message:  Console install failed.