Project deployments based on Ansible Service Broker (ASB) get stuck
Environment
- Red Hat Openshift (OCP) 3.11
Issue
- Just after to have upgraded from 3.10 to 3.11 with an existant ASB installation the provision of new APB fails and the projects remain in the pending state for an undefined time.
Resolution
- Removing and installing the ASB again solved the problem, please take a look at the following article to get familiar with the procedure.
Diagnostic Steps
- Provisioning fails and the projects are not deleted.
- ASB service is reachable
# oc logs <asb pod>
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
time="2019-04-09T02:55:51Z" level=info msg=" PROVISIONING "
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
time="2019-04-09T02:55:51Z" level=info msg="Spec.ID: 03b69500315d9859bb9441d9f9023784"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Name: rh-mediawiki-apb"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Image: registry.redhat.io/openshift3/mediawiki-apb:v3.11.59"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Description: Mediawiki apb implementation"
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
...
time="2019-04-09T02:56:07Z" level=error msg="Provision or Update action failed - Pod [ bundle-48cfd976-35f7-44ba-91fb-d10119d73810 ] failed with exit code [2]"
time="2019-04-09T02:56:07Z" level=info msg="Destroying APB sandbox..."
time="2019-04-09T02:56:07Z" level=info msg="Successfully deleted rolebinding bundle-48cfd976-35f7-44ba-91fb-d10119d73810, namespace rh-mediawiki-apb-prov-6hqpq"
# curl -vvv --cacert /etc/origin/master/service-signer.crt https://asb.openshift-ansible-service-broker.svc:1338
* About to connect() to asb.openshift-ansible-service-broker.svc port 1338 (#0)
* Trying 192.168.122.25...
* Connected to asb.openshift-ansible-service-broker.svc (192.168.122.25) port 1338 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/origin/master/service-signer.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=asb.openshift-ansible-service-broker.svc
* start date: may 10 08:07:02 2019 GMT
* expire date: may 09 08:07:03 2021 GMT
* common name: asb.openshift-ansible-service-broker.svc
* issuer: CN=openshift-service-serving-signer@1557467649
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: asb.openshift-ansible-service-broker.svc:1338
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 10 May 2019 08:12:00 GMT
< Content-Length: 162
<
{
"paths": [
"/apis",
"/healthz",
"/healthz/ping",
"/healthz/poststarthook/generic-apiserver-start-informers",
"/metrics",
"/osb/"
]
* Connection #0 to host asb.openshift-ansible-service-broker.svc left intact
$ oc projects |grep rh-media
rh-mediawiki-apb-depr-jzl2v
rh-mediawiki-apb-depr-nxvzh
rh-mediawiki-apb-prov-4kn78
rh-mediawiki-apb-prov-j9l4z
rh-mediawiki-apb-prov-mtrxl
$ oc projects | grep rh-media | while read p; do oc export project $p -o yaml; done
Command "export" is deprecated, use the oc get --export
apiVersion: project.openshift.io/v1
kind: Project
metadata:
annotations:
openshift.io/sa.scc.mcs: s0:c12,c9
openshift.io/sa.scc.supplemental-groups: 1000150000/10000
openshift.io/sa.scc.uid-range: 1000150000/10000
creationTimestamp: null
generateName: rh-mediawiki-apb-depr-
labels:
bundle-action: deprovision
bundle-fqname: rh-mediawiki-apb
bundle-pod-name: bundle-a3fb4d94-407a-4b7c-b4d4-bafa23166d0b
spec:
finalizers:
- kubernetes
status:
phase: Active <<<<<
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.
Comments