Project deployments based on Ansible Service Broker (ASB) get stuck

Solution Verified - Updated -

Environment

  • Red Hat Openshift (OCP) 3.11

Issue

  • Just after to have upgraded from 3.10 to 3.11 with an existant ASB installation the provision of new APB fails and the projects remain in the pending state for an undefined time.

Resolution

  • Removing and installing the ASB again solved the problem, please take a look at the following article to get familiar with the procedure.

Diagnostic Steps

  • Provisioning fails and the projects are not deleted.
  • ASB service is reachable
# oc logs <asb pod>
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
time="2019-04-09T02:55:51Z" level=info msg="                       PROVISIONING                         "
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
time="2019-04-09T02:55:51Z" level=info msg="Spec.ID: 03b69500315d9859bb9441d9f9023784"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Name: rh-mediawiki-apb"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Image: registry.redhat.io/openshift3/mediawiki-apb:v3.11.59"
time="2019-04-09T02:55:51Z" level=info msg="Spec.Description: Mediawiki apb implementation"
time="2019-04-09T02:55:51Z" level=info msg="============================================================"
...
time="2019-04-09T02:56:07Z" level=error msg="Provision or Update action failed - Pod [ bundle-48cfd976-35f7-44ba-91fb-d10119d73810 ] failed with exit code [2]"
time="2019-04-09T02:56:07Z" level=info msg="Destroying APB sandbox..."
time="2019-04-09T02:56:07Z" level=info msg="Successfully deleted rolebinding bundle-48cfd976-35f7-44ba-91fb-d10119d73810, namespace rh-mediawiki-apb-prov-6hqpq"

# curl -vvv --cacert /etc/origin/master/service-signer.crt https://asb.openshift-ansible-service-broker.svc:1338
* About to connect() to asb.openshift-ansible-service-broker.svc port 1338 (#0)
*   Trying 192.168.122.25...
* Connected to asb.openshift-ansible-service-broker.svc (192.168.122.25) port 1338 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/origin/master/service-signer.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*   subject: CN=asb.openshift-ansible-service-broker.svc
*   start date: may 10 08:07:02 2019 GMT
*   expire date: may 09 08:07:03 2021 GMT
*   common name: asb.openshift-ansible-service-broker.svc
*   issuer: CN=openshift-service-serving-signer@1557467649
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: asb.openshift-ansible-service-broker.svc:1338
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Fri, 10 May 2019 08:12:00 GMT
< Content-Length: 162
< 
{
  "paths": [
    "/apis",
    "/healthz",
    "/healthz/ping",
    "/healthz/poststarthook/generic-apiserver-start-informers",
    "/metrics",
    "/osb/"
  ]
* Connection #0 to host asb.openshift-ansible-service-broker.svc left intact

$ oc projects |grep rh-media
    rh-mediawiki-apb-depr-jzl2v
    rh-mediawiki-apb-depr-nxvzh
    rh-mediawiki-apb-prov-4kn78
    rh-mediawiki-apb-prov-j9l4z
    rh-mediawiki-apb-prov-mtrxl

$ oc projects | grep rh-media | while read p; do oc export project $p -o yaml; done 
Command "export" is deprecated, use the oc get --export
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c12,c9
    openshift.io/sa.scc.supplemental-groups: 1000150000/10000
    openshift.io/sa.scc.uid-range: 1000150000/10000
  creationTimestamp: null
  generateName: rh-mediawiki-apb-depr-
  labels:
    bundle-action: deprovision
    bundle-fqname: rh-mediawiki-apb
    bundle-pod-name: bundle-a3fb4d94-407a-4b7c-b4d4-bafa23166d0b
spec:
  finalizers:
  - kubernetes
status:
  phase: Active <<<<<

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments