The first step to check whether the Core is running correctly is to check whether the Nagios dashboard is reporting any issues, see how to access Nagios. You will find more information on how to troubleshoot any issues discovered by Nagios in the Nagios troubleshooting information.

To see the logging output of individual Core components, you must configure centralized logging in your OpenShift cluster. See Centralized Logging for Core Components for a detailed procedure.

The section Identifying Issues in a Core provides guidance on discovering Core failures by searching and filtering its logging output.

The section Identifying Issues in an MBaaS provides guidance in discovering if the issues you are facing are originating in the MBaaS.

This issue is caused due to Persistent Volume Claims being added to the list of volumes to preserve rather than the actual name of the Persistent Volume associated with the Persistent Volume Claim.

As a result, the periodic cleanup process would unmount the volume if a pod utilizing the Persistent Volume Claim had not yet entered running state.

The issue is logged in Bugzilla and has been marked as closed so should be resolved in a future release of Kubernetes.

oc get po
NAME                   READY     STATUS        RESTARTS   AGE
fh-messaging-1-gzlw6   0/1       Terminating   0          1h
mongodb-1-1-2pqbq      0/1       Terminating   0          1h
mongodb-initiator      1/1       Terminating  3 0          1h

Solution is to force delete the pods:

$ oc delete pod fh-messaging-1-gzlw6 --grace-period=0
pod "fh-messaging-1-gzlw6" deleted
$ oc delete pod mongodb-1-1-2pqbq --grace-period=0
pod "mongodb-1-1-2pqbq" deleted
$ oc delete pod mongodb-initiator --grace-period=0
pod "mongodb-initiator" deleted

1m   3s  8  mongodb-2-1-qlgwl Pod  FailedMount  {kubelet 10.10.0.99} Unable to mount volumes for pod "mongodb-2-1-qlgwl_qe-3node-rhmap4(f7898df8-41ce-11e6-9064-0abb8905d551)": unsupported volume type
1m   3s  8  mongodb-2-1-qlgwl Pod  FailedSync   {kubelet 10.10.0.99} Error syncing pod, skipping: unsupported volume type

The problem here is the PVC is being bound to an already bound Persistent Volume (in another namespace)

$ oc get pvc mongodb-claim-2 -n qe-3node-rhmap4
mongodb-claim-2   <none>    Bound     pveleven   50Gi       RWO           3h

$ oc get pv pveleven
NAME       CAPACITY   ACCESSMODES   STATUS    CLAIM                                 REASON    AGE
pveleven   50Gi       RWO           Bound     qe-3node-rhmap410-1/mongodb-claim-2             4d

Notice the CLAIM is a different namespace i.e qe-3node-rhmap410-1, not qe-3node-rhmap4. This is usually the result of a project being created with a name that already existed, but was deleted. When deleting a project in OpenShift, it doesn’t do a hard delete straight away. Rather, it marks a bunch of resources for deletion. This can cause unusual behaviour (such as this) if some resources still exist when the new project with the same name is created, and kubernetes thinks the resources marked for deletion actually belong to the new project.

The solution in this case was to remove the PVC, then recreate it, allowing it to Bind to a new PV.

oc export pvc mongodb-claim-2 -o yaml > mongodb-claim-2.yaml
oc delete pvc mongodb-claim-2
oc create -f ./mongodb-claim-2.yaml

oc scale --replicas=0 dc fh-supercore && oc scale --replicas=1 dc fh-supercore

If all else fails, you can contact Red Hat Support for more assistance. Including a fh-system-dump-tool report will help the support team arrive at a solution more rapidly.

The fh-system-dump-tool will search your installation for frequently found issues and also create a tar.gz archive containing all the recent logs and errors it can find. More information on this can be found here.

The first step to check whether an MBaaS is running correctly is to see the output of its health endpoint. The HTTP health endpoint in the MBaaS reports the health status of the MBaaS and of its individual components.

From the command line, run the following command:

curl https://<MBaaS URL>/sys/info/health

To find the MBaaS URL:

If the MBaaS is running correctly without any errors, the output of the command should be similar to the following, showing a "test_status": "ok" for each component:

{
 "status": "ok",
 "summary": "No issues to report. All tests passed without error",
 "details": [
  {
   "description": "Check fh-statsd running",
   "test_status": "ok",
   "result": {
    "id": "fh-statsd",
    "status": "OK",
    "error": null
   },
   "runtime": 6
  },
...
}

If there are any errors, the output will contain error messages in the result.error object of the details array for individual components. Use this information to identify which component is failing and to get information on further steps to resolve the failure.

You can also see a HTTP 503 Service Unavailable error returned from the health endpoint. This can happen in several situations:

Alternatively, you can see the result of this health check in the Studio. Navigate to the Admin > MBaaS Targets section, select your MBaaS target, and click Check the MBaaS Status.

If there is an error, you are presented with a screen showing the same information as described above. Use the provided links to OpenShift Web Console and the associated MBaaS Project in OpenShift to help with debugging of the issue on the OpenShift side.

To see the logging output of individual MBaaS components, you must configure centralized logging in your OpenShift cluster. See Enabling Centralized Logging for a detailed procedure.

The section Identifying Issues in an MBaaS provides guidance on discovering MBaaS failures by searching and filtering its logging output.

A MongoDB replica may get terminated in several situations:

To learn more about replication in MongoDB, see Replication in the official MongoDB documentation.

If the mongodb-service is not responding after the installation of the MBaaS, it is possible that some of the MongoDB replica set members failed to start up. This can happen due to a combination of the following factors:

The primary indicator of this situation is the mongodb-initiator pod not reaching the Completed status.

Run the following command to see the status of mongodb-initiator:

oc get pod mongodb-initiator

NAME                READY     STATUS      RESTARTS   AGE
mongodb-initiator   1/1       Running     0          5d

If the status is any other than Completed, the MongoDB replica set is not created properly. If mongodb-initiator stays in this state too long, it may be a signal that one of the MongoDB pods has failed to start. To confirm whether this is the case, check logs of mongodb-initiator using the following command:

oc logs mongodb-initiator

=> Waiting for 3 MongoDB endpoints ...mongodb-1
mongodb-2
mongodb-3
=> Waiting for all endpoints to accept connections...

If the above message is the last one in the output, it signifies that some of the MongoDB pods are not responding.

Check the event log by running the following command:

oc get ev

If the output contains a message similar to the following, you should continue with the below procedure to clean up the persistent volumes:

  FailedMount   {kubelet ip-10-0-0-100.example.internal}   Unable to mount volumes for pod "mongodb-1-1-example-mbaas": Mount failed: exit status 32

The following procedure will guide you through the process of deleting contents of existing persistent volumes, creating new persistent volumes, and re-creating persistent volume claims.

To verify the replica set state as seen by each member, run the following command in the shell of a user logged in to OpenShift with access to the MBaaS project:

for i in `oc get po -a | grep -e "mongodb-[0-9]\+"  | awk '{print $1}'`; do
  echo "## ${i} ##"
  echo mongo admin -u admin -p \${MONGODB_ADMIN_PASSWORD} --eval "printjson\(rs.status\(\)\)" |  oc rsh --shell='/bin/bash' $i
done

For a fully consistent replica set, the output for each member would contain a members object listing details about each member. If the output resembles the following, containing the "ok" : 0 value for some members, proceed to the fix in order to make the replica set consistent.

## mongodb-1-1-8syid ##
MongoDB shell version: 2.4.9
connecting to: admin
{
    "startupStatus" : 1,
    "ok" : 0,
    "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}
## mongodb-2-1-m6ao1 ##
MongoDB shell version: 2.4.9
connecting to: admin
{
    "startupStatus" : 1,
    "ok" : 0,
    "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}
## mongodb-3-2-e0a11 ##
MongoDB shell version: 2.4.9
connecting to: admin
{
    "startupStatus" : 1,
    "ok" : 0,
    "errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}

Read more about the OpenShift Scheduler in the OpenShift documentation.

To verify that this is the problem, run the following command to list the event log:

oc get ev

If the output contains one of the following messages, you are most likely facing this problem – the nodes in your OpenShift cluster don’t fulfill some of the requirements.

Alternatively, it is also possible to correct the problem from the other side — change the deployment configurations in the MBaaS OpenShift template to match the setup of your OpenShift cluster.

To list all deployment configurations, run the following command:

oc get dc

NAME           TRIGGERS       LATEST
fh-mbaas       ConfigChange   1
fh-messaging   ConfigChange   1
fh-metrics     ConfigChange   1
fh-statsd      ConfigChange   1
mongodb-1      ConfigChange   1
mongodb-2      ConfigChange   1
mongodb-3      ConfigChange   1

To edit a deployment configuration, use the oc edit dc <deployment> command. For example, to edit the configuration of the fh-mbaas deployment, run the following command:

oc edit dc fh-mbaas

You can modify system resource requirements in the resources sections.

...
resources:
  limits:
    cpu: 800m
    memory: 800Mi
  requests:
    cpu: 200m
    memory: 200Mi
...

Changing a deployment configuration triggers a deployment operation.

If you changed the deployment configuration of any MBaaS component, the cluster should now be fully operational, with a potential limitation to quality of service.

To overcome this, create a .npmrc file in the root directory of your Cloud App. Use this file to specify the proxy location and possibly the npm registry location. For more information, see the npm configuration documentation. For example, to use a http(s) proxy and registry for company.com, you might create the following .npmrc file:

proxy=http://proxy.company.com:8080/
https-proxy=http://proxy.company.com:8080
registry=http://registry.company.com/

To validate this issue:

---
https://*.<domain>
---

where <domain> is your domain or subdomain, for example, https://*.openshift.example.com.

---
 "status": "ok",
 "summary": "No issues to report. All tests passed without error.",
 "details": [
---

Once the initial creation of the MongoDB ReplicaSet​​ in the rhmap-installer 3-node-mbaas playbook is complete, the MongoDB-Initiator Pod is deleted. Should the playbook now fail, it is not possible to re-run the playbook in an attempt to complete it - the playbook is not idempotent. The playbook will stall at the point of checking the MongoDB Initiator Pod has successfully completed.

To work-around this:

In the event of an error exception in the installation or upgrade of RHMAP using the Ansible installer, use the following steps