Chapter 3. Troubleshooting the RHMAP MBaaS
Overview
This document provides information on how to identify, debug, and resolve possible issues that can be encountered during installation and usage of the RHMAP MBaaS on OpenShift 3.
In Common Problems, you can see a list of resolutions for problems that might occur during installation or usage of the MBaaS.
3.1. Checking the Health Endpoint of the MBaaS
The first step to check whether an MBaaS is running correctly is to see the output of its health endpoint. The HTTP health endpoint in the MBaaS reports the health status of the MBaaS and of its individual components.
From the command line, run the following command:
curl https://<MBaaS URL>/sys/info/health
To find the MBaaS URL:
-
Navigate to the MBaaS that you want to check, by selecting
Admin, thenMBaaS Targets, and then choosing the MBaaS. -
Copy the value of the
MBaaS URLfield from the MBaaS details screen.
If the MBaaS is running correctly without any errors, the output of the command should be similar to the following, showing a "test_status": "ok" for each component:
{
"status": "ok",
"summary": "No issues to report. All tests passed without error",
"details": [
{
"description": "Check fh-statsd running",
"test_status": "ok",
"result": {
"id": "fh-statsd",
"status": "OK",
"error": null
},
"runtime": 6
},
...
}
If there are any errors, the output will contain error messages in the result.error object of the details array for individual components. Use this information to identify which component is failing and to get information on further steps to resolve the failure.
You can also see a HTTP 503 Service Unavailable error returned from the health endpoint. This can happen in several situations:
- The MBaaS hasn’t finished deploying.
- The URL of the MBaaS is not reachable on port 80. Check your network configuration.
- Provisioning of the MBaaS has failed. See the Prerequisites and Before the Installation (for manual installation) sections of the Provisioning an MBaaS in Red Hat OpenShift Enterprise 3 guide and make sure your OpenShift cluster fulfills all the listed requirements.
Alternatively, you can see the result of this health check in the Studio. Navigate to the Admin > MBaaS Targets section, select your MBaaS target, and click Check the MBaaS Status.
If there is an error, you are presented with a screen showing the same information as described above. Use the provided links to OpenShift Web Console and the associated MBaaS Project in OpenShift to help with debugging of the issue on the OpenShift side.
3.2. Analyzing the MBaaS Logs
To see the logging output of individual MBaaS components, you must configure centralized logging in your OpenShift cluster. See Enabling Centralized Logging for a detailed procedure.
The section Identifying Issues in an MBaaS provides guidance on discovering MBaaS failures by searching and filtering its logging output.
3.3. Troubleshooting Common Problems
Summary
The replica set is susceptible to down time if the replica set members configuration is not up to date with the actual set of pods. There must be at least two members active at any time, in order for an election of a primary member to happen. Without a primary member, the MongoDB service won’t perform any read or write operations.
A MongoDB replica may get terminated in several situations:
- A node hosting a MongoDB replica is terminated or evacuated.
- A re-deploy is triggered on one of the MongoDB Deployment objects in the project – manually or by a configuration change.
- One of the MongoDB deployments is scaled down to zero pods, then scaled back up to one pod.
To learn more about replication in MongoDB, see Replication in the official MongoDB documentation.
Fix
The following procedure shows you how to re-configure a MongoDB replica set into a fully operational state. You must synchronize the list of replica set members with the actual set of MongoDB pods in the cluster, and set a primary member of the replica set.
Note the MongoDB endpoints.
oc get ep | grep mongo
Make note of the list of endpoints. It is used later to set the replica set members configuration.
NAME ENDPOINTS AGE mongodb-1 10.1.2.152:27017 17h mongodb-2 10.1.4.136:27017 17h mongodb-3 10.1.5.16:27017 17h
Log in to the oldest MongoDB replica pod.
List all the MongoDB replica pods.
oc get po -l app=mongodb-3-node-replica-set
In the output, find the pod with the highest value in the
AGEfield.NAME READY STATUS RESTARTS AGE mongodb-1-1-4nsrv 1/1 Running 0 19h mongodb-2-1-j4v3x 1/1 Running 0 3h mongodb-3-2-7tezv 1/1 Running 0 1h
In this case, it is
mongodb-1-1-4nsrvwith an age of 19 hours.Log in to the pod using
oc rsh.oc rsh mongodb-1-1-4nsrv
Open a MongoDB shell on the primary member.
mongo admin -u admin -p ${MONGODB_ADMIN_PASSWORD} --host ${MONGODB_REPLICA_NAME}/localhostMongoDB shell version: 2.4.9 connecting to: rs0/localhost:27017/admin [...] Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb-user rs0:PRIMARY>List the configured members.
Run
rs.conf()in the MongoDB shell.rs0:PRIMARY> rs.conf() { "_id" : "rs0", "version" : 56239, "members" : [ { "_id" : 3, "host" : "10.1.0.2:27017" }, { "_id" : 4, "host" : "10.1.1.2:27017" }, { "_id" : 5, "host" : "10.1.6.4:27017" } ] }Ensure all hosts have either
PRIMARYorSECONDARYstatus.Run the following command. It may take several seconds to complete.
rs0:PRIMARY> rs.status().members.forEach(function(member) {print(member.name + ' :: ' + member.stateStr)})mongodb-1:27017 :: PRIMARY mongodb-2:27017 :: SECONDARY mongodb-3:27017 :: SECONDARY rs0:PRIMARY>
There must be exactly one
PRIMARYnode. All the other nodes must beSECONDARY. If a member is in aSTARTUP,STARTUP2,RECOVERING, orUNKNOWNstate, try running the above command again in a few minutes. These states may signify that the replica set is performing a startup, recovery, or other procedure potentially resulting in an operational state.
Result
After applying the fix, all three MongoDB pods will be members of the replica set. If one of the three members terminates unexpectedly, the two remaining members are enough to keep the MongoDB service fully operational.
The described situation can result from an attempt to create an MBaaS with the same name as a previously deleted one. We suggest you use unique names for every MBaaS installation.
If the mongodb-service is not responding after the installation of the MBaaS, it is possible that some of the MongoDB replica set members failed to start up. This can happen due to a combination of the following factors:
-
The most likely cause of failure in MongoDB startup is the presence of a
mongod.locklock file and journal files in the MongoDB data folder, left over from an improperly terminated MongoDB instance. - If a MongoDB pod is terminated, the associated persistent volumes transition to a Released state. When a new MongoDB pod replaces a terminated one, it may get attached to the same persistent volume which was attached to the terminated MongoDB instance, and thus get exposed to the files created by the terminated instance.
SSH access and administrator rights on the OpenShift master and the NFS server are required for the following procedure.
This procedure describes a fix only for persistent volumes backed by NFS. Refer to Configuring Persistent Storage in the official OpenShift documentation for general information on handling other volume types.
The primary indicator of this situation is the mongodb-initiator pod not reaching the Completed status.
Run the following command to see the status of mongodb-initiator:
oc get pod mongodb-initiator
NAME READY STATUS RESTARTS AGE mongodb-initiator 1/1 Running 0 5d
If the status is any other than Completed, the MongoDB replica set is not created properly. If mongodb-initiator stays in this state too long, it may be a signal that one of the MongoDB pods has failed to start. To confirm whether this is the case, check logs of mongodb-initiator using the following command:
oc logs mongodb-initiator
=> Waiting for 3 MongoDB endpoints ...mongodb-1 mongodb-2 mongodb-3 => Waiting for all endpoints to accept connections...
If the above message is the last one in the output, it signifies that some of the MongoDB pods are not responding.
Check the event log by running the following command:
oc get ev
If the output contains a message similar to the following, you should continue with the below procedure to clean up the persistent volumes:
FailedMount {kubelet ip-10-0-0-100.example.internal} Unable to mount volumes for pod "mongodb-1-1-example-mbaas": Mount failed: exit status 32The following procedure will guide you through the process of deleting contents of existing persistent volumes, creating new persistent volumes, and re-creating persistent volume claims.
Find the NFS paths.
On the OpenShift master node, execute the following command to find the paths of all persistent volumes associated with an MBaaS. Replace
<mbaas-project-name>with the name of the MBaaS project in OpenShift.list=$(oc get pv | grep <mbaas-project-name> | awk '{ print $1}'); for pv in ${list[@]} ; do path=$(oc describe pv ${pv} | grep Path: | awk '{print $2}' | tr -d '\r') echo ${path} doneExample output:
/nfs/exp222 /nfs/exp249 /nfs/exp255
Delete all contents of the found NFS paths.
Log in to the NFS server using
ssh.Execute the following command to list contents of the paths. Replace
<NFS paths>with the list of paths from the previous step, separated by spaces.for path in <NFS paths> ; do echo ${path} sudo ls -l ${path} echo " " doneExample output:
/nfs/exp222 -rw-------. 1001320000 nfsnobody admin.0 -rw-------. 1001320000 nfsnobody admin.ns -rw-------. 1001320000 nfsnobody fh-mbaas.0 -rw-------. 1001320000 nfsnobody fh-mbaas.ns -rw-------. 1001320000 nfsnobody fh-metrics.0 -rw-------. 1001320000 nfsnobody fh-metrics.ns -rw-------. 1001320000 nfsnobody fh-reporting.0 -rw-------. 1001320000 nfsnobody fh-reporting.ns drwxr-xr-x. 1001320000 nfsnobody journal -rw-------. 1001320000 nfsnobody local.0 -rw-------. 1001320000 nfsnobody local.1 -rw-------. 1001320000 nfsnobody local.ns -rwxr-xr-x. 1001320000 nfsnobody mongod.lock drwxr-xr-x. 1001320000 nfsnobody _tmp /nfs/exp249 drwxr-xr-x. 1001320000 nfsnobody journal -rw-------. 1001320000 nfsnobody local.0 -rw-------. 1001320000 nfsnobody local.ns -rwxr-xr-x. 1001320000 nfsnobody mongod.lock drwxr-xr-x. 1001320000 nfsnobody _tmp /nfs/exp255 drwxr-xr-x. 1001320000 nfsnobody journal -rw-------. 1001320000 nfsnobody local.0 -rw-------. 1001320000 nfsnobody local.ns -rwxr-xr-x. 1001320000 nfsnobody mongod.lock drwxr-xr-x. 1001320000 nfsnobody _tmp
WarningMake sure to back up all data before proceeding. The following operation may result in irrecoverable loss of data.
If the listed contents of the paths resemble the output shown above, delete all contents of the found NFS paths. Replace
<NFS paths>with the list of paths from step 1, separated by spaces.for path in <NFS paths> do if [ -z ${path+x} ] then echo "path is unset" else echo "path is set to '$path'" cd ${path} && rm -rf ./* fi doneRe-create persistent volumes.
Log in to the OpenShift master node using
ssh.Navigate to the directory which contains the YAML files that were used to create the persistent volumes.
Execute the following command to delete and re-create the persistent volumes. Replace
<mbaas-project-name>with the name of the MBaaS project in OpenShift.list=$(oc get pv | grep <mbaas-project-name> | awk '{ print $1}'); for pv in ${list}; do oc delete pv ${pv} oc create -f ${pv}.yaml doneThe persistent volumes are now re-created and in Available state.
NoteThe re-created persistent volumes will not be used by OpenShift again for the same persistent volume claims. Make sure you have at least three additional persistent volumes in Available state.
Re-create persistent volume claims for MongoDB.
Create three JSON files, with the following names:
-
mongodb-claim-1.json -
mongodb-claim-2.json -
mongodb-claim-3.json
Copy the following contents into each file. Change the
metadata.namevalue to match the name of the file without the suffix. For example, the contents for themongodb-claim-1.jsonfile are as follows:{ "kind": "PersistentVolumeClaim", "apiVersion": "v1", "metadata": { "name": "mongodb-claim-1" }, "spec": { "accessModes": ["ReadWriteOnce"], "resources": { "requests": { "storage": "50Gi" } } } }Run the following command to re-create the persistent volume claims.
for pvc in mongodb-claim-1 mongodb-claim-2 mongodb-claim-3; do oc delete pvc ${pvc} oc create -f ${pvc}.json done-
Verify that
mongodb-initiatorproceeds with initialization.Run the following command to see the logs of
mongodb-initiator.oc logs mongodb-initiator -f
After
mongodb-initiatorcompletes its work, the log output should contain the following message, indicating that the MongoDB replica set was successfully created.=> Successfully initialized replSet
Result
The MongoDB service is fully operational with all three replicas attached to their persistent volumes. The persistent volumes left in Released state from the previous installation are now in the Available state, ready for use by other persistent volume claims.
Summary
If some of the MBaaS components start to crash, this may be because they can not connect to a primary member in the MongoDB replica set. This usually indicates that the replica set configuration has become inconsistent. This can happen if a majority of the member pods get replaced and have new IP addresses. In this case, data cannot be written to or read from MongoDB replica set in the MBaaS project.
To verify the replica set state as seen by each member, run the following command in the shell of a user logged in to OpenShift with access to the MBaaS project:
for i in `oc get po -a | grep -e "mongodb-[0-9]\+" | awk '{print $1}'`; do
echo "## ${i} ##"
echo mongo admin -u admin -p \${MONGODB_ADMIN_PASSWORD} --eval "printjson\(rs.status\(\)\)" | oc rsh --shell='/bin/bash' $i
done
For a fully consistent replica set, the output for each member would contain a members object listing details about each member. If the output resembles the following, containing the "ok" : 0 value for some members, proceed to the fix in order to make the replica set consistent.
## mongodb-1-1-8syid ##
MongoDB shell version: 2.4.9
connecting to: admin
{
"startupStatus" : 1,
"ok" : 0,
"errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}
## mongodb-2-1-m6ao1 ##
MongoDB shell version: 2.4.9
connecting to: admin
{
"startupStatus" : 1,
"ok" : 0,
"errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}
## mongodb-3-2-e0a11 ##
MongoDB shell version: 2.4.9
connecting to: admin
{
"startupStatus" : 1,
"ok" : 0,
"errmsg" : "loading local.system.replset config (LOADINGCONFIG)"
}Fix
You can make the replica set consistent by forcing a re-deploy.
Note the MongoDB endpoints which are in an error status.
oc get po
Example:
```bash NAME READY STATUS RESTARTS AGE mongodb-1-1-pu0fz 1/1 Error 0 1h ```
Force a deploy of this Pod
oc deploy mongodb-1 --latest
Result
The replica starts replicating properly again and dependent MBaaS components start working again.
Summary
If some of the MBaaS components are not starting up after the installation, it may be the case that the OpenShift scheduler failed to find suitable nodes on which to schedule the pods of those MBaaS components. This means that the OpenShift cluster doesn’t contain all the nodes required by the MBaaS OpenShift template, or that those nodes don’t satisfy the requirements on system resources, node labels, and other parameters.
Read more about the OpenShift Scheduler in the OpenShift documentation.
To verify that this is the problem, run the following command to list the event log:
oc get ev
If the output contains one of the following messages, you are most likely facing this problem – the nodes in your OpenShift cluster don’t fulfill some of the requirements.
-
Failed for reason MatchNodeSelector and possibly others -
Failed for reason PodExceedsFreeCPU and possibly others
Fix
To fix this problem, configure nodes in your OpenShift cluster to match the requirements of the MBaaS OpenShift template.
Apply correct labels to nodes.
Refer to Apply Node Labels in the guide Provisioning an MBaaS in Red Hat OpenShift Enterprise 3 for details on what labels must be applied to nodes.
Make sure the OpenShift cluster has sufficient resources for the MBaaS components, Cloud Apps, and Cloud Services it runs.
Configure the machines used as OpenShift nodes to have more CPU power and internal memory available, or add more nodes to the cluster. Refer to the guide on Overcommitting and Compute Resources in the OpenShift documentation for more information on how containers use system resources.
Clean up the OpenShift instance.
Delete unused projects from the OpenShift instance.
Alternatively, it is also possible to correct the problem from the other side — change the deployment configurations in the MBaaS OpenShift template to match the setup of your OpenShift cluster.
Changing the deployment configurations may negatively impact the performance and reliability of the MBaaS. Therefore, this is not a recommended approach.
To list all deployment configurations, run the following command:
oc get dc
NAME TRIGGERS LATEST fh-mbaas ConfigChange 1 fh-messaging ConfigChange 1 fh-metrics ConfigChange 1 fh-statsd ConfigChange 1 mongodb-1 ConfigChange 1 mongodb-2 ConfigChange 1 mongodb-3 ConfigChange 1
To edit a deployment configuration, use the oc edit dc <deployment> command. For example, to edit the configuration of the fh-mbaas deployment, run the following command:
oc edit dc fh-mbaas
You can modify system resource requirements in the resources sections.
...
resources:
limits:
cpu: 800m
memory: 800Mi
requests:
cpu: 200m
memory: 200Mi
...Changing a deployment configuration triggers a deployment operation.
Result
If you changed the setup of nodes in the OpenShift cluster to match the requirements of the MBaaS OpenShift template, the MBaaS is now fully operational without any limitation to quality of service.
If you changed the deployment configuration of any MBaaS component, the cluster should now be fully operational, with a potential limitation to quality of service.
Summary
If your OpenShift installation only has outbound internet access via a proxy server, you may have issues when the node.js builder in OpenShift attempts to do an npm installation. Also, if you have a custom npm registry, you need to configure npm to avoid issues with node.js builder.
To overcome this, create a .npmrc file in the root directory of your Cloud App. Use this file to specify the proxy location and possibly the npm registry location. For more information, see the npm configuration documentation. For example, to use a http(s) proxy and registry for company.com, you might create the following .npmrc file:
proxy=http://proxy.company.com:8080/ https-proxy=http://proxy.company.com:8080 registry=http://registry.company.com/
Result
You can now deploy your app to the RHMAP MBaaS.
Summary
If you create an MBaaS target in Studio and the DNS URL does not properly form, the route in the OpenShift project is not created.
The DNS URL is set in the 'OpenShift Router DNS' field.
To validate this issue:
- Open RHMAP Studio
- Click on Admin
- Click on MBaaS Targets
- Choose your MBaaS
- Click on Check the MBaaS Status
- Click on Raw MBaaS Health Response
The output is similar to the following:
--- {"userDetail":"getaddrinfo ENOTFOUND", "systemDetail":"getaddrinfo ENOTFOUND - { [Error: getaddrinfo ENOTFOUND] code: 'ENOTFOUND', errno: 'ENOTFOUND', syscall: 'getaddrinfo' } ","code":"ENOTFOUND"} ---
Fix
To resolve this issue, create a new MBaaS using the following format for the 'OpenShift Router DNS' field:
--- https://*.<domain> ---
where <domain> is your domain or subdomain, for example, https://*.openshift.example.com.
Result
Check the MBaaS status. The output is similar to the following:
--- "status": "ok", "summary": "No issues to report. All tests passed without error.", "details": [ ---
Where did the comment section go?
Red Hat's documentation publication system recently went through an upgrade to enable speedier, more mobile-friendly content. We decided to re-evaluate our commenting platform to ensure that it meets your expectations and serves as an optimal feedback mechanism. During this redesign, we invite your input on providing feedback on Red Hat documentation via the discussion platform.