Node execution failure is seen when manipulating applications in OpenShift Enterprise

Solution Unverified - Updated -

Environment

  • OpenShift Enterprise 1.2
  • OpenShift Enterprise 2.0

Issue

  • Node execution failure (error getting fact) or Node execution failure (error getting result from node) is seen when creating or deleting applications.

Resolution

  • Ensure that the clocks on each system are in sync. The broker, activemq, and node systems all must have the exact same time.
  • It is best to ensure that all systems are using ntpd and syncing from the same server.

  • If these servers are on a VmWare host, be sure that vmware-tools is not syncing the clock to an external source. The following vmware knowledgebase articles will prove helpful in ensuring this:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1189

Root Cause

  • When the time is out of sync on these systems, mcollective may drop messages. This is because a creating time is given to each message, as well as how many seconds until it expires (60 by default). If the message is received and the 60 seconds has passed, the message will be dropped.
  • If a node's clock is, for example, 80 seconds ahead of the broker every message will get dropped as it will appear to be over 80 seconds old.

Diagnostic Steps

  • Mcollective logs show the following:
WARN -- : runner.rb:65:in `rescue in block in run' Message b2aac99fa93a58d58df65dc2c813399b from uid=48@broker.example.com created at 1393446847 is 10741 seconds old, TTL is 60

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.