11.2. Testing Clock Skew

A frequently encountered problem is clock skew. Every MCollective request includes a time stamp, provided by the clock on the sending host. If the clock of a sender is substantially behind that of a recipient, the recipient drops the message. Consequently, a host will not appear in the output from the oo-mco ping command if its clock is substantially behind.

Inspect the /var/log/openshift/node/ruby193-mcollective.log file on node hosts to verify:

W, [2012-09-28T11:32:26.249636 #11711]  WARN -- : runner.rb:62:in `run' Message 236aed5ad9e9967eb1447d49392e76d8 from uid=0@broker.example.com created at 1348845978 is 368 seconds old, TTL is 60

The above message indicates that the current host received a message that was 368 seconds old, and it was discarded because its time-to-live (TTL), which is the duration for which it is considered relevant, is only 60 seconds. You can also run the date command on the different hosts and compare the output across those hosts to verify that the clocks are synchronized.

The recommended solution is to configure NTP, as described in a previous section. Alternatively, see Section 5.3, “Configuring Time Synchronization” for information on how to set the time manually.