8.5.4.2. Supported Watchman Plug-ins

Plug-ins are used to expand the events and conditions on which the Watchman tool takes action. These plug-ins are located in the /etc/openshift/watchman/plugins.d directory, and are outlined in the following table.
The Watchman tool automatically loads any plug-in whose file is added to the /etc/openshift/watchman/plugins.d directory. To disable a plug-in, move the plug-in file from the plugins.d directory and place it into an unused directory for backup. Ensure to restart the Watchman tool any time a change to the plugins.d directory is made: 
# service openshift-watchman restart

Table 8.5. Supported Watchman Plug-ins

Watchman Plug-in Name Plug-in Filename Function
Syslog syslog_plugin.rb This searches the /var/log/messages file for any messages logged by cgroups when a gear is destroyed and restarts the gear if required.
JBoss jboss_plugin.rb This searches JBoss cartridge server.log files for out-of-memory exceptions and restarts gears if required.
Gear State gear_state_plugin.rb This compares the last state change commanded by the user against the current status of the gear in order to find the best use for resources. For example, this plug-in kills any processes running on a stopped gear, and restarts a started gear if it has no running processes.
Throttler throttler_plugin.rb This uses cgroups to monitor CPU usage and restricts usage if required.
Metrics metrics_plugin.rb This gathers and publishes gear-level metrics such as cgroups data for all gears on a node host at a configurable interval.
OOM oom_plugin.rb Available starting with OpenShift Enterprise 2.1.4, this monitors for gears under out-of-memory (OOM) state, attempts to resolve problems, and restarts gears if required.
Enabling the Watchman Metrics Plug-in

As well as adding it to the plugins.d directory, as outlined above, enabling the Metrics plug-in requires an extra step. Edit the /etc/openshift/node.conf file and ensure the following line is uncommented to enable the Metrics plug-in: 

WATCHMAN_METRICS_ENABLED=true
Restart the Watchman service for the changes to take effect

The Metrics plug-in logs its data to /var/log/openshift/node/platform.log by default, which is the node platform log file. However, if you have Syslog enabled, log data is sent to the syslog file with type=metric in each Metrics logline.

Example 8.2. Logged Metrics Data

Jun 10 16:25:39 vm openshift-platform[29398]: type=metric appName=php6 gear=53961099e659c55b08000102 app=53961099e659c55b08000102 ns=demo quota.blocks.used=988 quota.blocks.limit=1048576 quota.files.used=229 quota.files.limit=80000
Configuring the Throttler Plug-in

To configure the throttler plug-in, edit the following parameters in the /etc/openshift/resource_limits.conf file on a node host:

Example 8.3. Throttler Plug-in Configuration Parameters

[cg_template_throttled]
cpu_shares=128
cpu_cfs_quota_us=100000
apply_period=120
apply_percent=30
restore_percent=70

The throttler plug-in works with cgroups. It watches for gears that are using an excessive amount of CPU time, and when the gear's CPU time is being used more than the amount defined by the apply_percent parameter the gear is placed into a 'throttled' cgroup quota. The throttler plug-in continues to watch the gear until it is using the amount of CPU time defined by the restore_percent parameter or less. When the amount is stabilized, the gear is placed back into the default cgroup limit.
The cpu_shares and cpu_cfs_quota parameters define the throttle cgroup templates to apply to any throttled gears. The apply_period parameter defines how long a gear is throttled before it is restored into the default cgroup limit. For example, using the default parameters, a gear is throttled once it uses over 30 percent of a gear's CPU usage for 120 seconds, and a throttled gear is unthrottled once it is using less than 70 percent of the throttled CPU usage for 120 seconds.