8.6.3. Configuring a Routing Daemon or Listener

Prerequisites:

After configuring a routing plug-in on the broker host, you must configure a routing listener, which can be a script or daemon, that listens for notifications of application life-cycle events. The listener must program your chosen external routing solution using the notifications to dynamically route traffic to the gears.
If you are using the sample routing plug-in described in Section 8.6.2, “Configuring the Sample Routing Plug-In”, starting in OpenShift Enterprise 2.2 you can install and configure a sample routing daemon to listen to the event notifications published on ActiveMQ by the sample routing plug-in. The routing daemon supports integration with an external nginx or Nginx Plus® routing back end. Starting in OpenShift Enterprise 2.2.4, the routing daemon also supports integration with an external F5 BIG-IP LTM® (Local Traffic Manager™) version 11.6.0 routing back end. Otherwise, you must develop your own routing daemon or listener for your chosen routing solution.
Both options, configuring the sample daemon or developing your own listener, are detailed below.
Configuring the Sample Routing Daemon

The following procedure assumes that you have already set up nginx, Nginx Plus®, or LTM® as a routing back end as described in Section 8.6.1, “Selecting an External Routing Solution”.

Procedure 8.19. To Install and Configure the Sample Routing Daemon:

  1. The sample routing daemon is provided by the rubygem-openshift-origin-routing-daemon package. The host you are installing the routing daemon on must have the Red Hat OpenShift Enterprise 2.2 Infrastructure channel enabled to access the package. See Section 7.1, “Configuring Broker Host Entitlements” for more information.
    For nginx or Nginx Plus® usage, because the routing daemon directly manages the nginx configuration files, you must install the package on the same host where nginx or Nginx Plus® is running. Nginx Plus® offers features such as a REST API and clustering, but the current version of the routing daemon must still be run on the same host.
    For LTM® usage, you must install the package on a Red Hat Enterprise Linux 6 host that is separate from the host where LTM® is running. This is because the daemon manages LTM® using a SOAP or REST interface.
    Install the rubygem-openshift-origin-routing-daemon package on the appropriate host: 
    # yum install rubygem-openshift-origin-routing-daemon
  2. Edit the /etc/openshift/routing-daemon.conf file and set the ACTIVEMQ_* parameters to the appropriate host address, credentials, and ActiveMQ topic or queue destination: 
    ACTIVEMQ_HOST=broker.example.com
ACTIVEMQ_USER=routinginfo
ACTIVEMQ_PASSWORD=routinginfopasswd
ACTIVEMQ_PORT=61613
ACTIVEMQ_DESTINATION=/topic/routinginfo
    In OpenShift Enterprise 2.1.2 and later, you can set the ACTIVEMQ_HOST parameter as a comma-separated list of host:port pairs if you are using multiple ActiveMQ brokers: 
    ACTIVEMQ_HOST='192.168.59.163:61613,192.168.59.147:61613'
  3. If you optionally enabled SSL connections per ActiveMQ host in the routing plug-in, set the plugin.activemq* parameters in this file to the same values used in the /opt/rh/ruby193/root/etc/mcollective/client.cfg file on the broker: 
    plugin.activemq.pool.size = 1
plugin.activemq.pool.1.host = activemq.example.com
plugin.activemq.pool.1.port = 61614
plugin.activemq.pool.1.ssl = true
plugin.activemq.pool.1.ssl.ca = /etc/keys/activemq.example.com.crt
plugin.activemq.pool.1.ssl.key = /etc/keys/activemq.example.com.key
plugin.activemq.pool.1.ssl.cert = /etc/keys/activemq.example.com.crt
    If you have multiple pools, ensure that plugin.activemq.pool.size is set appropriately and create unique blocks for each pool: 
    plugin.activemq.pool.size = 2
plugin.activemq.pool.1.host = activemq1.example.com
plugin.activemq.pool.1.port = 61614
plugin.activemq.pool.1.ssl = true
plugin.activemq.pool.1.ssl.ca = /etc/keys/activemq1.example.com.crt
plugin.activemq.pool.1.ssl.key = /etc/keys/activemq1.example.com.key
plugin.activemq.pool.1.ssl.cert = /etc/keys/activemq1.example.com.crt

plugin.activemq.pool.2.host = activemq2.example.com
plugin.activemq.pool.2.port = 61614
plugin.activemq.pool.2.ssl = true
plugin.activemq.pool.2.ssl.ca = /etc/keys/activemq2.example.com.crt
plugin.activemq.pool.2.ssl.key = /etc/keys/activemq2.example.com.key
plugin.activemq.pool.2.ssl.cert = /etc/keys/activemq2.example.com.crt
    The files set in the *ssl.ca, *ssl.key, and *ssl.cert parameters must be copied from the ActiveMQ broker or brokers and placed locally for the routing daemon to use.
    Note that while setting the plugin.activemq* parameters overrides the ACTIVEMQ_HOST and ACTIVEMQ_PORT parameters in this file, the ACTIVEMQ_USERNAME and ACTIVEMQ_PASSWORD parameters in this file are still used by the routing daemon and must be set.
  4. Set the CLOUD_DOMAIN parameter to the domain you are using: 
    CLOUD_DOMAIN=example.com
  5. To use a different prefix in URLs for high-availability applications, you can modify the HA_DNS_PREFIX parameter: 
    HA_DNS_PREFIX="ha-"
    This parameter and the HA_DNS_PREFIX parameter in the /etc/openshift/broker.conf file, covered in Section 8.6.4, “Enabling Support for High-Availability Applications” , must be set to the same value.
  6. If you are using nginx or Nginx Plus®, set the LOAD_BALANCER parameter to the nginx module: 
    LOAD_BALANCER=nginx
    If you are using LTM®, set the LOAD_BALANCER parameter to the f5 module: 
    LOAD_BALANCER=f5
    Ensure that only one LOAD_BALANCER line is uncommented and enabled in the file.
  7. If you are using nginx or Nginx Plus®, set the appropriate values for the following nginx module parameters if they differ from the defaults: 
    NGINX_CONFDIR=/opt/rh/nginx16/root/etc/nginx/conf.d
NGINX_SERVICE=nginx16-nginx
    If you are using Nginx Plus®, you can uncomment and set the following parameters to enable health checking. This enables active health checking and takes servers out of the upstream pool without having a client request initiate the check. 
    NGINX_PLUS=true
NGINX_PLUS_HEALTH_CHECK_INTERVAL=2s
NGINX_PLUS_HEALTH_CHECK_FAILS=1
NGINX_PLUS_HEALTH_CHECK_PASSES=5
NGINX_PLUS_HEALTH_CHECK_URI=/
NGINX_PLUS_HEALTH_CHECK_MATCH_STATUS=200
NGINX_PLUS_HEALTH_CHECK_SHARED_MEMORY=64k
  8. If you are using LTM®, set the appropriate values for the following parameters to match your LTM® configuration: 
    BIGIP_HOST=127.0.0.1
BIGIP_USERNAME=admin
BIGIP_PASSWORD=passwd
BIGIP_SSHKEY=/etc/openshift/bigip.key
    Set the following parameters to match the LTM® virtual server names you created: 
    VIRTUAL_SERVER=ose-vserver
VIRTUAL_HTTPS_SERVER=https-ose-vserver
    Also set the MONITOR_NAME parameter to match your LTM® configuration: 
    MONITOR_NAME=monitor_name
    For the lbaas module, set the appropriate values for the following parameters to match your LBaaS configuration: 
    LBAAS_HOST=127.0.0.1
LBAAS_TENANT=openshift
LBAAS_TIMEOUT=300
LBAAS_OPEN_TIMEOUT=300
LBAAS_KEYSTONE_HOST=10.0.0.1
LBAAS_KEYSTONE_USERNAME=user
LBAAS_KEYSTONE_PASSWORD=passwd
LBAAS_KEYSTONE_TENANT=lbms
  9. By default, new pools are created and named with the form pool_ose_{appname}_{namespace}_80. You can optionally override this defaults by setting appropriate value for the POOL_NAME parameter: 
    POOL_NAME=pool_ose_%a_%n_80
    If you change this value, set it to contain the following format so each application gets its own uniquely named pool:
    • %a is expanded to the name of the application.
    • %n is expanded to the application's namespace (domain).
  10. The BIG-IP LTM back end can add an existing monitor to newly created pools. The following settings control how these monitors are created: 
    #MONITOR_NAME=monitor_ose_%a_%n
#MONITOR_PATH=/health_check.php
#MONITOR_UP_CODE=1
MONITOR_TYPE=http-ecv
#MONITOR_TYPE=https-ecv
#MONITOR_INTERVAL=10
#MONITOR_TIMEOUT=5
    Set the MONITOR_NAME parameter to the name of the monitor to use, and set the MONITOR_PATH parameter to the path name to use for the monitor. Alternatively, leave either parameter unspecified to disable the monitor functionality.
    As with the POOL_NAME and ROUTE_NAME parameters, the MONITOR_NAME and MONITOR_PATH parameters both can contain %a and %n formats, which are expanded the same way. Unlike the POOL_NAME and ROUTE_NAME parameters, however, you may or may not want to reuse the same monitor for different applications. The routing daemon automatically creates a new monitor when the format used from the MONITOR_NAME parameter expands a string that does not match the name of any existing monitor.
    Set the MONITOR_UP_CODE parameter to the code that indicates that a pool member is up, or leave it unspecified to use the default value of 1.
    MONITOR_TYPE specifies the type of probe that the external load-balancer should use to check the health status of applications. The only other recognized value for MONITOR_TYPE is https-ecv, which defines the protocol to be HTTPS. All other values for MONITOR_TYPE translate to HTTP.
    Note that ECV stands for “extended content verification", referring to the fact that the monitor makes an HTTP request and looks at the reply to verify that it is the expected response (meaning the application server is responding), as opposed to merely pinging the server to ensure it is returning an ICMP ping reply (meaning the operating system is responding).
    Set the MONITOR_INTERVAL parameter to the interval at which the monitor sends requests, or leave it unspecified to use the default value of 10.
    Set the MONITOR_TIMEOUT parameter to the monitor's timeout for its requests, or leave it unset to use the default value of 5.
    It is expected that for each pool member, the routing solution sends a GET request to the resource identified on that host by the value of the MONITOR_PATH parameter for the associated monitor, and that the host responds with the value of the MONITOR_UP_CODE parameter if the host is up or some other response if the host is not up.
  11. You can change the port that nginx or Nginx Plus® listens on for HTTP or HTTPS, if required, by setting the following parameters: 
    SSL_PORT=443
HTTP_PORT=80
    For Nginx Plus®, setting the above parameters is all that is required. For nginx 1.6 (from Red Hat Software Collections), however, you must also modify the /opt/rh/nginx16/root/etc/nginx/nginx.conf file to listen on different ports. For example for HTTP, change 80 on the following line to another port: 
    listen       80;
    In both cases (nginx 1.6 and Nginx Plus®), ensure the SSL_PORT and HTTP_PORT parameters are set to the ports you intend nginx or Nginx Plus® to listen to, and ensure your host firewall configuration allows ingress traffic on these ports.
  12. Start the routing daemon: 
    # chkconfig openshift-routing-daemon on
# service openshift-routing-daemon start
Developing a Custom Routing Listener

If you are not using the sample routing daemon, you can develop your own listener to listen to the event notifications published on ActiveMQ by the sample routing plug-in. The plug-in creates notification messages for the following events:

Table 8.3. Sample Routing Plug-in Notifications

Event Message Format Additional Details
Application created
:action => :create_application,
:app_name => app.name,
:namespace => app.domain.namespace,
:scalable => app.scalable,
:ha => app.ha,
 
Application deleted
:action => :delete_application,
:app_name => app.name,
:namespace => app.domain.namespace
:scalable => app.scalable,
:ha => app.ha,
 
Public endpoint created
:action => :add_public_endpoint,
:app_name => app.name,
:namespace => app.domain.namespace,
:gear_id => gear._id.to_s,
:public_port_name => endpoint_name,
:public_address => public_ip,
:public_port => public_port.to_i,
:protocols => protocols,
:types => types,
:mappings => mappings
Values for the protocols variable include:
  • tcp, http, https, ws, wss, tls, mongodb, mysql
Values for the types variable include:
  • load_balancer, web_framework, database, plugin, other
These variables depend on values set in the cartridge manifest.
Public endpoint deleted
:action => :remove_public_endpoint,
:app_name => app.name,
:namespace => app.domain.namespace,
:gear_id => gear._id.to_s,
:public_address => public_ip,
:public_port => public_port.to_i
 
SSL certificate added
:action => :add_ssl,
:app_name => app.name,
:namespace => app.domain.namespace,
:alias => fqdn,
:ssl => ssl_cert,
:private_key => pvt_key,
:pass_phrase => passphrase
 
SSL certificate removed
:action => :remove_ssl,
:app_name => app.name,
:namespace => app.domain.namespace,
:alias => fqdn
 
Alias added
:action => :add_alias,
:app_name => app.name,
:namespace => app.domain.namespace,
:alias => alias_str
 
Alias removed
:action => :remove_alias,
:app_name => app.name,
:namespace => app.domain.namespace,
:alias => alias_str
 

Note

The add_gear and delete_gear actions have been deprecated. Use add_public_endpoint for add_gear and remove_public_endpoint for delete_gear instead.
Follow these guidelines when developing a routing listener for the sample routing plug-in:

Routing Listener Guidelines

  1. Listen to the ActiveMQ topic routinginfo. Verify that the user credentials match those configured in the /etc/openshift/plugins.d/openshift-origin-routing-activemq.conf file of the sample routing plug-in.
  2. For each gear event, reload the routing table of the router.
    1. Use the protocols value provided with the add_public_endpoint action to tailor your routing methods.
    2. Use the types value to identify the type of endpoint.
    3. Use the mappings value to identify URL routes. Routes that are not root may require source IP or SSL certificate verifications. A common use case involves administrative consoles such as phpMyAdmin.
  3. Look for actions involving SSL certificates, such as add_ssl and remove_ssl, and decide whether to configure the router accordingly for incoming requests.
  4. Look for actions involving aliases, such as add_alias and remove_alias. Aliases must always be accommodated for during the application's life cycle.

Note

The add_public_endpoint and remove_public_endpoint actions do not correspond to the actual addition and removal of gears, but rather to the exposure and concealment of ports. One gear added to an application may result in several exposed ports, which will all result in respective add_public_endpoint notifications at the router level.

Example 8.18. Simple Routing Listener

The following listener.rb script file is an example model for a simple routing listener. This Ruby script uses Nginx as the external routing solution, and the pseudo code provided is an example only. The example handles the following tasks:
  • Look for messages with an add_public_endpoint action and a load_balancer type, then edit the router configuration file for the application.
  • Look for messages with a remove_public_endpoint action and a load_balancer type, then edit the router configuration file for the application.
  • Look for messages with a delete_application action and remove the router configuration file for the application. 
#!/usr/bin/ruby

require 'rubygems'
require 'stomp'
require 'yaml'

CONF_DIR='/etc/nginx/conf.d/'

def add_haproxy(appname, namespace, ip, port)
  scope = "#{appname}-#{namespace}"
  file = File.join(CONF_DIR, "#{scope}.conf")
  if File.exist?(file)
    `sed -i 's/upstream #{scope} {/&\\n      server #{ip}:#{port};/' #{file}`
  else
    # write a new one
    template = <<-EOF
    upstream #{scope} {
      server #{ip}:#{port};
    }
    server {
      listen 8000;
      server_name ha-#{scope}.dev.rhcloud.com;
      location / {
        proxy_pass http://#{scope};
      }
    }
EOF
    File.open(file, 'w') { |f| f.write(template) }
  end
  `nginx -s reload`
end


c = Stomp::Client.new("routinginfo", "routinginfopasswd", "localhost", 61613, true)
c.subscribe('/topic/routinginfo') { |msg|
  h = YAML.load(msg.body)
  if h[:action] == :add_public_endpoint 
    if h[:types].include? "load_balancer"
       add_haproxy(h[:app_name], h[:namespace], h[:public_address], h[:public_port])
       puts "Added routing endpoint for #{h[:app_name]}-#{h[:namespace]}"
    end
  elsif h[:action] == :remove_public_endpoint
  # script does not actually act upon the remove_public_endpoint as written
  elsif h[:action] == :delete_application
     scope = '#{h[:app_name]}-#{h[:namespace]}'
     file = File.join(CONF_DIR, "#{scope}.conf")
     if File.exist?(file)
       `rm -f #{file}`
       `nginx -s reload`
       puts "Removed configuration for #{scope}"
     end
  end
}
c.join