Mcollective action fails during add-gear for scalable apps

Solution Unverified - Updated -

Issue

  • An issue repeats frequently (but not 100%) on several manually scalable OpenShift applications that need to be deployed.

We're following the manual scaling techniques documented here:

https://www.openshift.com/blogs/manual-scaling-on-openshift-using-marker-files

  • Sometimes after running an "add-gear", gears are being created and added to ~/haproxy/conf/gear-registry.db but not to ~/haproxy/conf/haproxy.cfg.
  • Furthermore, we discovered that the new gears' ports weren't getting exposed via:
rhc ssh users --gears 'grep -A1 -B1 $OPENSHIFT_JBOSSEAP_IP /etc/openshift/port-proxy.cfg'
  • Upon further investigation in /var/log/mcollective.log during the gear creation we're seeing the "expose-port" action fail with the following style error:
I, [2013-08-29T19:11:07.154950 #1872]  INFO -- : openshift.rb:91:in `execute_action' Executing action [expose-port] using method oo_expose_port with args [{"--with-app-uuid"=>"521b792d62bb3c525f00036d", "--with-app-name"=>"users", "--with-container-uuid"=>"521f9cb962bb3ce3e4000002", "--with-container-name"=>"521f9cb962bb3ce3e4000002", "--with-namespace"=>"igoprd", "--with-uid"=>15154, "--with-request-id"=>"21ed4a0f9cb6d6023c4310b53bdb1e93", "--cart-name"=>"jbosseap-6.0", "--component-name"=>"jbosseap-6.0", "--with-software-version"=>"6.0", "--cartridge-vendor"=>"redhat"}]
E, [2013-08-29T19:11:07.204793 #1872] ERROR -- : openshift.rb:170:in `rescue in with_container_from_args' System proxy set for 78796=>127.29.153.1:8080 failed(1): stdout: Proxy port must be an integer between 16384 and 65535
Error: Failed at 78796 127.29.153.1:8080
ERROR: Failed to process: setproxies 78796 127.29.153.1:8080
Error: Failed to update proxy.
 stderr: Proxy port must be an integer between 16384 and 65535
Error: Failed at 78796 127.29.153.1:8080
ERROR: Failed to process: setproxies 78796 127.29.153.1:8080
Error: Failed to update proxy.
: uid=15154
  • In that case, it appears OSE 1.2 has picked port 78796, which is not a valid port number (because it's >65535).

  • We don't have that many ports allocated, and PROXY_MIN_PORT_NUM=35531 in /etc/openshift/node.conf

Environment

  • OpenShift Enterprise 1.2

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase of over 48,000 articles and solutions.

Current Customers and Partners

Log in for full access

Log In
Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.