Mcollective action fails during add-gear for scalable apps

Issue

• An issue repeats frequently (but not 100%) on several manually scalable OpenShift applications that need to be deployed.

We're following the manual scaling techniques documented here:

https://www.openshift.com/blogs/manual-scaling-on-openshift-using-marker-files

• Sometimes after running an "add-gear", gears are being created and added to ~/haproxy/conf/gear-registry.db but not to ~/haproxy/conf/haproxy.cfg.
• Furthermore, we discovered that the new gears' ports weren't getting exposed via:

rhc ssh users --gears 'grep -A1 -B1 $OPENSHIFT_JBOSSEAP_IP /etc/openshift/port-proxy.cfg'

• Upon further investigation in /var/log/mcollective.log during the gear creation we're seeing the "expose-port" action fail with the following style error:

I, [2013-08-29T19:11:07.154950 #1872]  INFO -- : openshift.rb:91:in `execute_action' Executing action [expose-port] using method oo_expose_port with args [{"--with-app-uuid"=>"521b792d62bb3c525f00036d", "--with-app-name"=>"users", "--with-container-uuid"=>"521f9cb962bb3ce3e4000002", "--with-container-name"=>"521f9cb962bb3ce3e4000002", "--with-namespace"=>"igoprd", "--with-uid"=>15154, "--with-request-id"=>"21ed4a0f9cb6d6023c4310b53bdb1e93", "--cart-name"=>"jbosseap-6.0", "--component-name"=>"jbosseap-6.0", "--with-software-version"=>"6.0", "--cartridge-vendor"=>"redhat"}]
E, [2013-08-29T19:11:07.204793 #1872] ERROR -- : openshift.rb:170:in `rescue in with_container_from_args' System proxy set for 78796=>127.29.153.1:8080 failed(1): stdout: Proxy port must be an integer between 16384 and 65535
Error: Failed at 78796 127.29.153.1:8080
ERROR: Failed to process: setproxies 78796 127.29.153.1:8080
Error: Failed to update proxy.
 stderr: Proxy port must be an integer between 16384 and 65535
Error: Failed at 78796 127.29.153.1:8080
ERROR: Failed to process: setproxies 78796 127.29.153.1:8080
Error: Failed to update proxy.
: uid=15154

• In that case, it appears OSE 1.2 has picked port 78796, which is not a valid port number (because it's >65535).

• We don't have that many ports allocated, and PROXY_MIN_PORT_NUM=35531 in /etc/openshift/node.conf