Debugging haproxy logs via syslogd

This discussion is related to our other issue of not being able to access the docker registry externally (https://access.redhat.com/discussions/3487851). to diagnose that issue, we are wanting to view the actual haproxy logs. The documentation seems to state that the only way to do that is via syslogd:

https://docs.openshift.com/container-platform/3.7/admin_guide/router.html

My plan is to use a node's syslog to receive the requests. To that end, I've edited /etc/rsyslog.conf to have the following:

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

and done a:

systemctl restart rsyslog.service

on that node (10.0.9.79 in our case).

From the router pod (via a terminal from the web UI), I can then run something like:

echo '14>sourcehost message text' | nc -v -u -w 5 10.0.9.79 514

to simulate a syslog message. I had to install nc on the node via 'yum install nc' to run this from the node, but it works and I see the log message in /var/log/messages on the 10.0.9.79 node. I can also run this from the router pod:

~~~
echo '14>sourcehost message text' | nc -v -u -w 5 10.0.9.79 514

and I see those messages in the /var/log/messages on 10.0.9.79 as well.

Configuring the routers to log via syslog seems to be done via:

oc set env dc/router -n default --overwrite ROUTER_LOG_LEVEL=debug ROUTER_SYSLOG_ADDRESS=10.0.9.79

After I run that new router instances are spun up, and I see some haproxy information related to my runtime failures:

Jun 18 14:45:59 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60591 [18/Jun/2018:14:45:58.202] fe_no_sni~ openshift_default/ 1242/-1/-1/-1/1242 503 3278 - - SC-- 1/0/0/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Jun 18 14:45:59 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60591 [18/Jun/2018:14:45:58.202] public_ssl be_no_sni/fe_no_sni 1/0/1278 6766 -- 0/0/0/0/0 0/0
Jun 18 14:46:03 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60605 [18/Jun/2018:14:46:02.876] fe_no_sni~ openshift_default/ 189/-1/-1/-1/189 503 3278 - - SC-- 1/0/0/0/0 0/0 "GET /v2/ HTTP/1.1"
Jun 18 14:46:03 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60605 [18/Jun/2018:14:46:02.875] public_ssl be_no_sni/fe_no_sni 1/0/230 3475 -- 0/0/0/0/0 0/0
Jun 18 14:46:33 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60629 [18/Jun/2018:14:46:31.478] fe_no_sni~ openshift_default/ 2099/-1/-1/-1/2099 503 3278 - - SC-- 1/0/0/0/0 0/0 "GET /v2/ HTTP/1.1"
Jun 18 14:46:33 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60629 [18/Jun/2018:14:46:31.478] public_ssl be_no_sni/fe_no_sni 1/0/2139 3475 -- 0/0/0/0/0 0/0
Jun 18 14:46:43 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60635 [18/Jun/2018:14:46:33.671] fe_no_sni~ fe_no_sni/ -1/-1/-1/-1/10004 408 212 - - cR-- 1/0/0/0/0 0/0 ""
Jun 18 14:46:43 ip-10-0-9-79.ec2.internal haproxy[177]: 10.0.146.25:60635 [18/Jun/2018:14:46:33.671] public_ssl be_no_sni/fe_no_sni 1/0/10116 409 -- 0/0/0/0/0 0/0
~~~

but I am not sure how to interpret that output. Any guidance would be greatly appreciated.

Thanks,

Scott Hasse (on behalf of Dave Hannon)