Octavia failing to create load balancers

Issue

• When trying to create a load balancer in Octavia with the following command:

openstack loadbalancer create --name testlb1 --vip-network-id external

• The load balancer enter the PENDING_CREATE state:

$ openstack loadbalancer show fa463c03-715e-434e-ada1-42ac016096ff
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-03-17T20:13:55                  |
| description         |                                      |
| flavor_id           |                                      |
| id                  | fa463c03-715e-434e-ada1-42ac016096ff |
| listeners           |                                      |
| name                | testlb1                              |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 4bb09e3a56cd451b9d260ad6c111fd96     |
| provider            | octavia                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 128.31.25.15                         |
| vip_network_id      | 71b97520-69af-4c35-8153-cdf827d96e60 |
| vip_port_id         | e49eb4d0-b604-4f60-bfb1-a1598358006d |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 3a43b38a-0a5e-40ad-bf98-27f9a57e3df5 |
+---------------------+--------------------------------------+

• It stays in that for around 30 minutes, and eventually ends up in the ERROR state:

+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-03-17T20:13:55                  |
| description         |                                      |
| flavor_id           |                                      |
| id                  | fa463c03-715e-434e-ada1-42ac016096ff |
| listeners           |                                      |
| name                | testlb1                              |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 4bb09e3a56cd451b9d260ad6c111fd96     |
| provider            | octavia                              |
| provisioning_status | ERROR                                |
| updated_at          | 2020-03-17T20:39:45                  |
| vip_address         | 10.10.10.10                         |
| vip_network_id      | 71b97520-69af-4c35-8153-cdf827d96e60 |
| vip_port_id         | e49eb4d0-b604-4f60-bfb1-a1598358006d |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 3a43b38a-0a5e-40ad-bf98-27f9a57e3df5 |
+---------------------+--------------------------------------+

• During this time, the only log messages seen in /var/log/containers/octavia/*.log on all of the controllers that mention the load balancer id are (note these are out of order because they come from two different hosts):

/var/log/containers/octavia/octavia.log:2020-03-17 16:13:57.927 27 INFO octavia.api.v2.controllers.load_balancer [req-ee0d0960-bc8b-4560-a876-a1b9df92c75a - 4bb09e3a56cd451b9d260ad6c111fd96 - default default] Sending created Load Balancer fa463c03-715e-434e-ada1-42ac016096ff to the handler

/var/log/containers/octavia/worker.log:2020-03-17 16:13:57.952 24 INFO octavia.controller.queue.endpoint [-] Creating load balancer 'fa463c03-715e-434e-ada1-42ac016096ff'...

/var/log/containers/octavia/worker.log:                          |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'fa463c03-715e-434e-ada1-42ac016096ff'}, 'provides': None}

/var/log/containers/octavia/worker.log:                                |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'fa463c03-715e-434e-ada1-42ac016096ff'}, 'provides': None}

/var/log/containers/octavia/worker.log:2020-03-17 16:39:45.422 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting Amphora allocation for the load balance

• There is no indication why creating the load balancer failed, or what it was waiting for while in the PENDING_CREATE state.

• Logging in the amphora VM and looking at amphora-agent.log, we notice the following errors:

[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Failed to send error message.