Octavia failing to create load balancers

Solution In Progress - Updated -

Issue

  • When trying to create a load balancer in Octavia with the following command:
openstack loadbalancer create --name testlb1 --vip-network-id external
  • The load balancer enter the PENDING_CREATE state:
$ openstack loadbalancer show fa463c03-715e-434e-ada1-42ac016096ff
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-03-17T20:13:55                  |
| description         |                                      |
| flavor_id           |                                      |
| id                  | fa463c03-715e-434e-ada1-42ac016096ff |
| listeners           |                                      |
| name                | testlb1                              |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 4bb09e3a56cd451b9d260ad6c111fd96     |
| provider            | octavia                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 128.31.25.15                         |
| vip_network_id      | 71b97520-69af-4c35-8153-cdf827d96e60 |
| vip_port_id         | e49eb4d0-b604-4f60-bfb1-a1598358006d |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 3a43b38a-0a5e-40ad-bf98-27f9a57e3df5 |
+---------------------+--------------------------------------+
  • It stays in that for around 30 minutes, and eventually ends up in the ERROR state:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| created_at          | 2020-03-17T20:13:55                  |
| description         |                                      |
| flavor_id           |                                      |
| id                  | fa463c03-715e-434e-ada1-42ac016096ff |
| listeners           |                                      |
| name                | testlb1                              |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 4bb09e3a56cd451b9d260ad6c111fd96     |
| provider            | octavia                              |
| provisioning_status | ERROR                                |
| updated_at          | 2020-03-17T20:39:45                  |
| vip_address         | 10.10.10.10                         |
| vip_network_id      | 71b97520-69af-4c35-8153-cdf827d96e60 |
| vip_port_id         | e49eb4d0-b604-4f60-bfb1-a1598358006d |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 3a43b38a-0a5e-40ad-bf98-27f9a57e3df5 |
+---------------------+--------------------------------------+
  • During this time, the only log messages seen in /var/log/containers/octavia/*.log on all of the controllers that mention the load balancer id are (note these are out of order because they come from two different hosts):
/var/log/containers/octavia/octavia.log:2020-03-17 16:13:57.927 27 INFO octavia.api.v2.controllers.load_balancer [req-ee0d0960-bc8b-4560-a876-a1b9df92c75a - 4bb09e3a56cd451b9d260ad6c111fd96 - default default] Sending created Load Balancer fa463c03-715e-434e-ada1-42ac016096ff to the handler

/var/log/containers/octavia/worker.log:2020-03-17 16:13:57.952 24 INFO octavia.controller.queue.endpoint [-] Creating load balancer 'fa463c03-715e-434e-ada1-42ac016096ff'...

/var/log/containers/octavia/worker.log:                          |__Atom 'STANDALONE-octavia-get-amphora-for-lb-subflow-octavia-mapload-balancer-to-amphora' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'fa463c03-715e-434e-ada1-42ac016096ff'}, 'provides': None}

/var/log/containers/octavia/worker.log:                                |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.LoadBalancerIDToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'loadbalancer_id': u'fa463c03-715e-434e-ada1-42ac016096ff'}, 'provides': None}

/var/log/containers/octavia/worker.log:2020-03-17 16:39:45.422 24 WARNING octavia.controller.worker.tasks.database_tasks [-] Reverting Amphora allocation for the load balance
  • There is no indication why creating the load balancer failed, or what it was waiting for while in the PENDING_CREATE state.

  • Logging in the amphora VM and looking at amphora-agent.log, we notice the following errors:

[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:01 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:06 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:11 +0000] [1246] [DEBUG] Failed to send error message.
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Error processing SSL request.
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Invalid request from ip=::ffff:10.10.10.10: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake failure (_ssl.c:1826)
[2020-03-26 13:36:16 +0000] [1246] [DEBUG] Failed to send error message.

Environment

  • Red Hat OpenStack Platform 13.0 (RHOSP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content