Red Hat Training

A Red Hat training course is available for Red Hat OpenStack Platform

Chapter 3. Services Configuration Reference

This chapter outlines the configuration for each specific service in the Overcloud that uses load balancing. Use this configuration as a guide to configuring your own external load balancer. For details about these and other parameter, see the "HAProxy Configuration Manual" located in /usr/share/doc/haproxy-*/configuration.txt on the Controller nodes (or any system where the haproxy package is installed).

Note

Most services use a default health check configuration:

  • The interval between two consecutive health checks set to 2000 milliseconds (or 2 seconds).
  • After two successful health checks, a server is considered operational.
  • After five failed health checks, the service is considered dead.

Each service indicates the default health check or additional options in the Other information section of each service.

3.1. aodh

Port Number: 8042

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen aodh
  bind 172.16.20.250:8042
  bind 172.16.23.250:8042
  server overcloud-controller-0 172.16.20.150:8042 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8042 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8042 check fall 5 inter 2000 rise 2

3.2. ceilometer

Port Number: 8777

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen ceilometer
  bind 172.16.20.250:8777
  bind 172.16.23.250:8777
  server overcloud-controller-0 172.16.20.150:8777 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8777 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8777 check fall 5 inter 2000 rise 2

3.3. cinder

Port Number: 8776

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen cinder
  bind 172.16.20.250:8776
  bind 172.16.23.250:8776
  server overcloud-controller-0 172.16.20.150:8776 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8776 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8776 check fall 5 inter 2000 rise 2

3.4. glance_api

Port Number: 9292

Binds to: storage, external

Target network/server: storage on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen glance_api
  bind 172.16.23.250:9292
  bind 172.16.21.250:9292
  server overcloud-controller-0 172.16.21.150:9292 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.21.151:9292 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.21.152:9292 check fall 5 inter 2000 rise 2

3.5. glance_registry

Port Number: 9191

Binds to: internal_api

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen glance_registry
  bind 172.16.20.250:9191
  server overcloud-controller-0 172.16.20.150:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:9191 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:9191 check fall 5 inter 2000 rise 2

3.6. gnocchi

Port Number: 8041

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen gnocchi
  bind 172.16.20.250:8041
  bind 172.16.23.250:8041
  server overcloud-controller-0 172.16.20.150:8041 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8041 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8041 check fall 5 inter 2000 rise 2

3.7. heat_api

Port Number: 8004

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check
  • This service uses HTTP mode instead of the default TCP mode

HAProxy Example: 

listen heat_api
  bind 172.16.20.250:8004
  bind 172.16.23.250:8004
  mode http
  server overcloud-controller-0 172.16.20.150:8004 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8004 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8004 check fall 5 inter 2000 rise 2

3.8. heat_cfn

Port Number: 8000

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen heat_cfn
  bind 172.16.20.250:8000
  bind 172.16.23.250:8000
  server overcloud-controller-0 172.16.20.150:8000 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.152:8000 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.151:8000 check fall 5 inter 2000 rise 2

3.9. heat_cloudwatch

Port Number: 8003

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen heat_cloudwatch
  bind 172.16.20.250:8003
  bind 172.16.23.250:8003
  server overcloud-controller-0 172.16.20.150:8003 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8003 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8003 check fall 5 inter 2000 rise 2

3.10. horizon

Port Number: 80

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check
  • This service uses HTTP mode instead of the default TCP mode
  • This service uses cookie-based persistence for interactions with the UI

HAProxy Example: 

listen horizon
  bind 172.16.20.250:80
  bind 172.16.23.250:80
  mode http
  cookie SERVERID insert indirect nocache
  server overcloud-controller-0 172.16.20.150:80 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:80 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:80 check fall 5 inter 2000 rise 2

3.11. keystone_admin

Port Number: 35357

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen keystone_admin
  bind 172.16.23.250:35357
  bind 172.16.20.250:35357
  server overcloud-controller-0 172.16.20.150:35357 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:35357 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:35357 check fall 5 inter 2000 rise 2

3.12. keystone_admin_ssh

Port Number: 22

Binds to: internal_api

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen keystone_admin_ssh
  bind 172.16.20.250:22
  server overcloud-controller-0 172.16.20.150:22 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:22 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:22 check fall 5 inter 2000 rise 2

3.13. keystone_public

Port Number: 5000

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen keystone_public
  bind 172.16.20.250:5000
  bind 172.16.23.250:5000
  server overcloud-controller-0 172.16.20.150:5000 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:5000 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:5000 check fall 5 inter 2000 rise 2

3.14. mysql

Port Number: 3306

Binds to: internal_api

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check. However, the health checks use port 9200.
  • This service is load balanced to only one server at a time.
  • Each server is only used in load balancing when all other non-backup servers are unavailable.
  • If the server is marked down, all connections are immediately terminated.
  • Enable the sending of TCP keepalive packets on both sides.
  • Enable HTTP protocol to check on the servers health.
  • Configure a stickiness table to store IP address. This helps maintain persistence.
Important

The mysql service uses Galera to provide a highly available database cluster. While Galera supports an active/active configuration, we recommend using an active/passive enforced by the load balancer to avoid lock contention.

HAProxy Example: 

listen mysql
  bind 172.16.20.250:3306
  option tcpka
  option httpchk
  stick on dst
  stick-table type ip size 1000
  timeout client 0
  timeout server 0
  server overcloud-controller-0 172.16.20.150:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-1 172.16.20.151:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
  server overcloud-controller-2 172.16.20.152:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2

3.15. neutron

Port Number: 9696

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen neutron
  bind 172.16.20.250:9696
  bind 172.16.23.250:9696
  server overcloud-controller-0 172.16.20.150:9696 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:9696 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:9696 check fall 5 inter 2000 rise 2

3.16. nova_ec2

Port Number: 8773

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen nova_ec2
  bind 172.16.20.250:8773
  bind 172.16.23.250:8773
  server overcloud-controller-0 172.16.20.150:8773 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8773 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8773 check fall 5 inter 2000 rise 2

3.17. nova_metadata

Port Number: 8775

Binds to: internal_api

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen nova_metadata
  bind 172.16.20.250:8775
  server overcloud-controller-0 172.16.20.150:8775 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8775 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8775 check fall 5 inter 2000 rise 2

3.18. nova_novncproxy

Port Number: 6080

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check
  • The default balancing method is round-robin. However, for this service, use a source method. This method hashes the source IP address and divides it by the total weight of the running servers. This designates the server that receives the request. This ensures the same client IP address always reaches the same server as long as no server goes down or up. If the hash result changes due to a change in the number of running servers, the balancer redirects many clients to a different server.

HAProxy Example: 

listen nova_novncproxy
  bind 172.16.20.250:6080
  bind 172.16.23.250:6080
  balance source
  server overcloud-controller-0 172.16.20.150:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:6080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:6080 check fall 5 inter 2000 rise 2

3.19. nova_osapi

Port Number: 8774

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen nova_osapi
  bind 172.16.20.250:8774
  bind 172.16.23.250:8774
  server overcloud-controller-0 172.16.20.150:8774 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8774 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8774 check fall 5 inter 2000 rise 2

3.20. redis

Port Number: 6379

Binds to: internal_api (redis service IP)

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check.
  • Perform health checks using tcp-check send/expect sequences. The string to send is "info\ replication\r\n" and the response is "role:master"
  • The Redis service uses a password for authentication. For example, the HAProxy configuration uses a tcp-check with and AUTH method and the Redis administration password. The director normally generates a random password, but you can define a custom Redis password. See Section 4.2.2, “Configuring Load Balancing Options” for more information.
  • The default balancing method is round-robin. However, for this service, use a first method. This ensures the first server with available connection slots receives the connection.

HAProxy Example: 

listen redis
  bind 172.16.20.249:6379 transparent
  balance first
  option tcp-check
  tcp-check send AUTH\ p@55w0rd!\r\n
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  server overcloud-controller-0 172.16.20.150:6379 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:6379 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:6379 check fall 5 inter 2000 rise 2

3.21. sahara

Port Number: 8386

Binds to: internal_api, external

Target network/server: internal_api on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check
  • This service is an optional overcloud service. To install, including the environments/services/sahara.yaml environment file in your overcloud deployment.

HAProxy Example: 

listen sahara
  bind 172.16.20.250:8386
  bind 172.16.23.250:8386
  server overcloud-controller-0 172.16.20.150:8386 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.20.151:8386 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.20.152:8386 check fall 5 inter 2000 rise 2

3.22. swift_proxy_server

Port Number: 8080

Binds to: storage, external

Target network/server: storage on overcloud-controller-0, overcloud-controller-1, and overcloud-controller-2

Other information:

  • Each target server uses a default health check

HAProxy Example: 

listen swift_proxy_server
  bind 172.16.23.250:8080
  bind 172.16.21.250:8080
  server overcloud-controller-0 172.16.21.150:8080 check fall 5 inter 2000 rise 2
  server overcloud-controller-1 172.16.21.151:8080 check fall 5 inter 2000 rise 2
  server overcloud-controller-2 172.16.21.152:8080 check fall 5 inter 2000 rise 2