Chapter 5. Networking
Kubernetes ensures that pods are able to network with each other, and allocates each pod an IP address from an internal network. This ensures all containers within the pod behave as if they were on the same host. Giving each pod its own IP address means that pods can be treated like physical hosts or virtual machines in terms of port allocation, networking, naming, service discovery, load balancing, application configuration, and migration.
Creating links between pods is unnecessary, and it is not recommended that your pods talk to one another directly using the IP address. Instead, it is recommend that you create a service, then interact with the service.
5.1.2. OpenShift Dedicated DNS
If you are running multiple services, such as frontend and backend services for use with multiple pods, in order for the frontend pods to communicate with the backend services, environment variables are created for user names, service IPs, and more. If the service is deleted and recreated, a new IP address can be assigned to the service, and requires the frontend pods to be recreated in order to pick up the updated values for the service IP environment variable. Additionally, the backend service has to be created before any of the frontend pods to ensure that the service IP is generated properly, and that it can be provided to the frontend pods as an environment variable.
For this reason, OpenShift Dedicated has a built-in DNS so that the services can be reached by the service DNS as well as the service IP/port. OpenShift Dedicated supports split DNS by running SkyDNS on the master that answers DNS queries for services. The master listens to port 53 by default.
When the node starts, the following message indicates the Kubelet is correctly resolved to the master:
0308 19:51:03.118430 4484 node.go:197] Started Kubelet for node openshiftdev.local, server at 0.0.0.0:10250 I0308 19:51:03.118459 4484 node.go:199] Kubelet is setting 10.0.2.15 as a DNS nameserver for domain "local"
If the second message does not appear, the Kubernetes service may not be available.
On a node host, each container’s nameserver has the master name added to the front, and the default search domain for the container will be
.<pod_namespace>.cluster.local. The container will then direct any nameserver queries to the master before any other nameservers on the node, which is the default behavior for Docker-formatted containers. The master will answer queries on the
.cluster.local domain that have the following form:
Table 5.1. DNS Example Names
This prevents having to restart frontend pods in order to pick up new services, which would create a new IP for the service. This also removes the need to use environment variables, because pods can use the service DNS. Also, as the DNS does not change, you can reference database services as
db.local in configuration files. Wildcard lookups are also supported, because any lookups resolve to the service IP, and removes the need to create the backend service before any of the frontend pods, since the service name (and hence DNS) is established upfront.
This DNS structure also covers headless services, where a portal IP is not assigned to the service and the kube-proxy does not load-balance or provide routing for its endpoints. Service DNS can still be used and responds with multiple A records, one for each pod of the service, allowing the client to round-robin between each pod.
An OpenShift Dedicated route exposes a service at a host name, such as www.example.com, so that external clients can reach it by name.
DNS resolution for a host name is handled separately from routing. Your administrator may have configured a DNS wildcard entry that will resolve to the OpenShift Dedicated node that is running the OpenShift Dedicated router. If you are using a different host name you may need to modify its DNS records independently to resolve to the node that is running the router.
Each route consists of a name (limited to 63 characters), a service selector, and an optional security configuration.
Wildcard routes are disabled in OpenShift Dedicated.
5.2.2. Route Host Names
In order for services to be exposed externally, an OpenShift Dedicated route allows you to associate a service with an externally-reachable host name. This edge host name is then used to route traffic to the service.
When multiple routes from different namespaces claim the same host, the oldest route wins and claims it for the namespace. If additional routes with different path fields are defined in the same namespace, those paths are added. If multiple routes with the same path are used, the oldest takes priority.
A consequence of this behavior is that if you have two routes for a host name: an older one and a newer one. If someone else has a route for the same host name that they created between when you created the other two routes, then if you delete your older route, your claim to the host name will no longer be in effect. The other namespace now claims the host name and your claim is lost.
Example 5.1. A Route with a Specified Host:
apiVersion: v1 kind: Route metadata: name: host-route spec: host: www.example.com 1 to: kind: Service name: service-name
- Specifies the externally-reachable host name used to expose a service.
Example 5.2. A Route Without a Host:
apiVersion: v1 kind: Route metadata: name: no-route-hostname spec: to: kind: Service name: service-name
If a host name is not provided as part of the route definition, then OpenShift Dedicated automatically generates one for you. The generated host name is of the form:
The following example shows the OpenShift Dedicated-generated host name for the above configuration of a route without a host added to a namespace mynamespace:
Example 5.3. Generated Host Name
- The generated host name suffix is the default routing subdomain router.default.svc.cluster.local.
A cluster administrator can also customize the suffix used as the default routing subdomain for their environment.
5.2.3. Route Types
Routes can be either secured or unsecured. Secure routes provide the ability to use several types of TLS termination to serve certificates to the client. Routers support edge, passthrough, and re-encryption termination.
Example 5.4. Unsecured Route Object YAML Definition
apiVersion: v1 kind: Route metadata: name: route-unsecured spec: host: www.example.com to: kind: Service name: service-name
Unsecured routes are simplest to configure, as they require no key or certificates, but secured routes offer security for connections to remain private.
A secured route is one that specifies the TLS termination of the route. The available types of termination are described below.
22.214.171.124. Path Based Routes
Path based routes specify a path component that can be compared against a URL (which requires that the traffic for the route be HTTP based) such that multiple routes can be served using the same host name, each with a different path. Routers should match routes based on the most specific path to the least; however, this depends on the router implementation. The host name and path are passed through to the backend server so it should be able to successfully answer requests for them. For example: a request to http://example.com/foo/ that goes to the router will result in a pod seeing a request to http://example.com/foo/.
The following table shows example routes and their accessibility:
Table 5.2. Route Availability
|Route||When Compared to||Accessible|
www.example.com/test and www.example.com
Yes (Matched by the host, not the route)
Example 5.5. An Unsecured Route with a Path:
apiVersion: v1 kind: Route metadata: name: route-unsecured spec: host: www.example.com path: "/test" 1 to: kind: Service name: service-name
- The path is the only added attribute for a path-based route.
Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request.
126.96.36.199. Secured Routes
Secured routes specify the TLS termination of the route and, optionally, provide a key and certificate(s).
TLS termination in OpenShift Dedicated relies on SNI for serving custom certificates. Any non-SNI traffic received on port 443 is handled with TLS termination and a default certificate (which may not match the requested host name, resulting in validation errors).
Secured routes can use any of the following three types of secure TLS termination.
With edge termination, TLS termination occurs at the router, prior to proxying traffic to its destination. TLS certificates are served by the front end of the router, so they must be configured into the route, otherwise the router’s default certificate will be used for TLS termination.
Example 5.6. A Secured Route Using Edge Termination
apiVersion: v1 kind: Route metadata: name: route-edge-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 key: |- 4 -----BEGIN PRIVATE KEY----- [...] -----END PRIVATE KEY----- certificate: |- 5 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- caCertificate: |- 6 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
- 1 2
- The name of the object, which is limited to 63 characters.
edgefor edge termination.
keyfield is the contents of the PEM format key file.
certificatefield is the contents of the PEM format certificate file.
- An optional CA certificate may be required to establish a certificate chain for validation.
Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.
Edge-terminated routes can specify an
insecureEdgeTerminationPolicy that enables traffic on insecure schemes (
HTTP) to be disabled, allowed or redirected. The allowed values for
None or empty (for disabled),
Redirect. The default
Example 5.7. A Secured Route Using Edge Termination Allowing HTTP Traffic
apiVersion: v1 kind: Route metadata: name: route-edge-secured-allow-insecure 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 insecureEdgeTerminationPolicy: Allow 4 [ ... ]
Example 5.8. A Secured Route Using Edge Termination Redirecting HTTP Traffic to HTTPS
apiVersion: v1 kind: Route metadata: name: route-edge-secured-redirect-insecure 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 insecureEdgeTerminationPolicy: Redirect 4 [ ... ]
With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required.
Example 5.9. A Secured Route Using Passthrough Termination
apiVersion: v1 kind: Route metadata: name: route-passthrough-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: passthrough 3
The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates (also known as two-way authentication).
Passthrough routes can also have an
insecureEdgeTerminationPolicy. The only valid values are
None (or empty, for disabled) or
Re-encryption is a variation on edge termination where the router terminates TLS with a certificate, then re-encrypts its connection to the endpoint which may have a different certificate. Therefore the full path of the connection is encrypted, even over the internal network. The router uses health checks to determine the authenticity of the host.
Example 5.10. A Secured Route Using Re-Encrypt Termination
apiVersion: v1 kind: Route metadata: name: route-pt-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: reencrypt 3 key: [as in edge termination] certificate: [as in edge termination] caCertificate: [as in edge termination] destinationCACertificate: |- 4 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
- 1 2
- The name of the object, which is limited to 63 characters.
terminationfield is set to
reencrypt. Other fields are as in edge termination.
- Required for re-encryption.
destinationCACertificatespecifies a CA certificate to validate the endpoint certificate, securing the connection from the router to the destination pods. If the service is using a service signing certificate, or the administrator has specified a default CA certificate for the router and the service has a certificate signed by that CA, this field can be omitted.
destinationCACertificate field is left empty, the router automatically leverages the certificate authority that is generated for service serving certificates, and is injected into every pod as
/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt. This allows new routes that leverage end-to-end encryption without having to generate a certificate for the route. This is useful for custom routers or the F5 router, which might not allow the
destinationCACertificate unless the administrator has allowed it.
Re-encrypt routes can have an
insecureEdgeTerminationPolicy with all of the same values as edge-terminated routes.
5.2.4. Alternate Backends and Weights
A route is usually associated with one service through the
to: token with
kind: Service. All of the requests to the route are handled by endpoints in the service based on the load balancing strategy
It is possible to have as many as four services supporting the route. The portion of requests that are handled by each service is governed by the service
The first service is entered using the
to: token as before, and up to three additional services can be entered using the
alternateBackend: token. Each service must be
kind: Service which is the default.
Each service has a
weight associated with it. The portion of requests handled by the service is
sum_of_all_weights. When a service has more than one endpoint, the service’s weight is distributed among the endpoints with each endpoint getting at least 1. If the service
weight is 0 each of the service’s endpoints will get 0.
weight must be in the range 0-256. The default is 1. When the
weight is 0, the service does not participate in load-balancing but continues to serve existing persistent connections.
alternateBackends also use the
roundrobin load balancing strategy to ensure requests are distributed as expected to the services based on
roundrobin can be set for a route using a route annotation, or for the router in general using an environment variable.
The following is an example route configuration using alternate backends for A/B deployments.
A Route with alternateBackends and weights:
apiVersion: v1 kind: Route metadata: name: route-alternate-service annotations: haproxy.router.openshift.io/balance: roundrobin 1 spec: host: www.example.com to: kind: Service name: service-name 2 weight: 20 3 alternateBackends: - kind: Service name: service-name2 4 weight: 10 5 - kind: Service name: service-name3 6 weight: 10 7
- This route uses
roundrobinload balancing strategy
- The first service name is
service-namewhich may have 0 or more pods
- 4 6
- The alternateBackend services may also have 0 or more pods
- 3 5 7
- The total
service-namewill get 20/40 or 1/2 of the requests,
service-name3will each get 1/4 of the requests, assuming each service has 1 or more endpoints.
5.2.5. Route-specific Annotations
Using environment variables, a router can set the default options for all the routes it exposes. An individual route can override some of these defaults by providing specific configurations in its annotations.
For all the items outlined in this section, you can set annotations on the route definition for the route to alter its configuration
Table 5.3. Route Annotations
|Variable||Description||Environment Variable Used as Default|
| || |
Sets the load-balancing algorithm. Available options are
| || |
| || |
Specifies an optional cookie to use for this route. The name must consist of any combination of upper and lower case letters, digits, "_", and "-". The default is the hashed internal key name for the route.
| || |
Sets the maximum number of connections that are allowed to a backing pod from a router. Note: if there are multiple pods, each can have this many connections. But if you have multiple routers, there is no coordination among them, each may connect this many times. If not set, or set to 0, there is no limit.
| || |
| || |
Limits the number of concurrent TCP connections shared by an IP address.
| || |
Limits the rate at which an IP address can make HTTP requests.
| || |
Limits the rate at which an IP address can make TCP connections.
| || |
Sets a server-side timeout for the route. (TimeUnits)
| || |
Sets the interval for the back-end health checks. (TimeUnits)
| || |
Sets a whitelist for the route.
| || |
Sets a Strict-Transport-Security header for the edge terminated or re-encrypt route.
Example 5.11. A Route Setting Custom Timeout
apiVersion: v1 kind: Route metadata: annotations: haproxy.router.openshift.io/timeout: 5500ms 1 [...]
- Specifies the new timeout with HAProxy supported units (us, ms, s, m, h, d). If unit not provided, ms is the default.
Setting a server-side timeout value for passthrough routes too low can cause WebSocket connections to timeout frequently on that route.
5.2.6. Route-specific IP Whitelists
You can restrict access to a route to a select set of IP addresses by adding the
haproxy.router.openshift.io/ip_whitelist annotation on the route. The whitelist is a space-separated list of IP addresses and/or CIDRs for the approved source addresses. Requests from IP addresses that are not in the whitelist are dropped.
When editing a route, add the following annotation to define the desired source IP’s. Alternatively, use
oc annotate route <name>.
Allow only one specific IP address:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.10
Allow several IP addresses:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.10 192.168.1.11 192.168.1.12
Allow an IP CIDR network:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.0/24
Allow mixed IP addresses and IP CIDR networks:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 188.8.131.52 192.168.1.0/24 10.0.0.0/8