-
Language:
English
-
Language:
English
Red Hat Training
A Red Hat training course is available for OpenShift Dedicated
Chapter 5. Networking
5.1. Networking
5.1.1. Overview
Kubernetes ensures that pods are able to network with each other, and allocates each pod an IP address from an internal network. This ensures all containers within the pod behave as if they were on the same host. Giving each pod its own IP address means that pods can be treated like physical hosts or virtual machines in terms of port allocation, networking, naming, service discovery, load balancing, application configuration, and migration.
Creating links between pods is unnecessary, and it is not recommended that your pods talk to one another directly using the IP address. Instead, it is recommended that you create a service, then interact with the service.
5.1.2. OpenShift Dedicated DNS
If you are running multiple services, such as frontend and backend services for use with multiple pods, in order for the frontend pods to communicate with the backend services, environment variables are created for user names, service IPs, and more. If the service is deleted and recreated, a new IP address can be assigned to the service, and requires the frontend pods to be recreated in order to pick up the updated values for the service IP environment variable. Additionally, the backend service has to be created before any of the frontend pods to ensure that the service IP is generated properly, and that it can be provided to the frontend pods as an environment variable.
For this reason, OpenShift Dedicated has a built-in DNS so that the services can be reached by the service DNS as well as the service IP/port. OpenShift Dedicated supports split DNS by running SkyDNS on the master that answers DNS queries for services. The master listens to port 53 by default.
When the node starts, the following message indicates the Kubelet is correctly resolved to the master:
0308 19:51:03.118430 4484 node.go:197] Started Kubelet for node openshiftdev.local, server at 0.0.0.0:10250 I0308 19:51:03.118459 4484 node.go:199] Kubelet is setting 10.0.2.15 as a DNS nameserver for domain "local"
If the second message does not appear, the Kubernetes service may not be available.
On a node host, each container’s nameserver has the master name added to the front, and the default search domain for the container will be .<pod_namespace>.cluster.local
. The container will then direct any nameserver queries to the master before any other nameservers on the node, which is the default behavior for Docker-formatted containers. The master will answer queries on the .cluster.local
domain that have the following form:
Table 5.1. DNS Example Names
Object Type | Example |
---|---|
Default | <pod_namespace>.cluster.local |
Services | <service>.<pod_namespace>.svc.cluster.local |
Endpoints | <name>.<namespace>.endpoints.cluster.local |
This prevents having to restart frontend pods in order to pick up new services, which would create a new IP for the service. This also removes the need to use environment variables, because pods can use the service DNS. Also, as the DNS does not change, you can reference database services as db.local
in configuration files. Wildcard lookups are also supported, because any lookups resolve to the service IP, and removes the need to create the backend service before any of the frontend pods, since the service name (and hence DNS) is established upfront.
This DNS structure also covers headless services, where a portal IP is not assigned to the service and the kube-proxy does not load-balance or provide routing for its endpoints. Service DNS can still be used and responds with multiple A records, one for each pod of the service, allowing the client to round-robin between each pod.
5.2. Routes
5.2.1. Overview
An OpenShift Dedicated route exposes a service at a host name, such as www.example.com, so that external clients can reach it by name.
DNS resolution for a host name is handled separately from routing. Your administrator may have configured a DNS wildcard entry that will resolve to the OpenShift Dedicated node that is running the OpenShift Dedicated router. If you are using a different host name you may need to modify its DNS records independently to resolve to the node that is running the router.
Each route consists of a name (limited to 63 characters), a service selector, and an optional security configuration.
Wildcard routes are disabled in OpenShift Dedicated.
5.2.2. Route Host Names
In order for services to be exposed externally, an OpenShift Dedicated route allows you to associate a service with an externally-reachable host name. This edge host name is then used to route traffic to the service.
When multiple routes from different namespaces claim the same host, the oldest route wins and claims it for the namespace. If additional routes with different path fields are defined in the same namespace, those paths are added. If multiple routes with the same path are used, the oldest takes priority.
A consequence of this behavior is that if you have two routes for a host name: an older one and a newer one. If someone else has a route for the same host name that they created between when you created the other two routes, then if you delete your older route, your claim to the host name will no longer be in effect. The other namespace now claims the host name and your claim is lost.
Example 5.1. A Route with a Specified Host:
apiVersion: v1
kind: Route
metadata:
name: host-route
spec:
host: www.example.com 1
to:
kind: Service
name: service-name
- 1
- Specifies the externally-reachable host name used to expose a service.
Example 5.2. A Route Without a Host:
apiVersion: v1 kind: Route metadata: name: no-route-hostname spec: to: kind: Service name: service-name
If a host name is not provided as part of the route definition, then OpenShift Dedicated automatically generates one for you. The generated host name is of the form:
<route-name>[-<namespace>].<suffix>
The following example shows the OpenShift Dedicated-generated host name for the above configuration of a route without a host added to a namespace mynamespace:
Example 5.3. Generated Host Name
no-route-hostname-mynamespace.router.default.svc.cluster.local 1
- 1
- The generated host name suffix is the default routing subdomain router.default.svc.cluster.local.
A cluster administrator can also customize the suffix used as the default routing subdomain for their environment.
5.2.3. Route Types
Routes can be either secured or unsecured. Secure routes provide the ability to use several types of TLS termination to serve certificates to the client. Routers support edge, passthrough, and re-encryption termination.
Example 5.4. Unsecured Route Object YAML Definition
apiVersion: v1 kind: Route metadata: name: route-unsecured spec: host: www.example.com to: kind: Service name: service-name
Unsecured routes are simplest to configure, as they require no key or certificates, but secured routes offer security for connections to remain private.
A secured route is one that specifies the TLS termination of the route. The available types of termination are described below.
5.2.3.1. Path Based Routes
Path based routes specify a path component that can be compared against a URL (which requires that the traffic for the route be HTTP based) such that multiple routes can be served using the same host name, each with a different path. Routers should match routes based on the most specific path to the least; however, this depends on the router implementation. The host name and path are passed through to the backend server so it should be able to successfully answer requests for them. For example: a request to http://example.com/foo/ that goes to the router will result in a pod seeing a request to http://example.com/foo/.
The following table shows example routes and their accessibility:
Table 5.2. Route Availability
Route | When Compared to | Accessible |
---|---|---|
www.example.com/test | www.example.com/test | Yes |
www.example.com | No | |
www.example.com/test and www.example.com | www.example.com/test | Yes |
www.example.com | Yes | |
www.example.com | www.example.com/test | Yes (Matched by the host, not the route) |
www.example.com | Yes |
Example 5.5. An Unsecured Route with a Path:
apiVersion: v1
kind: Route
metadata:
name: route-unsecured
spec:
host: www.example.com
path: "/test" 1
to:
kind: Service
name: service-name
- 1
- The path is the only added attribute for a path-based route.
Path-based routing is not available when using passthrough TLS, as the router does not terminate TLS in that case and cannot read the contents of the request.
5.2.3.2. Secured Routes
Secured routes specify the TLS termination of the route and, optionally, provide a key and certificate(s).
TLS termination in OpenShift Dedicated relies on SNI for serving custom certificates. Any non-SNI traffic received on port 443 is handled with TLS termination and a default certificate (which may not match the requested host name, resulting in validation errors).
Secured routes can use any of the following three types of secure TLS termination.
Edge Termination
With edge termination, TLS termination occurs at the router, prior to proxying traffic to its destination. TLS certificates are served by the front end of the router, so they must be configured into the route, otherwise the router’s default certificate will be used for TLS termination.
Example 5.6. A Secured Route Using Edge Termination
apiVersion: v1 kind: Route metadata: name: route-edge-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 key: |- 4 -----BEGIN PRIVATE KEY----- [...] -----END PRIVATE KEY----- certificate: |- 5 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE----- caCertificate: |- 6 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
- 1 2
- The name of the object, which is limited to 63 characters.
- 3
- The
termination
field isedge
for edge termination. - 4
- The
key
field is the contents of the PEM format key file. - 5
- The
certificate
field is the contents of the PEM format certificate file. - 6
- An optional CA certificate may be required to establish a certificate chain for validation.
Because TLS is terminated at the router, connections from the router to the endpoints over the internal network are not encrypted.
Edge-terminated routes can specify an insecureEdgeTerminationPolicy
that enables traffic on insecure schemes (HTTP
) to be disabled, allowed or redirected. The allowed values for insecureEdgeTerminationPolicy
are: None
or empty (for disabled), Allow
or Redirect
. The default insecureEdgeTerminationPolicy
is to disable traffic on the insecure scheme. A common use case is to allow content to be served via a secure scheme but serve the assets (example images, stylesheets and javascript) via the insecure scheme.
Example 5.7. A Secured Route Using Edge Termination Allowing HTTP Traffic
apiVersion: v1 kind: Route metadata: name: route-edge-secured-allow-insecure 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 insecureEdgeTerminationPolicy: Allow 4 [ ... ]
Example 5.8. A Secured Route Using Edge Termination Redirecting HTTP Traffic to HTTPS
apiVersion: v1 kind: Route metadata: name: route-edge-secured-redirect-insecure 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: edge 3 insecureEdgeTerminationPolicy: Redirect 4 [ ... ]
Passthrough Termination
With passthrough termination, encrypted traffic is sent straight to the destination without the router providing TLS termination. Therefore no key or certificate is required.
Example 5.9. A Secured Route Using Passthrough Termination
apiVersion: v1 kind: Route metadata: name: route-passthrough-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: passthrough 3
The destination pod is responsible for serving certificates for the traffic at the endpoint. This is currently the only method that can support requiring client certificates (also known as two-way authentication).
Passthrough routes can also have an insecureEdgeTerminationPolicy
. The only valid values are None
(or empty, for disabled) or Redirect
.
Re-encryption Termination
Re-encryption is a variation on edge termination where the router terminates TLS with a certificate, then re-encrypts its connection to the endpoint which may have a different certificate. Therefore the full path of the connection is encrypted, even over the internal network. The router uses health checks to determine the authenticity of the host.
Example 5.10. A Secured Route Using Re-Encrypt Termination
apiVersion: v1 kind: Route metadata: name: route-pt-secured 1 spec: host: www.example.com to: kind: Service name: service-name 2 tls: termination: reencrypt 3 key: [as in edge termination] certificate: [as in edge termination] caCertificate: [as in edge termination] destinationCACertificate: |- 4 -----BEGIN CERTIFICATE----- [...] -----END CERTIFICATE-----
- 1 2
- The name of the object, which is limited to 63 characters.
- 3
- The
termination
field is set toreencrypt
. Other fields are as in edge termination. - 4
- Required for re-encryption.
destinationCACertificate
specifies a CA certificate to validate the endpoint certificate, securing the connection from the router to the destination pods. If the service is using a service signing certificate, or the administrator has specified a default CA certificate for the router and the service has a certificate signed by that CA, this field can be omitted.
If the destinationCACertificate
field is left empty, the router automatically leverages the certificate authority that is generated for service serving certificates, and is injected into every pod as /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
. This allows new routes that leverage end-to-end encryption without having to generate a certificate for the route. This is useful for custom routers or the F5 router, which might not allow the destinationCACertificate
unless the administrator has allowed it.
Re-encrypt routes can have an insecureEdgeTerminationPolicy
with all of the same values as edge-terminated routes.
5.2.4. Alternate Backends and Weights
A route is usually associated with one service through the to:
token with kind: Service
. All of the requests to the route are handled by endpoints in the service based on the load balancing strategy
It is possible to have as many as four services supporting the route. The portion of requests that are handled by each service is governed by the service weight
.
The first service is entered using the to:
token as before, and up to three additional services can be entered using the alternateBackend:
token. Each service must be kind: Service
which is the default.
Each service has a weight
associated with it. The portion of requests handled by the service is weight
/ sum_of_all_weights
. When a service has more than one endpoint, the service’s weight is distributed among the endpoints with each endpoint getting at least 1. If the service weight
is 0 each of the service’s endpoints will get 0.
The weight
must be in the range 0-256. The default is 1. When the weight
is 0, the service does not participate in load-balancing but continues to serve existing persistent connections.
When using alternateBackends
also use the roundrobin
load balancing strategy to ensure requests are distributed as expected to the services based on weight
. roundrobin
can be set for a route using a route annotation, or for the router in general using an environment variable.
The following is an example route configuration using alternate backends for A/B deployments.
A Route with alternateBackends and weights:
apiVersion: v1 kind: Route metadata: name: route-alternate-service annotations: haproxy.router.openshift.io/balance: roundrobin 1 spec: host: www.example.com to: kind: Service name: service-name 2 weight: 20 3 alternateBackends: - kind: Service name: service-name2 4 weight: 10 5 - kind: Service name: service-name3 6 weight: 10 7
- 1
- This route uses
roundrobin
load balancing strategy - 2
- The first service name is
service-name
which may have 0 or more pods - 4 6
- The alternateBackend services may also have 0 or more pods
- 3 5 7
- The total
weight
is 40.service-name
will get 20/40 or 1/2 of the requests,service-name2
andservice-name3
will each get 1/4 of the requests, assuming each service has 1 or more endpoints.
5.2.5. Route-specific Annotations
Using environment variables, a router can set the default options for all the routes it exposes. An individual route can override some of these defaults by providing specific configurations in its annotations.
Route Annotations
For all the items outlined in this section, you can set annotations on the route definition for the route to alter its configuration
Table 5.3. Route Annotations
Variable | Description | Environment Variable Used as Default |
---|---|---|
|
Sets the load-balancing algorithm. Available options are |
|
|
Disables the use of cookies to track related connections. If set to | |
| Specifies an optional cookie to use for this route. The name must consist of any combination of upper and lower case letters, digits, "_", and "-". The default is the hashed internal key name for the route. | |
| Sets the maximum number of connections that are allowed to a backing pod from a router. Note: if there are multiple pods, each can have this many connections. But if you have multiple routers, there is no coordination among them, each may connect this many times. If not set, or set to 0, there is no limit. | |
|
Setting | |
| Limits the number of concurrent TCP connections shared by an IP address. | |
| Limits the rate at which an IP address can make HTTP requests. | |
| Limits the rate at which an IP address can make TCP connections. | |
| Sets a server-side timeout for the route. (TimeUnits) |
|
| Sets the interval for the back-end health checks. (TimeUnits) |
|
| Sets a whitelist for the route. | |
| Sets a Strict-Transport-Security header for the edge terminated or re-encrypt route. |
Example 5.11. A Route Setting Custom Timeout
apiVersion: v1
kind: Route
metadata:
annotations:
haproxy.router.openshift.io/timeout: 5500ms 1
[...]
- 1
- Specifies the new timeout with HAProxy supported units (us, ms, s, m, h, d). If unit not provided, ms is the default.
Setting a server-side timeout value for passthrough routes too low can cause WebSocket connections to timeout frequently on that route.
5.2.6. Route-specific IP Whitelists
You can restrict access to a route to a select set of IP addresses by adding the haproxy.router.openshift.io/ip_whitelist
annotation on the route. The whitelist is a space-separated list of IP addresses and/or CIDRs for the approved source addresses. Requests from IP addresses that are not in the whitelist are dropped.
Some examples:
When editing a route, add the following annotation to define the desired source IP’s. Alternatively, use oc annotate route <name>
.
Allow only one specific IP address:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.10
Allow several IP addresses:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.10 192.168.1.11 192.168.1.12
Allow an IP CIDR network:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 192.168.1.0/24
Allow mixed IP addresses and IP CIDR networks:
metadata: annotations: haproxy.router.openshift.io/ip_whitelist: 180.5.61.153 192.168.1.0/24 10.0.0.0/8