Chapter 6. Knative Serving

6.1. Understanding Knative Serving

Knative Serving on OpenShift Container Platform enables developers to write cloud-native applications using serverless architecture.

Knative Serving supports deploying and managing cloud-native applications by providing a set of objects as Kubernetes custom resource definitions (CRDs) that define and control the behavior of serverless workloads on an OpenShift Container Platform cluster. For more information about CRDs, see Extending the Kubernetes API with custom resource definitions.

Developers use these CRDs to create custom resource (CR) instances that can be used as building blocks to address complex use cases. For example:

  • Rapidly deploying serverless containers.
  • Automatically scaling pods.

For more information about CRs, see Managing resources from Custom Resource Definitions.

6.1.1. Knative Serving custom resource definitions

Service
The service.serving.knative.dev CRD automatically manages the life cycle of your workload to ensure that the application is deployed and reachable through the network. It creates a route, a configuration, and a new revision for each change to a user created service, or custom resource. Most developer interactions in Knative are carried out by modifying services.
Revision
The revision.serving.knative.dev CRD is a point-in-time snapshot of the code and configuration for each modification made to the workload. Revisions are immutable objects and can be retained for as long as necessary.
Route
The route.serving.knative.dev CRD maps a network endpoint to one or more revisions. You can manage the traffic in several ways, including fractional traffic and named routes.
Configuration
The configuration.serving.knative.dev CRD maintains the desired state for your deployment. It provides a clean separation between code and configuration. Modifying a configuration creates a new revision.

6.2. Serverless applications

6.2.1. Serverless applications using Knative services

To deploy a serverless application using OpenShift Serverless, you must create a Knative service. Knative services are Kubernetes services, defined by a route and a configuration, and contained in a YAML file.

Example Knative service YAML

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: hello 1
  namespace: default 2
spec:
  template:
    spec:
      containers:
        - image: docker.io/openshift/hello-openshift 3
          env:
            - name: RESPONSE 4
              value: "Hello Serverless!"

1
The name of the application.
2
The namespace the application will use.
3
The image of the application.
4
The environment variable printed out by the sample application.

6.2.2. Creating serverless applications

You can create a serverless application by using one of the following methods:

6.2.2.1. Creating serverless applications using the OpenShift Container Platform web console

For more information about creating applications using the Developer perspective in the OpenShift Container Platform web console, see the documentation on Creating applications using the Developer perspective.

6.2.2.2. Creating serverless applications by using the Knative CLI

The following procedure describes how you can create a basic serverless application using the kn CLI.

Prerequisites

  • OpenShift Serverless Operator and Knative Serving are installed on your cluster.
  • You have installed the kn CLI.

Procedure

  • Create a Knative service:

    $ kn service create <service-name> --image <image> --env <key=value>

    Example command

    $ kn service create event-display \
        --image quay.io/openshift-knative/knative-eventing-sources-event-display:latest

    Example output

    Creating service 'event-display' in namespace 'default':
    
      0.271s The Route is still working to reflect the latest desired specification.
      0.580s Configuration "event-display" is waiting for a Revision to become ready.
      3.857s ...
      3.861s Ingress has not yet been reconciled.
      4.270s Ready to serve.
    
    Service 'event-display' created with latest revision 'event-display-bxshg-1' and URL:
    http://event-display-default.apps-crc.testing

6.2.2.2.1. Knative client multi-container support

You can use the kn container add command to print YAML container spec to standard output. This command is useful for multi-container use cases because it can be used along with other standard kn flags to create definitions. The kn container add command accepts all container-related flags that are supported for use with the kn service create command. The kn container add command can also be chained by using UNIX pipes (|) to create multiple container definitions at once.

6.2.2.2.1.1. Example commands
  • Add a container from an image and print it to standard output:

    $ kn container add <container_name> --image <image_uri>

    Example command

    $ kn container add sidecar --image docker.io/example/sidecar

    Example output

    containers:
    - image: docker.io/example/sidecar
      name: sidecar
      resources: {}

  • Chain two kn container add commands together, and then pass them to a kn service create command to create a Knative service with two containers:

    $ kn container add <first_container_name> --image <image_uri> | \
    kn container add <second_container_name> --image <image_uri> | \
    kn service create <service_name> --image <image_uri> --extra-containers -

    --extra-containers - specifies a special case where kn reads the pipe input instead of a YAML file.

    Example command

    $ kn container add sidecar --image docker.io/example/sidecar:first | \
    kn container add second --image docker.io/example/sidecar:second | \
    kn service create my-service --image docker.io/example/my-app:latest --extra-containers -

    The --extra-containers flag can also accept a path to a YAML file:

    $ kn service create <service_name> --image <image_uri> --extra-containers <filename>

    Example command

    $ kn service create my-service --image docker.io/example/my-app:latest --extra-containers my-extra-containers.yaml

6.2.2.3. Creating serverless applications using YAML

To create a serverless application by using YAML, you must create a YAML file that defines a Service object, then apply it by using oc apply.

Procedure

  1. Create a YAML file containing the following sample code:

    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      name: event-delivery
      namespace: default
    spec:
      template:
        spec:
          containers:
            - image: quay.io/openshift-knative/knative-eventing-sources-event-display:latest
              env:
                - name: RESPONSE
                  value: "Hello Serverless!"
  2. Navigate to the directory where the YAML file is contained, and deploy the application by applying the YAML file:

    $ oc apply -f <filename>

After the service is created and the application is deployed, Knative creates an immutable revision for this version of the application. Knative also performs network programming to create a route, ingress, service, and load balancer for your application and automatically scales your pods up and down based on traffic, including inactive pods.

6.2.3. Updating serverless applications by using the Knative CLI

You can use the kn service update command for interactive sessions on the command line as you build up a service incrementally. In contrast to the kn service apply command, when using the kn service update command you only have to specify the changes that you want to update, rather than the full configuration for the Knative service.

Example commands

  • Update a service by adding a new environment variable:

    $ kn service update <service_name> --env <key>=<value>
  • Update a service by adding a new port:

    $ kn service update <service_name> --port 80
  • Update a service by adding new request and limit parameters:

    $ kn service update <service_name> --request cpu=500m --limit memory=1024Mi --limit cpu=1000m
  • Assign the latest tag to a revision:

    $ kn service update <service_name> --tag <revision_name>=latest
  • Update a tag from testing to staging for the latest READY revision of a service:

    $ kn service update <service_name> --untag testing --tag @latest=staging
  • Add the test tag to a revision that receives 10% of traffic, and send the rest of the traffic to the latest READY revision of a service:

    $ kn service update <service_name> --tag <revision_name>=test --traffic test=10,@latest=90

6.2.4. Applying service declarations

You can declaratively configure a Knative service by using the kn service apply command. If the service does not exist it is created, otherwise the existing service is updated with the options that have been changed.

The kn service apply command is especially useful for shell scripts or in a continuous integration pipeline, where users typically want to fully specify the state of the service in a single command to declare the target state.

When using kn service apply you must provide the full configuration for the Knative service. This is different from the kn service update command, which only requires you to specify in the command the options that you want to update.

Example commands

  • Create a service:

    $ kn service apply <service_name> --image <image>
  • Add an environment variable to a service:

    $ kn service apply <service_name> --image <image> --env <key>=<value>
  • Read the service declaration from a JSON or YAML file:

    $ kn service apply <service_name> -f <filename>

6.2.5. Describing serverless applications by using the Knative CLI

You can describe a Knative service by using the kn service describe command.

Example commands

  • Describe a service:

    $ kn service describe --verbose <service_name>

    The --verbose flag is optional but can be included to provide a more detailed description. The difference between a regular and verbose output is shown in the following examples:

    Example output without --verbose flag

    Name:       hello
    Namespace:  default
    Age:        2m
    URL:        http://hello-default.apps.ocp.example.com
    
    Revisions:
      100%  @latest (hello-00001) [1] (2m)
            Image:  docker.io/openshift/hello-openshift (pinned to aaea76)
    
    Conditions:
      OK TYPE                   AGE REASON
      ++ Ready                   1m
      ++ ConfigurationsReady     1m
      ++ RoutesReady             1m

    Example output with --verbose flag

    Name:         hello
    Namespace:    default
    Annotations:  serving.knative.dev/creator=system:admin
                  serving.knative.dev/lastModifier=system:admin
    Age:          3m
    URL:          http://hello-default.apps.ocp.example.com
    Cluster:      http://hello.default.svc.cluster.local
    
    Revisions:
      100%  @latest (hello-00001) [1] (3m)
            Image:  docker.io/openshift/hello-openshift (pinned to aaea76)
            Env:    RESPONSE=Hello Serverless!
    
    Conditions:
      OK TYPE                   AGE REASON
      ++ Ready                   3m
      ++ ConfigurationsReady     3m
      ++ RoutesReady             3m

  • Describe a service in YAML format:

    $ kn service describe <service_name> -o yaml
  • Describe a service in JSON format:

    $ kn service describe <service_name> -o json
  • Print the service URL only:

    $ kn service describe <service_name> -o url

6.2.6. Verifying your serverless application deployment

To verify that your serverless application has been deployed successfully, you must get the application URL created by Knative, and then send a request to that URL and observe the output.

Note

OpenShift Serverless supports the use of both HTTP and HTTPS URLs, however the output from oc get ksvc will always print URLs using the http:// format.

Procedure

  1. Find the application URL:

    $ oc get ksvc <service_name>

    Example output

    NAME            URL                                        LATESTCREATED         LATESTREADY           READY   REASON
    event-delivery   http://event-delivery-default.example.com   event-delivery-4wsd2   event-delivery-4wsd2   True

  2. Make a request to your cluster and observe the output.

    Example HTTP request

    $ curl http://event-delivery-default.example.com

    Example HTTPS request

    $ curl https://event-delivery-default.example.com

    Example output

    Hello Serverless!

  3. Optional. If you receive an error relating to a self-signed certificate in the certificate chain, you can add the --insecure flag to the curl command to ignore the error:

    $ curl https://event-delivery-default.example.com --insecure

    Example output

    Hello Serverless!

    Important

    Self-signed certificates must not be used in a production deployment. This method is only for testing purposes.

  4. Optional. If your OpenShift Container Platform cluster is configured with a certificate that is signed by a certificate authority (CA) but not yet globally configured for your system, you can specify this with the curl command. The path to the certificate can be passed to the curl command by using the --cacert flag:

    $ curl https://event-delivery-default.example.com --cacert <file>

    Example output

    Hello Serverless!

6.2.7. Interacting with a serverless application using HTTP2 and gRPC

OpenShift Serverless supports only insecure or edge-terminated routes.

Insecure or edge-terminated routes do not support HTTP2 on OpenShift Container Platform. These routes also do not support gRPC because gRPC is transported by HTTP2.

If you use these protocols in your application, you must call the application using the ingress gateway directly. To do this you must find the ingress gateway’s public address and the application’s specific host.

Procedure

  1. Find the application host. See the instructions in Verifying your serverless application deployment.
  2. Find the ingress gateway’s public address:

    $ oc -n knative-serving-ingress get svc kourier

    Example output

    NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                                             PORT(S)                                                                                                                                      AGE
    kourier   LoadBalancer   172.30.51.103   a83e86291bcdd11e993af02b7a65e514-33544245.us-east-1.elb.amazonaws.com   80:31380/TCP,443:31390/TCP   67m

    The public address is surfaced in the EXTERNAL-IP field, and in this case is a83e86291bcdd11e993af02b7a65e514-33544245.us-east-1.elb.amazonaws.com.

  3. Manually set the host header of your HTTP request to the application’s host, but direct the request itself against the public address of the ingress gateway.

    $ curl -H "Host: hello-default.example.com" a83e86291bcdd11e993af02b7a65e514-33544245.us-east-1.elb.amazonaws.com

    Example output

    Hello Serverless!

    You can also make a gRPC request by setting the authority to the application’s host, while directing the request against the ingress gateway directly:

    grpc.Dial(
        "a83e86291bcdd11e993af02b7a65e514-33544245.us-east-1.elb.amazonaws.com:80",
        grpc.WithAuthority("hello-default.example.com:80"),
        grpc.WithInsecure(),
    )
    Note

    Ensure that you append the respective port, 80 by default, to both hosts as shown in the previous example.

6.2.8. Enabling communication with Knative applications on a cluster with restrictive network policies

If you are using a cluster that multiple users have access to, your cluster might use network policies to control which pods, services, and namespaces can communicate with each other over the network.

If your cluster uses restrictive network policies, it is possible that Knative system pods are not able to access your Knative application. For example, if your namespace has the following network policy, which denies all requests, Knative system pods cannot access your Knative application:

Example NetworkPolicy object that denies all requests to the namespace

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: deny-by-default
  namespace: example-namespace
spec:
  podSelector:
  ingress: []

To allow access to your applications from Knative system pods, you must add a label to each of the Knative system namespaces, and then create a NetworkPolicy object in your application namespace that allows access to the namespace for other namespaces that have this label.

Important

A network policy that denies requests to non-Knative services on your cluster still prevents access to these services. However, by allowing access from Knative system namespaces to your Knative application, you are allowing access to your Knative application from all namespaces in the cluster.

If you do not want to allow access to your Knative application from all namespaces on the cluster, you might want to use JSON Web Token authentication for Knative services instead (see the Knative Serving documentation). JSON Web Token authentication for Knative services requires Service Mesh.

Procedure

  1. Add the knative.openshift.io/system-namespace=true label to each Knative system namespace that requires access to your application:

    1. Label the knative-serving namespace:

      $ oc label namespace knative-serving knative.openshift.io/system-namespace=true
    2. Label the knative-serving-ingress namespace:

      $ oc label namespace knative-serving-ingress knative.openshift.io/system-namespace=true
    3. Label the knative-eventing namespace:

      $ oc label namespace knative-eventing knative.openshift.io/system-namespace=true
    4. Label the knative-kafka namespace:

      $ oc label namespace knative-kafka knative.openshift.io/system-namespace=true
  2. Create a NetworkPolicy object in your application namespace to allow access from namespaces with the knative.openshift.io/system-namespace label:

    Example NetworkPolicy object

    apiVersion: networking.k8s.io/v1
    kind: NetworkPolicy
    metadata:
      name: <network_policy_name> 1
      namespace: <namespace> 2
    spec:
      ingress:
      - from:
        - namespaceSelector:
            matchLabels:
              knative.openshift.io/system-namespace: "true"
      podSelector: {}
      policyTypes:
      - Ingress

    1
    Provide a name for your network policy.
    2
    The namespace where your application (Knative service) exists.

6.2.9. HTTPS redirection per service

You can enable or disable HTTPS redirection for a service by configuring the networking.knative.dev/httpOption annotation, as shown in the following example:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: example
  namespace: default
  annotations:
    networking.knative.dev/httpOption: "redirected"
spec:
  ...

6.2.10. Using kn CLI in offline mode

Important

The offline mode of the kn CLI is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see https://access.redhat.com/support/offerings/techpreview/.

6.2.10.1. About offline mode

Normally, when you execute kn service commands, the changes immediately propagate to the cluster. However, as an alternative, you can execute kn service commands in offline mode:

  1. When you create a service in offline mode, no changes happen on the cluster. Instead, the only thing that happens is the creation of the service descriptor file on your local machine.
  2. After the descriptor file is created, you can manually modify it and track it in a version control system.
  3. Finally, you can propagate changes to the cluster by using the kn service create -f, kn service apply -f, or oc apply -f commands on the descriptor files.

The offline mode has several uses:

  • You can manually modify the descriptor file before using it to make changes on the cluster.
  • You can locally track the descriptor file of a service in a version control system. This enables you to reuse the descriptor file in places other than the target cluster, for example in continuous integration (CI) pipelines, development environments, or demos.
  • You can examine the created descriptor files to learn about Knative services. In particular, you can see how the resulting service is influenced by the different arguments passed to the kn command.

The offline mode has its advantages: it is fast, and does not require a connection to the cluster. However, offline mode lacks server-side validation. Consequently, you cannot, for example, verify that the service name is unique or that the specified image can be pulled.

6.2.10.2. Creating a service using offline mode

Prerequisites

  • OpenShift Serverless Operator and Knative Serving are installed on your cluster.
  • You have installed the kn CLI.

Procedure

  1. In offline mode, create a local Knative service descriptor file:

    $ kn service create event-display \
        --image quay.io/openshift-knative/knative-eventing-sources-event-display:latest \
        --target ./ \
        --namespace test

    Example output

    Service 'event-display' created in namespace 'test'.

    • The --target ./ flag enables offline mode and specifies ./ as the directory for storing the new directory tree.

      If you do not specify an existing directory, but use a filename, such as --target my-service.yaml, then no directory tree is created. Instead, only the service descriptor file my-service.yaml is created in the current directory.

      The filename can have the .yaml, .yml, or .json extension. Choosing .json creates the service descriptor file in the JSON format.

    • The --namespace test option places the new service in the test namespace.

      If you do not use --namespace, and you are logged in to an OpenShift Container Platform cluster, the descriptor file is created in the current namespace. Otherwise, the descriptor file is created in the default namespace.

  2. Examine the created directory structure:

    $ tree ./

    Example output

    ./
    └── test
        └── ksvc
            └── event-display.yaml
    
    2 directories, 1 file

    • The current ./ directory specified with --target contains the new test/ directory that is named after the specified namespace.
    • The test/ directory contains the ksvc directory, named after the resource type.
    • The ksvc directory contains the descriptor file event-display.yaml, named according to the specified service name.
  3. Examine the generated service descriptor file:

    $ cat test/ksvc/event-display.yaml

    Example output

    apiVersion: serving.knative.dev/v1
    kind: Service
    metadata:
      creationTimestamp: null
      name: event-display
      namespace: test
    spec:
      template:
        metadata:
          annotations:
            client.knative.dev/user-image: quay.io/openshift-knative/knative-eventing-sources-event-display:latest
          creationTimestamp: null
        spec:
          containers:
          - image: quay.io/openshift-knative/knative-eventing-sources-event-display:latest
            name: ""
            resources: {}
    status: {}

  4. List information about the new service:

    $ kn service describe event-display --target ./ --namespace test

    Example output

    Name:       event-display
    Namespace:  test
    Age:
    URL:
    
    Revisions:
    
    Conditions:
      OK TYPE    AGE REASON

    • The --target ./ option specifies the root directory for the directory structure containing namespace subdirectories.

      Alternatively, you can directly specify a YAML or JSON filename with the --target option. The accepted file extensions are .yaml, .yml, and .json.

    • The --namespace option specifies the namespace, which communicates to kn the subdirectory that contains the necessary service descriptor file.

      If you do not use --namespace, and you are logged in to an OpenShift Container Platform cluster, kn searches for the service in the subdirectory that is named after the current namespace. Otherwise, kn searches in the default/ subdirectory.

  5. Use the service descriptor file to create the service on the cluster:

    $ kn service create -f test/ksvc/event-display.yaml

    Example output

    Creating service 'event-display' in namespace 'test':
    
      0.058s The Route is still working to reflect the latest desired specification.
      0.098s ...
      0.168s Configuration "event-display" is waiting for a Revision to become ready.
     23.377s ...
     23.419s Ingress has not yet been reconciled.
     23.534s Waiting for load balancer to be ready
     23.723s Ready to serve.
    
    Service 'event-display' created to latest revision 'event-display-00001' is available at URL:
    http://event-display-test.apps.example.com

6.3. Traffic management

A revision is a point-in-time snapshot of the code and configuration for each modification made to a Knative service. Each time the configuration of a service is updated, a new revision for the service is created. Revisions are immutable objects and can be retained for as long as they are required or used. Knative Serving revisions can be automatically scaled up and down according to incoming traffic.

You can manage traffic routing to different revisions of a Knative service by modifying the traffic spec of the service resource.

Knative service architecture

6.3.1. Traffic routing examples

When you create a Knative service, it does not have any default traffic spec settings. By setting the traffic spec, you can split traffic over any number of fixed revisions, or send traffic to the latest revision.

6.3.1.1. Traffic routing between multiple revisions

The following example shows how the list of revisions in the traffic spec can be extended so that traffic is split between multiple revisions.

This example sends 50% of traffic to the revision tagged as current, and 50% of traffic to the revision tagged as candidate:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: example-service
  namespace: default
spec:
...
  traffic:
  - tag: current
    revisionName: example-service-1
    percent: 50
  - tag: candidate
    revisionName: example-service-2
    percent: 50
  - tag: latest
    latestRevision: true
    percent: 0

6.3.1.2. Traffic routing to the latest revision

The following example shows a traffic spec where 100% of traffic is routed to the latest revision of the service. Under status, you can see the name of the latest revision that latestRevision resolves to:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: example-service
  namespace: default
spec:
...
  traffic:
  - latestRevision: true
    percent: 100
status:
  ...
  traffic:
  - percent: 100
    revisionName: example-service

6.3.1.3. Traffic routing to the current revision

The following example shows a traffic spec where 100% of traffic is routed to the revision tagged as current, and the name of that revision is specified as example-service. The revision tagged as latest is kept available, even though no traffic is routed to it:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: example-service
  namespace: default
spec:
...
  traffic:
  - tag: current
    revisionName: example-service
    percent: 100
  - tag: latest
    latestRevision: true
    percent: 0

6.3.2. Managing traffic between revisions by using the OpenShift Container Platform web console

After you create a serverless application, the application is displayed in the Topology view of the Developer perspective in the OpenShift Container Platform web console. The application revision is represented by the node, and the Knative service is indicated by a quadrilateral around the node.

Any new change in the code or the service configuration creates a new revision, which is a snapshot of the code at a given time. For a service, you can manage the traffic between the revisions of the service by splitting and routing it to the different revisions as required.

Procedure

To split traffic between multiple revisions of an application in the Topology view:

  1. Click the Knative service to see its overview in the side panel.
  2. Click the Resources tab, to see a list of Revisions and Routes for the service.

    Figure 6.1. Serverless application

    odc serverless app
  3. Click the service, indicated by the S icon at the top of the side panel, to see an overview of the service details.
  4. Click the YAML tab and modify the service configuration in the YAML editor, and click Save. For example, change the timeoutseconds from 300 to 301 . This change in the configuration triggers a new revision. In the Topology view, the latest revision is displayed and the Resources tab for the service now displays the two revisions.
  5. In the Resources tab, click Set Traffic Distribution to see the traffic distribution dialog box:

    1. Add the split traffic percentage portion for the two revisions in the Splits field.
    2. Add tags to create custom URLs for the two revisions.
    3. Click Save to see two nodes representing the two revisions in the Topology view.

      Figure 6.2. Serverless application revisions

      odc serverless revisions

6.3.3. Traffic management using the Knative CLI

You can use the kn service update command to split traffic between revisions of a service.

Example command

$ kn service update <service_name> --traffic <revision>=<percent>

Where:

  • <service_name> is the name of the Knative service that you are configuring traffic routing for.
  • <revision> is the revision that you want to configure to receive a percentage of traffic. You can either specify the name of the revision, or a tag that you assigned to the revision by using the --tag flag.
  • <percent> is the percentage of traffic that you want to send to the specified revision.

6.3.3.1. Multiple flags and order precedence

All traffic-related flags can be specified using a single kn service update command. kn defines the precedence of these flags. The order of the flags specified when using the command is not taken into account.

The precedence of the flags as they are evaluated by kn are:

  1. --untag: All the referenced revisions with this flag are removed from the traffic block.
  2. --tag: Revisions are tagged as specified in the traffic block.
  3. --traffic: The referenced revisions are assigned a portion of the traffic split.

    Important

    The --traffic flag can be specified multiple times in one command, and is valid only if the sum of the Percent values in all flags totals 100.

You can add tags to revisions and then split traffic according to the tags you have set.

6.3.3.2. Example traffic management commands

In the following command, the @latest tag means that blue resolves to the latest revision of the service:

$ kn service update example-service --tag green=revision-0001 --tag blue=@latest

After the green and blue tags have been applied, you can run the following command to split traffic for the service named example-service, by sending 80% of traffic to the revision green and 20% of traffic to the revision blue:

$ kn service update example-service --traffic green=80 --traffic blue=20

Alternatively, you could use the following command to send 80% of traffic to the latest revision and 20% to a revision named v1, without using tags:

$ kn service update example-service --traffic @latest=80 --traffic v1=20
Note

You can only use the identifier @latest once per command with the --traffic flag.

6.3.3.3. Knative CLI traffic management flags

The kn CLI supports traffic operations on the traffic block of a service as part of the kn service update command.

The following table displays a summary of traffic splitting flags, value formats, and the operation the flag performs. The Repetition column denotes whether repeating the particular value of flag is allowed in a kn service update command.

FlagValue(s)OperationRepetition

--traffic

RevisionName=Percent

Gives Percent traffic to RevisionName

Yes

--traffic

Tag=Percent

Gives Percent traffic to the revision having Tag

Yes

--traffic

@latest=Percent

Gives Percent traffic to the latest ready revision

No

--tag

RevisionName=Tag

Gives Tag to RevisionName

Yes

--tag

@latest=Tag

Gives Tag to the latest ready revision

No

--untag

Tag

Removes Tag from revision

Yes

6.3.3.4. Custom URLs for revisions

Assigning a --tag flag to a service by using the kn service update command creates a custom URL for the revision that is created when you update the service. The custom URL follows the pattern https://<tag>-<service_name>-<namespace>.<domain> or http://<tag>-<service_name>-<namespace>.<domain>.

The --tag and --untag flags use the following syntax:

  • Require one value.
  • Denote a unique tag in the traffic block of the service.
  • Can be specified multiple times in one command.
6.3.3.4.1. Example: Assign a tag to a revision

The following example assigns the tag latest to a revision named example-revision:

$ kn service update <service_name> --tag @latest=example-tag
6.3.3.4.2. Example: Remove a tag from a revision

You can remove a tag to remove the custom URL, by using the --untag flag.

Note

If a revision has its tags removed, and it is assigned 0% of the traffic, the revision is removed from the traffic block entirely.

The following command removes all tags from the revision named example-revision:

$ kn service update <service_name> --untag example-tag

6.3.4. Routing and managing traffic by using a blue-green deployment strategy

You can safely reroute traffic from a production version of an app to a new version, by using a blue-green deployment strategy.

Procedure

  1. Create and deploy an app as a Knative service.
  2. Find the name of the first revision that was created when you deployed the service, by viewing the output from the following command:

    $ oc get ksvc <service_name> -o=jsonpath='{.status.latestCreatedRevisionName}'

    Example command

    $ oc get ksvc example-service -o=jsonpath='{.status.latestCreatedRevisionName}'

    Example output

    $ example-service-00001

  3. Add the following YAML to the service spec to send inbound traffic to the revision:

    ...
    spec:
      traffic:
        - revisionName: <first_revision_name>
          percent: 100 # All traffic goes to this revision
    ...
  4. Verify that you can view your app at the URL output you get from running the following command:

    $ oc get ksvc <service_name>
  5. Deploy a second revision of your app by modifying at least one field in the template spec of the service and redeploying it. For example, you can modify the image of the service, or an env environment variable. You can redeploy the service by applying the service YAML file, or by using the kn service update command if you have installed the kn CLI.
  6. Find the name of the second, latest revision that was created when you redeployed the service, by running the command:

    $ oc get ksvc <service_name> -o=jsonpath='{.status.latestCreatedRevisionName}'

    At this point, both the first and second revisions of the service are deployed and running.

  7. Update your existing service to create a new, test endpoint for the second revision, while still sending all other traffic to the first revision:

    Example of updated service spec with test endpoint

    ...
    spec:
      traffic:
        - revisionName: <first_revision_name>
          percent: 100 # All traffic is still being routed to the first revision
        - revisionName: <second_revision_name>
          percent: 0 # No traffic is routed to the second revision
          tag: v2 # A named route
    ...

    After you redeploy this service by reapplying the YAML resource, the second revision of the app is now staged. No traffic is routed to the second revision at the main URL, and Knative creates a new service named v2 for testing the newly deployed revision.

  8. Get the URL of the new service for the second revision, by running the following command:

    $ oc get ksvc <service_name> --output jsonpath="{.status.traffic[*].url}"

    You can use this URL to validate that the new version of the app is behaving as expected before you route any traffic to it.

  9. Update your existing service again, so that 50% of traffic is sent to the first revision, and 50% is sent to the second revision:

    Example of updated service spec splitting traffic 50/50 between revisions

    ...
    spec:
      traffic:
        - revisionName: <first_revision_name>
          percent: 50
        - revisionName: <second_revision_name>
          percent: 50
          tag: v2
    ...

  10. When you are ready to route all traffic to the new version of the app, update the service again to send 100% of traffic to the second revision:

    Example of updated service spec sending all traffic to the second revision

    ...
    spec:
      traffic:
        - revisionName: <first_revision_name>
          percent: 0
        - revisionName: <second_revision_name>
          percent: 100
          tag: v2
    ...

    Tip

    You can remove the first revision instead of setting it to 0% of traffic if you do not plan to roll back the revision. Non-routeable revision objects are then garbage-collected.

  11. Visit the URL of the first revision to verify that no more traffic is being sent to the old version of the app.

6.4. Using OpenShift Logging

6.4.1. About deploying OpenShift Logging

OpenShift Container Platform cluster administrators can deploy OpenShift Logging using the OpenShift Container Platform web console or CLI to install the OpenShift Elasticsearch Operator and Red Hat OpenShift Logging Operator. When the operators are installed, you create a ClusterLogging custom resource (CR) to schedule OpenShift Logging pods and other resources necessary to support OpenShift Logging. The operators are responsible for deploying, upgrading, and maintaining OpenShift Logging.

The ClusterLogging CR defines a complete OpenShift Logging environment that includes all the components of the logging stack to collect, store and visualize logs. The Red Hat OpenShift Logging Operator watches the OpenShift Logging CR and adjusts the logging deployment accordingly.

Administrators and application developers can view the logs of the projects for which they have view access.

6.4.2. About deploying and configuring OpenShift Logging

OpenShift Logging is designed to be used with the default configuration, which is tuned for small to medium sized OpenShift Container Platform clusters.

The installation instructions that follow include a sample ClusterLogging custom resource (CR), which you can use to create a OpenShift Logging instance and configure your OpenShift Logging environment.

If you want to use the default OpenShift Logging install, you can use the sample CR directly.

If you want to customize your deployment, make changes to the sample CR as needed. The following describes the configurations you can make when installing your OpenShift Logging instance or modify after installation. See the Configuring sections for more information on working with each component, including modifications you can make outside of the ClusterLogging custom resource.

6.4.2.1. Configuring and Tuning OpenShift Logging

You can configure your OpenShift Logging environment by modifying the ClusterLogging custom resource deployed in the openshift-logging project.

You can modify any of the following components upon install or after install:

Memory and CPU
You can adjust both the CPU and memory limits for each component by modifying the resources block with valid memory and CPU values:
spec:
  logStore:
    elasticsearch:
      resources:
        limits:
          cpu:
          memory: 16Gi
        requests:
          cpu: 500m
          memory: 16Gi
      type: "elasticsearch"
  collection:
    logs:
      fluentd:
        resources:
          limits:
            cpu:
            memory:
          requests:
            cpu:
            memory:
        type: "fluentd"
  visualization:
    kibana:
      resources:
        limits:
          cpu:
          memory:
        requests:
          cpu:
          memory:
      type: kibana
Elasticsearch storage
You can configure a persistent storage class and size for the Elasticsearch cluster using the storageClass name and size parameters. The Red Hat OpenShift Logging Operator creates a persistent volume claim (PVC) for each data node in the Elasticsearch cluster based on these parameters.
  spec:
    logStore:
      type: "elasticsearch"
      elasticsearch:
        nodeCount: 3
        storage:
          storageClassName: "gp2"
          size: "200G"

This example specifies each data node in the cluster will be bound to a PVC that requests "200G" of "gp2" storage. Each primary shard will be backed by a single replica.

Note

Omitting the storage block results in a deployment that includes ephemeral storage only.

  spec:
    logStore:
      type: "elasticsearch"
      elasticsearch:
        nodeCount: 3
        storage: {}
Elasticsearch replication policy

You can set the policy that defines how Elasticsearch shards are replicated across data nodes in the cluster:

  • FullRedundancy. The shards for each index are fully replicated to every data node.
  • MultipleRedundancy. The shards for each index are spread over half of the data nodes.
  • SingleRedundancy. A single copy of each shard. Logs are always available and recoverable as long as at least two data nodes exist.
  • ZeroRedundancy. No copies of any shards. Logs may be unavailable (or lost) in the event a node is down or fails.

6.4.2.2. Sample modified ClusterLogging custom resource

The following is an example of a ClusterLogging custom resource modified using the options previously described.

Sample modified ClusterLogging custom resource

apiVersion: "logging.openshift.io/v1"
kind: "ClusterLogging"
metadata:
  name: "instance"
  namespace: "openshift-logging"
spec:
  managementState: "Managed"
  logStore:
    type: "elasticsearch"
    retentionPolicy:
      application:
        maxAge: 1d
      infra:
        maxAge: 7d
      audit:
        maxAge: 7d
    elasticsearch:
      nodeCount: 3
      resources:
        limits:
          memory: 32Gi
        requests:
          cpu: 3
          memory: 32Gi
        storage:
          storageClassName: "gp2"
          size: "200G"
      redundancyPolicy: "SingleRedundancy"
  visualization:
    type: "kibana"
    kibana:
      resources:
        limits:
          memory: 1Gi
        requests:
          cpu: 500m
          memory: 1Gi
      replicas: 1
  collection:
    logs:
      type: "fluentd"
      fluentd:
        resources:
          limits:
            memory: 1Gi
          requests:
            cpu: 200m
            memory: 1Gi

6.4.3. Using OpenShift Logging to find logs for Knative Serving components

Procedure

  1. Get the Kibana route:

    $ oc -n openshift-logging get route kibana
  2. Use the route’s URL to navigate to the Kibana dashboard and log in.
  3. Check that the index is set to .all. If the index is not set to .all, only the OpenShift Container Platform system logs will be listed.
  4. Filter the logs by using the knative-serving namespace. Enter kubernetes.namespace_name:knative-serving in the search box to filter results.
Note

Knative Serving uses structured logging by default. You can enable the parsing of these logs by customizing the OpenShift Logging Fluentd settings. This makes the logs more searchable and enables filtering on the log level to quickly identify issues.

6.4.4. Using OpenShift Logging to find logs for services deployed with Knative Serving

With OpenShift Logging, the logs that your applications write to the console are collected in Elasticsearch. The following procedure outlines how to apply these capabilities to applications deployed by using Knative Serving.

Procedure

  1. Get the Kibana route:

    $ oc -n openshift-logging get route kibana
  2. Use the route’s URL to navigate to the Kibana dashboard and log in.
  3. Check that the index is set to .all. If the index is not set to .all, only the OpenShift system logs will be listed.
  4. Filter the logs by using the knative-serving namespace. Enter a filter for the service in the search box to filter results.

    Example filter

    kubernetes.namespace_name:default AND kubernetes.labels.serving_knative_dev\/service:{service_name}

    You can also filter by using /configuration or /revision.

  5. Narrow your search by using kubernetes.container_name:<user_container> to only display the logs generated by your application. Otherwise, you will see logs from the queue-proxy.
Note

Use JSON-based structured logging in your application to allow for the quick filtering of these logs in production environments.

6.5. Tracing requests using Jaeger

Using Jaeger with OpenShift Serverless allows you to enable distributed tracing for your serverless applications on OpenShift Container Platform.

Distributed tracing records the path of a request through the various services that make up an application.

It is used to tie information about different units of work together, to understand a whole chain of events in a distributed transaction. The units of work might be executed in different processes or hosts.

Developers can visualize call flows in large architectures with distributed tracing. which is useful for understanding serialization, parallelism, and sources of latency.

For more information about Jaeger, see Jaeger architecture and Installing Jaeger.

6.5.1. Configuring Jaeger for use with OpenShift Serverless

Prerequisites

To configure Jaeger for use with OpenShift Serverless, you will need:

  • Cluster administrator permissions on an OpenShift Container Platform cluster.
  • A current installation of OpenShift Serverless Operator and Knative Serving.
  • A current installation of the Jaeger Operator.

Procedure

  1. Create and apply a Jaeger custom resource (CR) that contains the following:

    Jaeger CR

    apiVersion: jaegertracing.io/v1
    kind: Jaeger
    metadata:
      name: jaeger
      namespace: default

  2. Enable tracing for Knative Serving, by editing the KnativeServing CR and adding a YAML configuration for tracing:

    Tracing YAML example

    apiVersion: operator.knative.dev/v1alpha1
    kind: KnativeServing
    metadata:
      name: knative-serving
      namespace: knative-serving
    spec:
      config:
        tracing:
          sample-rate: "0.1" 1
          backend: zipkin 2
          zipkin-endpoint: http://jaeger-collector.default.svc.cluster.local:9411/api/v2/spans 3
          debug: "false" 4

    1
    The sample-rate defines sampling probability. Using sample-rate: "0.1" means that 1 in 10 traces will be sampled.
    2
    backend must be set to zipkin.
    3
    The zipkin-endpoint must point to your jaeger-collector service endpoint. To get this endpoint, substitute the namespace where the Jaeger CR is applied.
    4
    Debugging should be set to false. Enabling debug mode by setting debug: "true" allows all spans to be sent to the server, bypassing sampling.

Verification

You can access the Jaeger web console to see tracing data, by using the jaeger route.

  1. Get the jaeger route’s hostname by entering the following command:

    $ oc get route jaeger

    Example output

    NAME     HOST/PORT                         PATH   SERVICES       PORT    TERMINATION   WILDCARD
    jaeger   jaeger-default.apps.example.com          jaeger-query   <all>   reencrypt     None

  2. Open the endpoint address in your browser to view the console.

6.6. Configuring routes for Knative services

Knative leverages OpenShift Container Platform TLS termination to provide routing for Knative services. When a Knative service is created, a OpenShift Container Platform route is automatically created for the service. This route is managed by the OpenShift Serverless Operator. The OpenShift Container Platform route exposes the Knative service through the same domain as the OpenShift Container Platform cluster.

You can disable Operator control of OpenShift Container Platform routing so that you can configure a Knative route to directly use your TLS certificates instead.

Knative routes can also be used alongside the OpenShift Container Platform route to provide additional fine-grained routing capabilities, such as traffic splitting.

6.6.1. Customizing labels and annotations for OpenShift Container Platform routes

OpenShift Container Platform routes support the use of custom labels and annotations, which you can configure by modifying the metadata spec of a Knative service. Custom labels and annotations are propagated from the service to the Knative route, then to the Knative ingress, and finally to the OpenShift Container Platform route.

Prerequisites

  • You must have the OpenShift Serverless Operator and Knative Serving installed on your OpenShift Container Platform cluster.

Procedure

  1. Create a Knative service that contains the label or annotation that you want to propagate to the OpenShift Container Platform route:

    • To create a service by using YAML:

      Example service created by using YAML

      apiVersion: serving.knative.dev/v1
      kind: Service
      metadata:
        name: <service_name>
        labels:
          <label_name>: <label_value>
        annotations:
          <annotation_name>: <annotation_value>
      ...

    • To create a service by using the kn CLI, enter:

      Example service created by using a kn command

      $ kn service create <service_name> \
        --image=<image> \
        --annotation <annotation_name>=<annotation_value> \
        --label <label_value>=<label_value>

  2. Verify that the OpenShift Container Platform route has been created with the annotation or label that you added by inspecting the output from the following command:

    Example command for verification

    $ oc get routes.route.openshift.io \
         -l serving.knative.openshift.io/ingressName=<service_name> \ 1
         -l serving.knative.openshift.io/ingressNamespace=<service_namespace> \ 2
         -n knative-serving-ingress -o yaml \
             | grep -e "<label_name>: \"<label_value>\""  -e "<annotation_name>: <annotation_value>" 3

    1
    Use the name of your service.
    2
    Use the namespace where your service was created.
    3
    Use your values for the label and annotation names and values.

6.6.2. Configuring OpenShift Container Platform routes for Knative services

If you want to configure a Knative service to use your TLS certificate on OpenShift Container Platform, you must disable the automatic creation of a route for the service by the OpenShift Serverless Operator and instead manually create a route for the service.

Note

When you complete the following procedure, the default OpenShift Container Platform route in the knative-serving-ingress namespace is not created. However, the Knative route for the application is still created in this namespace.

Prerequisites

  • The OpenShift Serverless Operator and Knative Serving component must be installed on your OpenShift Container Platform cluster.
Note

You must modify the replaceable values in the example commands for the following procedure.

Procedure

  1. Create a Knative service that includes the serving.knative.openshift.io/disableRoute=true annotation:

    Important

    The serving.knative.openshift.io/disableRoute=true annotation instructs OpenShift Serverless to not automatically create a route for you. However, the service still shows a URL and reaches a status of Ready. This URL does not work externally until you create your own route with the same hostname as the hostname in the URL.

    1. Create a Knative Service resource:

      Example resource

      apiVersion: serving.knative.dev/v1
      kind: Service
      metadata:
        name: <service_name>
        annotations:
          serving.knative.openshift.io/disableRoute: "true"
      spec:
        template:
          spec:
            containers:
            - image: <image>
      ...

    2. Apply the Service resource:

      $ oc apply -f <filename>
    3. Optional. Create a Knative service by using the kn service create command:

      Example kn command

      $ kn service create <service_name> \
        --image=gcr.io/knative-samples/helloworld-go \
        --annotation serving.knative.openshift.io/disableRoute=true

  2. Verify that no OpenShift Container Platform route has been created for the service:

    Example command

    $ $ oc get routes.route.openshift.io \
      -l serving.knative.openshift.io/ingressName=$KSERVICE_NAME \
      -l serving.knative.openshift.io/ingressNamespace=$KSERVICE_NAMESPACE \
      -n knative-serving-ingress

    You will see the following output:

    No resources found in knative-serving-ingress namespace.
  3. Create a Route resource in the knative-serving-ingress namespace:

    apiVersion: route.openshift.io/v1
    kind: Route
    metadata:
      annotations:
        haproxy.router.openshift.io/timeout: 600s 1
      name: <route_name> 2
      namespace: knative-serving-ingress 3
    spec:
      host: <service_host> 4
      port:
        targetPort: http2
      to:
        kind: Service
        name: kourier
        weight: 100
      tls:
        insecureEdgeTerminationPolicy: Allow
        termination: edge 5
        key: |-
          -----BEGIN PRIVATE KEY-----
          [...]
          -----END PRIVATE KEY-----
        certificate: |-
          -----BEGIN CERTIFICATE-----
          [...]
          -----END CERTIFICATE-----
        caCertificate: |-
          -----BEGIN CERTIFICATE-----
          [...]
          -----END CERTIFICATE----
      wildcardPolicy: None
    1
    The timeout value for the OpenShift Container Platform route. You must set the same value as the max-revision-timeout-seconds setting (600s by default).
    2
    The name of the OpenShift Container Platform route.
    3
    The namespace for the OpenShift Container Platform route. This must be knative-serving-ingress.
    4
    The hostname for external access. You can set this to <service_name>-<service_namespace>.<domain>.
    5
    The certificates you want to use. Currently, only edge termination is supported.
  4. Apply the Route resource:

    $ oc apply -f <filename>

6.6.3. Setting cluster availability to cluster local

By default, Knative services are published to a public IP address. Being published to a public IP address means that Knative services are public applications, and have a publicly accessible URL.

Publicly accessible URLs are accessible from outside of the cluster. However, developers may need to build back-end services that are only be accessible from inside the cluster, known as private services. Developers can label individual services in the cluster with the networking.knative.dev/visibility=cluster-local label to make them private.

Important

For OpenShift Serverless 1.15.0 and newer versions, the serving.knative.dev/visibility label is no longer available. You must update existing services to use the networking.knative.dev/visibility label instead.

Procedure

  • Set the visibility for your service by adding the networking.knative.dev/visibility=cluster-local label:

    $ oc label ksvc <service_name> networking.knative.dev/visibility=cluster-local

Verification

  • Check that the URL for your service is now in the format http://<service_name>.<namespace>.svc.cluster.local, by entering the following command and reviewing the output:

    $ oc get ksvc

    Example output

    NAME            URL                                                                         LATESTCREATED     LATESTREADY       READY   REASON
    hello           http://hello.default.svc.cluster.local                                      hello-tx2g7       hello-tx2g7       True

6.6.4. Additional resources

6.7. Monitoring Knative services

You can use the OpenShift Container Platform monitoring stack to record and view health checks and metrics for your Knative services. This section describes the following topics:

  • What metrics Knative services expose by default
  • How to configure exposing custom metrics
  • How to configure the monitoring stack to scrape the exposed metrics
  • How to view the metrics of a service
Note

Scraping the metrics does not affect autoscaling of a Knative service, because scraping requests do not go through the activator. Consequently, no scraping takes place if no pods are running.

Additional resources

6.7.1. Knative service metrics exposed by default

Table 6.1. Metrics exposed by default for each Knative service on port 9090

Metric name, unit, and typeDescriptionMetric tags

queue_requests_per_second

Metric unit: dimensionless

Metric type: gauge

Number of requests per second that hit the queue proxy.

Formula: stats.RequestCount / r.reportingPeriodSeconds

stats.RequestCount is calculated directly from the networking pkg stats for the given reporting duration.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

queue_proxied_operations_per_second

Metric unit: dimensionless

Metric type: gauge

Number of proxied requests per second.

Formula: stats.ProxiedRequestCount / r.reportingPeriodSeconds

stats.ProxiedRequestCount is calculated directly from the networking pkg stats for the given reporting duration.

 

queue_average_concurrent_requests

Metric unit: dimensionless

Metric type: gauge

Number of requests currently being handled by this pod.

Average concurrency is calculated at the networking pkg side as follows:

  • When a req change happens, the time delta between changes is calculated. Based on the result, the current concurrency number over delta is computed and added to the current computed concurrency. Additionally, a sum of the deltas is kept.

    Current concurrency over delta is computed as follows:

    global_concurrency × delta

  • Each time a reporting is done, the sum and current computed concurrency are reset.
  • When reporting the average concurrency the current computed concurrency is divided by the sum of deltas.
  • When a new request comes in, the global concurrency counter is increased. When a request is completed, the counter is decreased.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

queue_average_proxied_concurrent_requests

Metric unit: dimensionless

Metric type: gauge

Number of proxied requests currently being handled by this pod:

stats.AverageProxiedConcurrency

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

process_uptime

Metric unit: seconds

Metric type: gauge

The number of seconds that the process has been up.

destination_configuration="event-display", destination_namespace="pingsource1", destination_pod="event-display-00001-deployment-6b455479cb-75p6w", destination_revision="event-display-00001"

Table 6.2. Metrics exposed by default for each Knative service on port 9091

Metric name, unit, and typeDescriptionMetric tags

request_count

Metric unit: dimensionless

Metric type: counter

The number of requests that are routed to queue-proxy.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

request_latencies

Metric unit: milliseconds

Metric type: histogram

The response time in milliseconds.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

app_request_count

Metric unit: dimensionless

Metric type: counter

The number of requests that are routed to user-container.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

app_request_latencies

Metric unit: milliseconds

Metric type: histogram

The response time in milliseconds.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

queue_depth

Metric unit: dimensionless

Metric type: gauge

The current number of items in the serving and waiting queue, or not reported if unlimited concurrency. breaker.inFlight is used.

configuration_name="event-display", container_name="queue-proxy", namespace_name="apiserversource1", pod_name="event-display-00001-deployment-658fd4f9cf-qcnr5", response_code="200", response_code_class="2xx", revision_name="event-display-00001", service_name="event-display"

6.7.2. Knative service with custom application metrics

You can extend the set of metrics exported by a Knative service. The exact implementation depends on your application and the language used.

The following listing implements a sample Go application that exports the count of processed events custom metric.

package main

import (
  "fmt"
  "log"
  "net/http"
  "os"

  "github.com/prometheus/client_golang/prometheus" 1
  "github.com/prometheus/client_golang/prometheus/promauto"
  "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
  opsProcessed = promauto.NewCounter(prometheus.CounterOpts{ 2
     Name: "myapp_processed_ops_total",
     Help: "The total number of processed events",
  })
)


func handler(w http.ResponseWriter, r *http.Request) {
  log.Print("helloworld: received a request")
  target := os.Getenv("TARGET")
  if target == "" {
     target = "World"
  }
  fmt.Fprintf(w, "Hello %s!\n", target)
  opsProcessed.Inc() 3
}

func main() {
  log.Print("helloworld: starting server...")

  port := os.Getenv("PORT")
  if port == "" {
     port = "8080"
  }

  http.HandleFunc("/", handler)

  // Separate server for metrics requests
  go func() { 4
     mux := http.NewServeMux()
     server := &http.Server{
        Addr: fmt.Sprintf(":%s", "9095"),
        Handler: mux,
     }
     mux.Handle("/metrics", promhttp.Handler())
     log.Printf("prometheus: listening on port %s", 9095)
     log.Fatal(server.ListenAndServe())
  }()

   // Use same port as normal requests for metrics
  //http.Handle("/metrics", promhttp.Handler()) 5
  log.Printf("helloworld: listening on port %s", port)
  log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}
1
Including the Prometheus packages.
2
Defining the opsProcessed metric.
3
Incrementing the opsProcessed metric.
4
Configuring to use a separate server for metrics requests.
5
Configuring to use the same port as normal requests for metrics and the metrics subpath.

6.7.3. Configuration for scraping custom metrics

Custom metrics scraping is performed by an instance of Prometheus purposed for user workload monitoring. After you enable user workload monitoring and create the application, you need a configuration that defines how the monitoring stack will scrape the metrics.

The following sample configuration defines the ksvc for your application and configures the service monitor. The exact configuration depends on your application and how it exports the metrics.

apiVersion: serving.knative.dev/v1 1
kind: Service
metadata:
  name: helloworld-go
spec:
  template:
    metadata:
      labels:
        app: helloworld-go
      annotations:
    spec:
      containers:
      - image: docker.io/skonto/helloworld-go:metrics
        resources:
          requests:
            cpu: "200m"
        env:
        - name: TARGET
          value: "Go Sample v1"
---
apiVersion: monitoring.coreos.com/v1 2
kind: ServiceMonitor
metadata:
  labels:
  name: helloworld-go-sm
spec:
  endpoints:
  - port: queue-proxy-metrics
    scheme: http
  - port: app-metrics
    scheme: http
  namespaceSelector: {}
  selector:
    matchLabels:
       name:  helloworld-go-sm
---
apiVersion: v1 3
kind: Service
metadata:
  labels:
    name:  helloworld-go-sm
  name:  helloworld-go-sm
spec:
  ports:
  - name: queue-proxy-metrics
    port: 9091
    protocol: TCP
    targetPort: 9091
  - name: app-metrics
    port: 9095
    protocol: TCP
    targetPort: 9095
  selector:
    serving.knative.dev/service: helloworld-go
  type: ClusterIP
1
Application specification.
2
Configuration of which application’s metrics are scraped.
3
Configuration of the way metrics are scraped.

6.7.4. Examining metrics of a service

After you have configured the application to export the metrics and the monitoring stack to scrape them, you can examine the metrics in the web console.

Procedure

  1. Optional: Run requests against your application that you will be able to see in the metrics:

    $ hello_route=$(oc get ksvc helloworld-go -n ns1 -o jsonpath='{.status.url}') && \
        curl $hello_route

    Example output

    Hello Go Sample v1!

  2. In the web console, navigate to the MonitoringMetrics interface.
  3. In the input field, enter the query for the metric you want to observe, for example:

    revision_app_request_count{namespace="ns1", job="helloworld-go-sm"}

    Another example:

    myapp_processed_ops_total{namespace="ns1", job="helloworld-go-sm"}
  4. Observe the visualized metrics:

    Observing metrics of a service
    Observing metrics of a service

6.7.5. Examining metrics of a service in the dashboard

You can examine the metrics using a dedicated dashboard that aggregates queue proxy metrics by namespace.

Procedure

  1. In the web console, navigate to the MonitoringMetrics interface.
  2. Select the Knative User Services (Queue Proxy metrics) dashboard.
  3. Select the Namespace, Configuration, and Revision that correspond to your application.
  4. Observe the visualized metrics:

    Observing metrics of a service using a dashboard

6.8. Metrics

Metrics enable developers to monitor how Knative services are performing.

6.8.1. Prerequisites

  • To view metrics for Knative components on OpenShift Container Platform, you need access to the web console Developer perspective.
Warning

If Service Mesh is enabled with mTLS, metrics for Knative Serving are disabled by default because Service Mesh prevents Prometheus from scraping metrics.

For information about resolving this issue, see Integrating Service Mesh with OpenShift Serverless.

6.8.2. Queue proxy metrics

Each Knative service has a proxy container that proxies the connections to the application container. A number of metrics are reported for the queue proxy performance.

You can use the following metrics to measure if requests are queued at the proxy side and the actual delay in serving requests at the application side.

Metric nameDescriptionTypeTagsUnit

revision_request_count

The number of requests that are routed to queue-proxy pod.

Counter

configuration_name, container_name, namespace_name, pod_name, response_code, response_code_class, revision_name, service_name

Integer (no units)

revision_request_latencies

The response time of revision requests.

Histogram

configuration_name, container_name, namespace_name, pod_name, response_code, response_code_class, revision_name, service_name

Milliseconds

revision_app_request_count

The number of requests that are routed to the user-container pod.

Counter

configuration_name, container_name, namespace_name, pod_name, response_code, response_code_class, revision_name, service_name

Integer (no units)

revision_app_request_latencies

The response time of revision app requests.

Histogram

configuration_name, namespace_name, pod_name, response_code, response_code_class, revision_name, service_name

Milliseconds

revision_queue_depth

The current number of items in the serving and waiting queues. This metric is not reported if unlimited concurrency is configured.

Gauge

configuration_name, event-display, container_name, namespace_name, pod_name, response_code_class, revision_name, service_name

Integer (no units)