Kubernetes privilege escalation and access to sensitive information in OpenShift products and services - CVE-2018-1002105

Impacted Products

Red Hat Product Security has rated CVE-2018-1002105 as having a security impact of Critical.

The following Red Hat Product versions are impacted:

• Red Hat OpenShift Container Platform 3.x
• Red Hat OpenShift Online
• Red Hat OpenShift Dedicated

Attack Description and Impact

The OpenShift API server is a component which handles the API requests for OpenShift Container Platform 3. The OpenShift API server had a vulnerability in the ‘upgradeawarehandler’ type which acts as a reverse proxy to the compute nodes and other services added as API Extensions. A flaw in the reverse proxy prevented it from closing connections from the downstream service when there was an error. This allowed the user making the API call to escalate their privileges.

There are 2 ways to use this vulnerability to attack OpenShift Container Platform, OpenShift Online, or OpenShift Dedicated. The first involves abusing pod exec, attach, or portforward privileges granted to a normal user, and the second involves attacking the API extensions feature which provides the service catalog and access to additional features in OpenShift Container Platform 3.6 and later.

If a user has pod exec/attach/portforward privileges for any pod, their privileges can be escalated to cluster-admin, and any API call to a compute node Kubelet api can be achieved. An API call to the Kubelet api allows the user to exec into any container running on the same node as the pod which they have privileges for, including privileged containers which have read/write access to the host filesystem.

The second attack doesn’t require any privileges and exploits the API extension feature used by ‘metrics-server’ and ‘servicecatalog’ in OpenShift Container Platform, OpenShift Online, and Dedicated. It allows an unauthenticated user to gain cluster-admin privileges to any API extension deployed to the cluster including ‘servicecatalog’. Cluster-admin access to ‘servicecatalog’ allows creation of service brokers in any namespace and on any node. 

Diagnose your vulnerability

To check the version of OpenShift container platform installed send an http request to the API server, such as (Note: The URL should be the same as the webconsole URL):

   curl https://openshift.example.com:8443/version/openshift | grep gitVersion

*port is configuration specific

Alternatively if you are logged in with the ‘oc’ command you can check the version with this command:

   oc version

Any versions of OpenShift Container Platform older than those listed below are vulnerable:

• v3.11.43-1
• v3.10.72-1
• v3.9.51-1
• v3.8.44-1
• v3.7.72-1
• v3.6.173.0.140-1
• v3.5.5.31.80-1
• v3.4.1.44.57-1
• v3.3.1.46.45-1
• v3.2.1.34-2

Take Action

Customers running affected versions of Red Hat products are strongly recommended to update them as soon as errata are available.  

OpenShift Online (Starter, Pro) have been remediated.  OpenShift Dedicated customers should speak with their support contact to confirm the status/schedule for these fixes. 

Updates for Affected Products

Mitigation

Fixes for OpenShift Container Platform versions 3.2 and higher have been shipped to mitigate the flaw.   

Users who update to the latest versions do not need to apply further mitigations. However, if the updates cannot be applied, mitigating the issue is recommended. 

Mitigate the pod exec/attach/port-forward attack (OCP 3.2->3.11)

Change the default admin, and edit roles which have pod attach/exec/port-forward:

$oc edit clusterrole admin

Add the ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, and remove the pod attach/exec/portforward permissions. Example:

apiVersion: v1
kind: ClusterRole
metadata:
   annotations:
      openshift.io/description: A user that has edit rights within the project and can
        change the project's membership.
   creationTimestamp: 2018-11-21T01:34:04Z
   name: admin
   resourceVersion: "148192"
   selfLink: /oapi/v1/clusterroles/admin
   uid: 87dd0306-ed2d-11e8-9456-fa163e65ec84
rules:
- apiGroups:
  - ""
   attributeRestrictions: null
   resources:
  - pods
  - pods/proxy
   verbs:

Do the same for the ‘edit’ role which also has these permissions.

For a single user in a project:

Check the roles assigned to the user. In this case, the admin role is assigned to the quicklab user:
 
$ oc project myproj
$ oc adm policy who-can get pods/exec
Namespace: test
Verb:      get
Resource:  pods/exec
 
Users:  quicklab
        system:admin
        system:serviceaccount:kube-system:generic-garbage-collector
        system:serviceaccount:kube-system:namespace-controller
 
Groups: system:cluster-admins
        system:masters
 
 
$ oc get rolebindings
NAME                    ROLE                    USERS      GROUPS                          SERVICE ACCOUNTS   SUBJECTS
admin           /admin          quicklab                                                      
system:deployers        /system:deployer                                                   deployer           
system:image-builders   /system:image-builder                                              builder            
system:image-pullers    /system:image-puller               system:serviceaccounts:myproj
$ oc export clusterrole admin > admin-mitigate.yml

Edit admin-mitigate.yml by adding a ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, changing the name and removing the pod attach/exec/portforward permissions. Example:

apiVersion: v1
kind: ClusterRole
metadata:
  annotations:
    openshift.io/description: A user that has edit rights within the project and can
      change the project's membership.
    rbac.authorization.kubernetes.io/autoupdate: "false"
  creationTimestamp: null
  name: admin-mitigate
rules:
- apiGroups:
  - ""
  attributeRestrictions: null
  resources:
  - pods
  - pods/proxy
  verbs:
   ...

Create a new role with the new configuration:

$ cat admin-mitigate.yml | oc create -f -

Assign this role to any users who you don’t want to have those permissions:

$ oc adm policy remove-role-from-user admin quicklab
role "admin" removed: "quicklab"
$oc adm policy add-role-to-user admin-mitigate quicklab
role “admin-mitigate’”added: “quicklab”

Mitigating the API Extension attack (OCP 3.6 -> 3.11)

To mitigate the API extension attack until such time that a cluster upgrade to a patched version can be completed, it is possible to delete the affected API Extensions. Please review each services carefully to ensure this will not cause a loss of critical functionality, in which case alternate protections such as network level restrictions to known sources may be more appropriate.

Get a list of services which need to be disabled:

$ oc get apiservices -o=custom-columns=NAME:.metadata.name,SERVICE:.spec.service,STATUS:.status.conditions[0].type,VALUE:.status.conditions[0].status | grep -v '<nil>'

For example, on OCP 3.11 the services might include:

NAME	SERVICE	STATUS	VALUE
v1beta1.servicecatalog.k8s.io	map[name:apiserver namespace:kube-service-catalog]	Available	True

Disable the services listed in the output, for example:

$ oc delete svc apiserver -n kube-service-catalog

For this example, to re-enable service-catalog run the openshift-service-catalog playbook. Please refer to the product documentation to re-enable other services.

$ ansible-playbook [-i /path/to/inventory]
/usr/share/ansible/openshift-ansible/playbooks/openshift-service-catalog/config.yml