Kubernetes privilege escalation and access to sensitive information in OpenShift products and services - CVE-2018-1002105
Updated
A flaw has been detected in kubernetes which allows privilege escalation and access to sensitive information in OpenShift products and services. This issue has been assigned CVE-2018-1002105 and has a security impact of Critical.
All 3.x versions of OpenShift Container Platform allow for compromise of pods (multiple running container instances) running on a compute node to which a pod is scheduled with normal user privilege. This access could include access to all secrets, pods, environment variables, running pod/container processes, and persistent volumes.
Additionally, on OpenShift Container Platform versions 3.6 and higher, this vulnerability allows cluster-admin level access to any API hosted by an aggregated API server. This includes the ‘metrics-service’ and ‘servicecatalog’ as possible targets. Cluster-admin level access to the service catalog allows creation of brokered services by an unauthenticated user with escalated privileges in any namespace and on any node. This could allow an attacker to deploy malicious code, or alter existing brokered services.
For OpenShift Dedicated environments, a regular user, with pod exec/attach/portforward permissions, can gain cluster-level administrative privileges on any compute node that can run that pod. This includes exec access to all running workloads, all current secrets, logs, etc..
Background Information
The OpenShift API Server is included in all OpenShift Container Platform installations and handles all the administration tasks for the cluster. Administrators and developers don’t usually call the API directly, but use the ‘oc’ binary, which calls the API server.
The API server provides various functionality, including the following oc commands:
- oc exec
- oc port-forward
- oc rsh
It does this by acting as a reverse proxy to the kubelet running on the compute nodes. When connecting to the kubelet in order to fulfill any of the above commands, it opens a websocket connection which connects stdin, stdout, or stderr to the administrator or developer’s original call. For more information, read the Remote Commands section of the OpenShift Container Platform 3.11 architecture guide.
The API Server also acts as a reverse proxy when implementing the API Aggregation feature of Kubernetes. API aggregation enables the installation of additional Application Programming Interfaces (APIs) into the core API Server. Those additional APIs are referred to as API extensions by upstream Kubernetes. API Extensions allow an architect to extend the features of OpenShift Container Platform 3.
Acknowledgements
Red Hat would like to thank the Kubernetes Product Security Team for reporting this issue. Upstream acknowledges Darren Shepherd as the original reporter.
Additional References
Video: Kubernetes Privilege Flaw Escallation Explained by Red Hat
Blog: The Kubernetes privilege escalation flaw: Innovation still needs IT security expertise
Blog: Understanding the critical Kubernetes privilege escalation flaw in OpenShift 3
Impacted Products
Red Hat Product Security has rated CVE-2018-1002105 as having a security impact of Critical.
The following Red Hat Product versions are impacted:
- Red Hat OpenShift Container Platform 3.x
- Red Hat OpenShift Online
- Red Hat OpenShift Dedicated
Attack Description and Impact
The OpenShift API server is a component which handles the API requests for OpenShift Container Platform 3. The OpenShift API server had a vulnerability in the ‘upgradeawarehandler’ type which acts as a reverse proxy to the compute nodes and other services added as API Extensions. A flaw in the reverse proxy prevented it from closing connections from the downstream service when there was an error. This allowed the user making the API call to escalate their privileges.
There are 2 ways to use this vulnerability to attack OpenShift Container Platform, OpenShift Online, or OpenShift Dedicated. The first involves abusing pod exec, attach, or portforward privileges granted to a normal user, and the second involves attacking the API extensions feature which provides the service catalog and access to additional features in OpenShift Container Platform 3.6 and later.
If a user has pod exec/attach/portforward privileges for any pod, their privileges can be escalated to cluster-admin, and any API call to a compute node Kubelet api can be achieved. An API call to the Kubelet api allows the user to exec into any container running on the same node as the pod which they have privileges for, including privileged containers which have read/write access to the host filesystem.
The second attack doesn’t require any privileges and exploits the API extension feature used by ‘metrics-server’ and ‘servicecatalog’ in OpenShift Container Platform, OpenShift Online, and Dedicated. It allows an unauthenticated user to gain cluster-admin privileges to any API extension deployed to the cluster including ‘servicecatalog’. Cluster-admin access to ‘servicecatalog’ allows creation of service brokers in any namespace and on any node.
Diagnose your vulnerability
To check the version of OpenShift container platform installed send an http request to the API server, such as (Note: The URL should be the same as the webconsole URL):
curl https://openshift.example.com:8443/version/openshift | grep gitVersion
*port is configuration specific
Alternatively if you are logged in with the ‘oc’ command you can check the version with this command:
oc version
Any versions of OpenShift Container Platform older than those listed below are vulnerable:
- v3.11.43-1
- v3.10.72-1
- v3.9.51-1
- v3.8.44-1
- v3.7.72-1
- v3.6.173.0.140-1
- v3.5.5.31.80-1
- v3.4.1.44.57-1
- v3.3.1.46.45-1
- v3.2.1.34-2
Take Action
Customers running affected versions of Red Hat products are strongly recommended to update them as soon as errata are available.
OpenShift Online (Starter, Pro) have been remediated. OpenShift Dedicated customers should speak with their support contact to confirm the status/schedule for these fixes.
Updates for Affected Products
Product | Package | Advisory/Update |
---|---|---|
OpenShift Container Platform v3.11 | kubernetes | RHSA-2018:3537 |
OpenShift Container Platform v3.10 | kubernetes | RHSA-2018:3549 |
OpenShift Container Platform v3.9 | kubernetes | RHSA-2018:2908 |
OpenShift Container Platform v3.8 | kubernetes | RHSA-2018:3551 |
OpenShift Container Platform v3.7 | kubernetes | RHSA-2018:2906 |
OpenShift Container Platform v3.6 | kubernetes | RHSA-2018:3598 |
OpenShift Container Platform v3.5 | kubernetes | RHSA-2018:3624 |
OpenShift Container Platform v3.4 | kubernetes | RHSA-2018:3752 |
OpenShift Container Platform v3.3 | kubernetes | RHSA-2018:3754 |
OpenShift Container Platform v3.2 | kubernetes | RHSA-2018:3742 |
Mitigation
Fixes for OpenShift Container Platform versions 3.2 and higher have been shipped to mitigate the flaw.
Users who update to the latest versions do not need to apply further mitigations. However, if the updates cannot be applied, mitigating the issue is recommended.
Mitigate the pod exec/attach/port-forward attack (OCP 3.2->3.11)
Change the default admin, and edit roles which have pod attach/exec/port-forward:
$oc edit clusterrole admin
Add the ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, and remove the pod attach/exec/portforward permissions. Example:
apiVersion: v1
kind: ClusterRole
metadata:
annotations:
openshift.io/description: A user that has edit rights within the project and can
change the project's membership.
creationTimestamp: 2018-11-21T01:34:04Z
name: admin
resourceVersion: "148192"
selfLink: /oapi/v1/clusterroles/admin
uid: 87dd0306-ed2d-11e8-9456-fa163e65ec84
rules:
- apiGroups:
- ""
attributeRestrictions: null
resources:
- pods
- pods/proxy
verbs:
Do the same for the ‘edit’ role which also has these permissions.
For a single user in a project:
Check the roles assigned to the user. In this case, the admin role is assigned to the quicklab user:
$ oc project myproj
$ oc adm policy who-can get pods/exec
Namespace: test
Verb: get
Resource: pods/exec
Users: quicklab
system:admin
system:serviceaccount:kube-system:generic-garbage-collector
system:serviceaccount:kube-system:namespace-controller
Groups: system:cluster-admins
system:masters
$ oc get rolebindings
NAME ROLE USERS GROUPS SERVICE ACCOUNTS SUBJECTS
admin /admin quicklab
system:deployers /system:deployer deployer
system:image-builders /system:image-builder builder
system:image-pullers /system:image-puller system:serviceaccounts:myproj
$ oc export clusterrole admin > admin-mitigate.yml
Edit admin-mitigate.yml by adding a ‘rbac.authorization.kubernetes.io/autoupdate’ annotation with a value of false, changing the name and removing the pod attach/exec/portforward permissions. Example:
apiVersion: v1
kind: ClusterRole
metadata:
annotations:
openshift.io/description: A user that has edit rights within the project and can
change the project's membership.
rbac.authorization.kubernetes.io/autoupdate: "false"
creationTimestamp: null
name: admin-mitigate
rules:
- apiGroups:
- ""
attributeRestrictions: null
resources:
- pods
- pods/proxy
verbs:
...
Create a new role with the new configuration:
$ cat admin-mitigate.yml | oc create -f -
Assign this role to any users who you don’t want to have those permissions:
$ oc adm policy remove-role-from-user admin quicklab
role "admin" removed: "quicklab"
$oc adm policy add-role-to-user admin-mitigate quicklab
role “admin-mitigate’”added: “quicklab”
Mitigating the API Extension attack (OCP 3.6 -> 3.11)
To mitigate the API extension attack until such time that a cluster upgrade to a patched version can be completed, it is possible to delete the affected API Extensions. Please review each services carefully to ensure this will not cause a loss of critical functionality, in which case alternate protections such as network level restrictions to known sources may be more appropriate.
Get a list of services which need to be disabled:
$ oc get apiservices -o=custom-columns=NAME:.metadata.name,SERVICE:.spec.service,STATUS:.status.conditions[0].type,VALUE:.status.conditions[0].status | grep -v '<nil>'
For example, on OCP 3.11 the services might include:
NAME SERVICE STATUS VALUE v1beta1.servicecatalog.k8s.io map[name:apiserver namespace:kube-service-catalog] Available True
Disable the services listed in the output, for example:
$ oc delete svc apiserver -n kube-service-catalog
For this example, to re-enable service-catalog run the openshift-service-catalog playbook. Please refer to the product documentation to re-enable other services.
$ ansible-playbook [-i /path/to/inventory]
/usr/share/ansible/openshift-ansible/playbooks/openshift-service-catalog/config.yml
Comments