Running chroot from a unprivileged pod fails in OCP 4

Solution Verified - Updated -

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift on AWS (ROSA)
    • 4
  • Red Hat OpenShift Dedicated (OSD)
    • 4
  • Red Hat OpenShift on Azure (ARO)
    • 4

Issue

  • SFTP pods failing to start because missing permissions.

  • Pod fail to start (crashLoopBackOff) due to not enough permission provided to the pod with the following error:

    mkdir: cannot create directory '/var/run/sftp': Permission denied
/entrypoint: Error on line 34: mkdir -p "$(dirname $userConfFinalPath)"

  • Pods fail to start (crashLoopBackOff) due to the SCC (Security Context Constraints) being too restrictive with the following error:

    chroot("/run/sshd"): Operation not permitted [preauth]

Resolution


WARNING

The following instructions use the "Privileged" Security Context Constraint (SCC). While this SCC is part of the default OpenShift constraints it has highly elevated access that can be used maliciously. Before following this process, familiarize yourself with the risks associated with elevating privileges in a container, and only provide elevated privileges when other safeguards are in place.

Using the security principle of least privilege Red Hat recommends rewriting container images to use the default constraint of "Restricted"

The purpose of this example is to highlight that certain vendor and open-source images can fail in OpenShift due to Security Constraints. This failure to run on OpenShift is intended by design to ensure the OpenShift cluster remains secure. The design follows the security principle of least privilege and highlights assumptions made by the image producer that their image will be run in an environment that is less secure. For example, running using Podman on a local machine.

This failure is not a bug but a feature, that is designed to keep Openshift clusters secure.

By default, Openshift uses the Restricted Security Context Constraint for deploying pods. This restricted constraint should be adequate for most, if not all, well developed images. It is common to find third-party container images that fail to run on OpenShift. Red Hat recommends assigning images the least privilege possible, by rewriting container images to be able to use the default constraint of Restricted, rather than using the following example to elevate privileges of third-party images.

The following is an example of an image which will continue to fail and crashLoopBackOff until the the Privileged SCC is assigned to the service account and specific capabilities (SYS_CHROOT) are added to the deployment and leaves the service account, application and the OpenShift cluster more vulnerable to malicious activity. It is not recommended to use this example in a production or secured environment.

  1. Create a new project

    $ oc new-project sftp

  2. Setup privileged Security Context and Service account
    This is configured via a ServiceAccount, Role and RoleBinding

    $ oc create serviceaccount sftp-serviceaccount

$ oc create role privileged-scc --verb=use --resource-name=privileged --resource=securitycontextconstraints

$ cat << EOF > rolebinding.yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: use-privileged-scc
subjects:
 - kind: ServiceAccount
   name: sftp-serviceaccount
roleRef:
 kind: Role
 name: privileged-scc
 apiGroup: rbac.authorization.k8s.io
EOF

    Create the role binding for the service account above

    oc create -f rolebinding.yaml

  3. Create SFTP app

    $ oc new-app atmoz/sftp

  4. Add Security Context and to deployment

    Use oc edit deployment <sftpApp> to add the following to the deployed SFTP app. Add the lines with <<--- Add/Edit this line mark

    $ oc get deployment sftp -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 annotations:
 deployment.kubernetes.io/revision: "4"
 image.openshift.io/triggers: '[{"from":{"kind":"ImageStreamTag","name":"sftp:latest"},"fieldPath":"spec.template.spec.containers[?(@.name==\"sftp\")].image"}]'
 openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: "2021-09-27T12:16:07Z"
 generation: 4
 labels:
   app: sftp
   app.kubernetes.io/component: sftp
   app.kubernetes.io/instance: sftp
 managedFields:
...
...
 name: sftp
 namespace: sftp2
 resourceVersion: "2648726"
 selfLink: /apis/apps/v1/namespaces/sftp2/deployments/sftp
 uid: 27cd72df-5c46-4ea9-af85-beba5a48eacf
spec:
 progressDeadlineSeconds: 600
 replicas: 1
 revisionHistoryLimit: 10
 selector:
   matchLabels:
     deployment: sftp
 strategy:
   rollingUpdate:
     maxSurge: 25%
     maxUnavailable: 25%
   type: RollingUpdate
 template:
   metadata:
     annotations:
       openshift.io/generated-by: OpenShiftNewApp
     creationTimestamp: null
     labels:
       deployment: sftp
   spec:
     containers:
     - image: atmoz/sftp@sha256:dda74e3ad4f0fbb6e0ce08d56eed856b8734a3f39217498e80b1e748116428c1
       imagePullPolicy: IfNotPresent
       name: sftp
       ports:
       - containerPort: 22
         protocol: TCP
       resources: {}
       securityContext:        <<--- Add/Edit this line
         capabilities:         <<--- Add/Edit this line
           add:                <<--- Add/Edit this line
           - SYS_CHROOT        <<--- Add/Edit this line
         runAsGroup: 0         <<--- Add/Edit this line
         runAsUser: 0          <<--- Add/Edit this line
       terminationMessagePath: /dev/termination-log
       terminationMessagePolicy: File
       volumeMounts:
       - mountPath: /etc/sftp
         name: volume-dhvsn
     dnsPolicy: ClusterFirst
     restartPolicy: Always
     schedulerName: default-scheduler
     securityContext:                          <<--- Add/Edit this line
       fsGroup: 0                              <<--- Add/Edit this line
     serviceAccount: sftp-serviceaccount    
     serviceAccountName: sftp-serviceaccount   <<--- Add/Edit this line
     terminationGracePeriodSeconds: 30
     volumes:
     - configMap:
         defaultMode: 420
         name: sftp-etc-sftp
       name: volume-dhvsn
status:
...
...

  5. Setup SFTP Users

    $ cat <<EOF > users.conf
foo:123:1001:100:upload
bar:abc:1002:100:upload
baz:xyz:1003:100:upload
EOF

$ oc create configmap sftp-etc-sftp --from-file=users.conf

$ oc set volumes deployment.apps/sftp --add --type configmap --mount-path=/etc/sftp --configmap-name=sftp-etc-sftp

  6. Test SFTP access from the SFTP pod

    $ oc get pods
NAME                        READY   STATUS    RESTARTS   AGE
pod/sftp-664c977c4c-s577r   1/1     Running   0          57s

$ oc rsh sftp-664c977c4c-s577r
# sftp foo@localhost
foo@localhost's password:
Connected to foo@localhost.
sftp>

    The SFTP service is available from insde the pod

  7. Test SFTP access from the internet

    Use oc edit service sftp to change the service from an an internally available service to an externally facing service. Add the lines with <<--- Add/Edit this line mark

    $ oc get service sftp -o yaml
apiVersion: v1
kind: Service
metadata:
 annotations:
   openshift.io/generated-by: OpenShiftNewApp
 creationTimestamp: "2021-09-27T12:16:07Z"
 finalizers:
 - service.kubernetes.io/load-balancer-cleanup
 labels:
   app: sftp
   app.kubernetes.io/component: sftp
   app.kubernetes.io/instance: sftp
 managedFields:
...
...
 name: sftp
 namespace: sftp2
 resourceVersion: "2655603"
 selfLink: /api/v1/namespaces/sftp2/services/sftp
 uid: ed185485-da90-4bc8-9963-108cf7269672
spec:
 clusterIP: 172.30.162.39
 clusterIPs:
 - 172.30.162.39
 externalTrafficPolicy: Cluster
 ports:
 - name: 22-tcp
   nodePort: 31183
   port: 22
   protocol: TCP
   targetPort: 22
 selector:
   deployment: sftp
 sessionAffinity: None
 type: LoadBalancer           <<--- From: ClusterIP  To: LoadBalancer
status:
 ...

    Get the new external IP

    $ oc get service
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/sftp   LoadBalancer   172.30.162.39   52.224.184.41   22:31183/TCP   36m

    Notice the external IP, use it to access the cluster.

    $ sftp foo@52.224.184.41
load pubkey "/Users/[userID]/.ssh/id_rsa": invalid format
The authenticity of host '52.224.184.41 (52.224.184.41)' can't be established.
ED25519 key fingerprint is SHA256:b1QqrV1yNUTfxqEjGVP1vCvdgmqT6nPScTM4FI3vYDE.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '52.224.184.41' (ED25519) to the list of known hosts.
Enter passphrase for key '/Users/[userID]/.ssh/id_rsa':
foo@52.224.184.41's password:
Connected to 52.224.184.41.
sftp>

Root Cause

OpenShift uses a default SCC (Security Context Constraint) that has been designed to be secure. This restricted SCC may not always be practical so other SCCs are also provided as part of the installation.

If none of these additional contexts are practical, then new SCCs can be created and tailored to suit the requirements of the application.

By using the least restrictive privileged SCC more can be done within the container but at the cost of being more insecure.

Diagnostic Steps

Check the pod logs of any pods in CrashLoopBackOff.

If any pods show issues related to permissions, similar to the error below, then loosening the SCCs for the pod may allow the pod to function correctly:

$ oc logs pod/<somePod>
[...]
mkdir: cannot create directory '/var/run/sftp': Permission denied
/entrypoint: Error on line 34: mkdir -p "$(dirname $userConfFinalPath)"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments