Running tcpdump inside an OpenShift pod to capture network traffic

Resolution

When it is needed to run tcpdump in a different way, or if the following method does not work, please check the following solutions:

• If it is needed to run tcpdump from a node and not from a pod, refer to running tcpdump from a RHEL CoreOS OpenShift node

• If the following procedure does not run successfully and still the SSH access to the nodes is available, please consider the solution running tcpdump inside a OpenShift 4 pod with SSH access to the nodes. It is also possible to configure a SSH bastion host in AWS: deploying a bastion along side OpenShift 4 cluster on AWS.

• For ROSA, OSD and ARO clusters, if cluster owner has missing permissions to follow this procedure, please refer to running a tcpdump container as a sidecar on ROSA/OSD/ARO clusters.

• An automated tool called ksniff is also available (although not directly supported by Red Hat). Steps on how to use it are documented here: packet capture inside Pod using community ksniff with OpenShift 4.

Using tcpdump within a pod with no SSH access to nodes

1. Open a debug shell on the node where the target pod is running:

   $ oc debug node/<nodename>
   

   If using a custom debug image, make sure that tcpdump and nsenter are available in it.

   Note1: It is NOT required to run chroot /host and toolbox in currently supported OpenShift versions. The oc debug node/<nodename> has tcpdump already.

   Note2: Take note of the namespace created for the debug pod, it will be used in the coming steps.

2. Build the required nsenter parameters so that can enter the pod namespace:

   In currently supported OpenShift versions (4.9 and higher), follow these steps:

   # NAME=<pod-name>
   # NAMESPACE=<pod-namespace>
   # pod_id=$(chroot /host crictl pods --namespace ${NAMESPACE} --name ${NAME} -q)
   # ns_path="/host$(chroot /host bash -c "crictl inspectp $pod_id | jq '.info.runtimeSpec.linux.namespaces[]|select(.type==\"network\").path' -r")"
   # nsenter_parameters="--net=${ns_path}"
   

   In 4.8 and lower (currently not supported), follow these steps:

   # NAME=<pod-name>
   # NAMESPACE=<pod-namespace>
   # pod_id=$(chroot /host crictl pods --namespace ${NAMESPACE} --name ${NAME} -q)
   # pid=$(chroot /host bash -c "runc state $pod_id | jq .pid")
   # nsenter_parameters="-n -t $pid"
   

3. Before running the tcpdump command, it is needed to determine the name of the correct interface to use. It is possible to list all the interfaces within the pod by running ip a or tcpdump -D as shown below:

   # nsenter $nsenter_parameters -- chroot /host ip a
   

   Or

   # nsenter $nsenter_parameters -- tcpdump -D
   

4. Once having the interface name, start the tcpdump in the container's network namespace using command below. Make sure to replace ${INTERFACE} with the relevant interface name found in the earlier step (or set the variable to the correct value in the shell).

   # nsenter $nsenter_parameters -- tcpdump -nn -i ${INTERFACE} -w /host/var/tmp/${HOSTNAME}_$(date +\%d_%m_%Y-%H_%M_%S-%Z).pcap ${TCPDUMP_EXTRA_PARAMS}
   

   Press ctrl+c when ready to end the capture and list the generated file(s):

   # ls /host/var/tmp/*.pcap
   

   Note: See this page for more details on using the tcpdump command: capturing network packets with tcpdump. It is also possible to replace tcpdump command with any other command needed to run from the pod's network namespace.

5. Before exiting the debug pod, open a new terminal and copy the .pcap file generated :

   $ oc project <debugPodNamespace>
   
   $ oc get pods
   NAME                                            READY   STATUS      RESTARTS   AGE
   ip-10-z-yyy-xxxus-west-2computeinternal-debug   1/1     Running     0          97s
   
   $ oc cp ip-10-z-yyy-xxxus-west-2computeinternal-debug:/host/var/tmp/<filename>.pcap <filename>.pcap
   

6. Once the .pcap file is copied, it can be removed from the node by running this from the debug pod:

   # rm /host/var/tmp/<filename>.pcap