Running tcpdump inside an OpenShift pod to capture network traffic

Resolution

• For OpenShift 3 clusters, please refer to How to use tcpdump or other system-level commands inside OpenShift v3 pod?

• For ROSA or OSD clusters, if cluster owner has missing permissions to follow this procedure, please refer to Running a tcpdump container as a sidecar on ROSA/OSD/ARO clusters.

• For OpenShift 4 clusters, if the following procedure does not run successfully and still the SSH access to the nodes is available, please consider the solution Running tcpdump inside a OpenShift 4 pod with SSH access to the nodes. It's also possible to configure a SSH bastion host in AWS: Deploying a Bastion along side your OpenShift 4.x cluster on AWS.

• An automated tool called ksniff is also available (although not directly supported by Red Hat). Steps on how to use it are documented here: Packet capture inside Pod using community ksniff with OpenShift 4.

Using tcpdump with no SSH access to nodes

1. Open a debug shell on the node where the target pod is running:

   $ oc debug node/<nodename>
   

   If you use a custom debug image, make sure that tcpdump and nsenter are available in it.

   Note1: It is NOT required to run chroot /host and toolbox in currently supported OpenShift versions. The oc debug node/<nodename> has tcpdump already.

   Note2: Take note of the namespace created for the debug pod, it will be used in the coming steps.

2. Build the required nsenter parameters so that you can enter the pod namespace.

   In 4.8 and lower, follow these steps:

   # NAME=<pod-name>
   # NAMESPACE=<pod-namespace>
   # pod_id=$(chroot /host crictl pods --namespace ${NAMESPACE} --name ${NAME} -q)
   # pid=$(chroot /host bash -c "runc state $pod_id | jq .pid")
   # nsenter_parameters="-n -t $pid"
   

   In 4.9 and higher, follow these steps instead:

   # NAME=<pod-name>
   # NAMESPACE=<pod-namespace>
   # pod_id=$(chroot /host crictl pods --namespace ${NAMESPACE} --name ${NAME} -q)
   # ns_path="/host$(chroot /host bash -c "crictl inspectp $pod_id | jq '.info.runtimeSpec.linux.namespaces[]|select(.type==\"network\").path' -r")"
   # nsenter_parameters="--net=${ns_path}"
   

3. Before running the tcpdump command, you need to determine the name of the correct interface to use. You can list all the interfaces within the pod by running ip a or tcpdump -D as shown below.

   # nsenter $nsenter_parameters -- chroot /host ip a
   

   Or

   # nsenter $nsenter_parameters -- tcpdump -D
   

4. Once you have the interface name, start the tcpdump in the container's network namespace using command below. Make sure to replace ${INTERFACE} with the relevant interface name found in the earlier step (or set the variable to the correct value in your shell).

   # nsenter $nsenter_parameters -- tcpdump -nn -i ${INTERFACE} -w /host/var/tmp/${HOSTNAME}_$(date +\%d_%m_%Y-%H_%M_%S-%Z).pcap ${TCPDUMP_EXTRA_PARAMS}
   

   Press ctrl+c when you are ready to end capture and list the generated file(s):

   # ls /host/var/tmp/*.pcap
   

   Note: See this page for more details on using the tcpdump command: Capturing network packets with tcpdump. You can also replace tcpdump command with any other command you want to run from the pod's network namespace.

5. Before exiting the debug pod, open a new terminal and copy the .pcap file generated :

   $ oc project <debugPodNamespace>
   
   $ oc get pods
   NAME                                            READY   STATUS      RESTARTS   AGE
   ip-10-z-yyy-xxxus-west-2computeinternal-debug   1/1     Running     0          97s
   
   $ oc cp ip-10-z-yyy-xxxus-west-2computeinternal-debug:/host/var/tmp/<filename>.pcap <filename>.pcap
   

6. Once the .pcap file is copied, it can be removed from the node by running this from the debug pod:

   # rm /host/var/tmp/<filename>.pcap