Unable to create more than 1024 Threads in the container.

Latest response

Recently we were getting the following exception in one of our Containers which was running a Java application. This container used to run perfectly on Openshift 3.11.

Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:717

As seen below, when you execute the aforementioned code in Openshift 3.11, it used to create 4000+ Threads. But in Openshift 4.2 it just created around 1000 Threads.

Openshift 3.11
ocp3.11

Openshift 4.2
ocp4.2

I also have made sure there is enough memory for the containers and ulimits are properly set within the container. Please refer below,
limits

Note: I used Openshift Playground for the above Tests.

What has changed in OCP 4.2 and how can I increase this limit? Thanks in advance.

Responses

I believe it's because this cgroup is in play:

$ cat /sys/fs/cgroup/pids/pids.max 
1024

while running your Java app, you can watch /sys/fs/cgroup/pids/pids.current until it approaches 1024 at which point your app bails out.

Now, I too wonder where in OpenShift this can be configured.

Did you ever find where we config pids.max ?

The Kubernetes feature is called SupportPodPidsLimit, it's off by default in Kubernetes and on by default in OpenShift. The way to configure this on OpenShift is to use feature gates, see: enabling features