Openshift 4: runtime/cgo: pthread_create failed: Resource temporarily unavailable

Issue

• CRI-O default PID limit is 1024.

• Facing the following issue related to thread ulimit exhaustion while executing random operations in Openshift 4 nodes:

  runtime/cgo: pthread_create failed: Resource temporarily unavailable
  SIGABRT: abort
  PC=0x7fa231ce1387 m=0 sigcode=18446744073709551610
  

• When running a must-gather a number of pthread, and SIGABRT tracebacks issues are observed in the output:

  [must-gather-example] POD 2023-02-14T16:46:48.261962360Z pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.261962360Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.261977300Z runtime/cgo: runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.261983733Z runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailablepthread_create failed: Resource temporarily unavailablepthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.261995679Z 
  [must-gather-example] POD 2023-02-14T16:46:48.261995679Z 
  [must-gather-example] POD 2023-02-14T16:46:48.262001642Z pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.262048600Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.262070498Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.262085073Z runtime/cgo: pthread_create failed: Resource temporarily unavailableruntime/cgo: runtime/cgo: 
  [must-gather-example] POD 2023-02-14T16:46:48.262132533Z pthread_create failed: Resource temporarily unavailable
  [must-gather-example] POD 2023-02-14T16:46:48.262132533Z pthread_create failed: Resource temporarily unavailable
  [...]
  [must-gather-example] POD 2023-02-14T16:46:48.267108861Z INFO: Finished downloading openshift-apiserver/audit.log from server-example-1-master-1
  [must-gather-example] POD 2023-02-14T16:46:48.268980090Z INFO: Started  downloading kube-apiserver/audit-2023-02-14T14-00-46.297.log from server-example-1-master-2
  [must-gather-example] POD 2023-02-14T16:46:48.269277449Z SIGABRT: abort
  [must-gather-example] POD 2023-02-14T16:46:48.269277449Z PC=0x7f145edf037f m=8 sigcode=18446744073709551610
  [must-gather-example] POD 2023-02-14T16:46:48.269277449Z 
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z goroutine 0 [idle]:
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z runtime: unknown pc 0x7f145edf037f
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z stack: frame={sp:0x7f14349427f0, fp:0x0} stack=[0x7f1434143270,0x7f1434942e70)
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f14349426f0:  0x0000000000000007  0x0000000000000007 
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f1434942700:  0x0000000000000000  0x0000000000000000 
  [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f1434942710:  0x0000000000000000  0x0000000000000000 
  [must-gather-example] POD 2023-02-14T16:46:48.269305838Z 0x00007f1434942720:  0x0000000000000000  0x0000000000000000