Openshift 4: runtime/cgo: pthread_create failed: Resource temporarily unavailable

Solution Verified - Updated -

Issue

  • CRI-O default PID limit is 1024.

  • Facing the following issue related to thread ulimit exhaustion while executing random operations in Openshift 4 nodes:

    runtime/cgo: pthread_create failed: Resource temporarily unavailable
    SIGABRT: abort
    PC=0x7fa231ce1387 m=0 sigcode=18446744073709551610
    
  • When running a must-gather a number of pthread, and SIGABRT tracebacks issues are observed in the output:

    [must-gather-example] POD 2023-02-14T16:46:48.261962360Z pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.261962360Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.261977300Z runtime/cgo: runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.261983733Z runtime/cgo: runtime/cgo: pthread_create failed: Resource temporarily unavailablepthread_create failed: Resource temporarily unavailablepthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.261995679Z 
    [must-gather-example] POD 2023-02-14T16:46:48.261995679Z 
    [must-gather-example] POD 2023-02-14T16:46:48.262001642Z pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.262048600Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.262070498Z runtime/cgo: pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.262085073Z runtime/cgo: pthread_create failed: Resource temporarily unavailableruntime/cgo: runtime/cgo: 
    [must-gather-example] POD 2023-02-14T16:46:48.262132533Z pthread_create failed: Resource temporarily unavailable
    [must-gather-example] POD 2023-02-14T16:46:48.262132533Z pthread_create failed: Resource temporarily unavailable
    [...]
    [must-gather-example] POD 2023-02-14T16:46:48.267108861Z INFO: Finished downloading openshift-apiserver/audit.log from server-example-1-master-1
    [must-gather-example] POD 2023-02-14T16:46:48.268980090Z INFO: Started  downloading kube-apiserver/audit-2023-02-14T14-00-46.297.log from server-example-1-master-2
    [must-gather-example] POD 2023-02-14T16:46:48.269277449Z SIGABRT: abort
    [must-gather-example] POD 2023-02-14T16:46:48.269277449Z PC=0x7f145edf037f m=8 sigcode=18446744073709551610
    [must-gather-example] POD 2023-02-14T16:46:48.269277449Z 
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z goroutine 0 [idle]:
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z runtime: unknown pc 0x7f145edf037f
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z stack: frame={sp:0x7f14349427f0, fp:0x0} stack=[0x7f1434143270,0x7f1434942e70)
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f14349426f0:  0x0000000000000007  0x0000000000000007 
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f1434942700:  0x0000000000000000  0x0000000000000000 
    [must-gather-example] POD 2023-02-14T16:46:48.269294190Z 0x00007f1434942710:  0x0000000000000000  0x0000000000000000 
    [must-gather-example] POD 2023-02-14T16:46:48.269305838Z 0x00007f1434942720:  0x0000000000000000  0x0000000000000000
    

Environment

  • Openshift Container Platform 4 (OCP)

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content