RHEL8: trace-cmd record -e all (or -e sched) -M <HEX_MASK> does not work on machine with >128 CPUs machine

Solution Verified - Updated -

Issue

On machines with a lot of CPUs, usually more than 128..192 CPUs, trace-cmd record cannot record events at all for specific CPU(s) and fails, for instance, like this:

[root@machine ~]# trace-cmd record -e sched:sched_switch -M 80000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
none
trace-cmd: Invalid argument
  Failed filter of /sys/kernel/tracing/events/sched/sched_switch/filter

or:

[root@machine ~]# trace-cmd record -e sched -M 80000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
### global filter ### 
# Use this to set filters for multiple events.
# Only events with the given fields will be affected.
# If no events are modified, an error message will be displayed here
trace-cmd: Invalid argument
  Failed filter of /sys/kernel/tracing/events/sched/filter

or:

user@machine:~$ sudo trace-cmd record -e all -M 80000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000
(common_pid!=71477)&&(common_pid!=71476)&&(common_pid!=71475)&&(common_pid!=71474)&&(common_pid!=71473)&&(common_pid!=71472)&&(common_pid!=71471)&&(common_pid!=71470)&&(common_pid!=71469)&&(common_pid!=71468)&&(common_pid!=71467)&&(common_pid!=71466)&&(common_pid!=71465)&&(common_pid!=71464)&&(common_pid!=71463)&&(common_pid!=71462)&&(common_pid!=71461)&&(common_pid!=71460)&&(common_pid!=71459)&&(common_pid!=71458)&&(common_pid!=71457)&&(common_pid!=71456)&&(common_pid!=71455)&&(common_pid!=71454)&&(common_pid!=71453)&&(common_pid!=71452)&&(common_pid!=71451)&&(common_pid!=71450)&&(common_pid!=71449)&&(common_pid!=71448)&&(common_pid!=71447)&&(common_pid!=71446)&&(common_pid!=71445)&&(common_pid!=71444)&&(common_pid!=71443)&&(common_pid!=71442)&&(common_pid!=71441)&&(common_pid!=71440)&&(common_pid!=71439)&&(common_pid!=71438)&&(common_pid!=71437)&&(common_pid!=71436)&&(common_pid!=71435)&&(common_pid!=71434)&&(common_pid!=71433)&&(common_pid!=71432)&&(common_pid!=71431)&&(common_pid!=71430)&&(common_pid!=71429)&&(common_pid!=71428)&&(common_pid!=71427)&&(common_pid!=71426)&&(common_pid!=71425)&&(common_pid!=71424)&&(common_pid!=71423)&&(common_pid!=71422)&&(common_pid!=71421)&&(common_pid!=71420)&&(common_pid!=71419)&&(common_pid!=71418)&&(common_pid!=71417)&&(common_pid!=71416)&&(common_pid!=71415)&&(common_pid!=71414)&&(common_pid!=71413)&&(common_pid!=71412)&&(common_pid!=71411)&&(common_pid!=71410)&&(common_pid!=71409)&&(common_pid!=71408)&&(common_pid!=71407)&&(common_pid!=71406)&&(common_pid!=71405)&&(common_pid!=71404)&&(common_pid!=71403)&&(common_pid!=71402)&&(common_pid!=71401)&&(common_pid!=71400)&&(common_pid!=71399)&&(common_pid!=71398)&&(common_pid!=71397)&&(common_pid!=71396)&&(common_pid!=71395)&&(common_pid!=71394)&&(common_pid!=71393)&&(common_pid!=71392)&&(common_pid!=71391)&&(common_pid!=71390)&&(common_pid!=71389)&&(common_pid!=71388)&&(common_pid!=71387)&&(common_pid!=71386)&&(common_pid!=71385)&&(common_pid!=71384)&&(common_pid!=71383)&&(common_pid!=71382)&&(common_pid!=71381)&&(common_pid!=71380)&&(common_pid!=71379)&&(common_pid!=71378)&&(common_pid!=71377)&&(common_pid!=71376)&&(common_pid!=71375)&&(common_pid!=71374)&&(common_pid!=71373)&&(common_pid!=71372)&&(common_pid!=71371)&&(common_pid!=71370)&&(common_pid!=71369)&&(common_pid!=71368)&&(common_pid!=71367)&&(common_pid!=71366)
trace-cmd: Invalid argument
  Failed filter of /sys/kernel/tracing/events/sched/sched_switch/filter

Environment

  • Red Hat Enterprise Linux 8
  • trace-cmd package version older than 2.7-9
  • machine with more than 128 CPUs

Subscriber exclusive content

A Red Hat subscription provides unlimited access to our knowledgebase, tools, and much more.

Current Customers and Partners

Log in for full access

Log In

New to Red Hat?

Learn more about Red Hat subscriptions

Using a Red Hat product through a public cloud?

How to access this content