Selecting the correct scheduler policy for an application's threads is not always a straightforward task. In general, realtime policies should be used for time critical or important tasks that need to be scheduled quickly and do not run for extended periods of time. Normal policies will generally yield better data throughput results than realtime policies because they let the scheduler run threads more efficiently (that is, they do not need to reschedule for pre-emption as often).
If you are managing large numbers of threads and are concerned mainly with data throughput (network packets per second, writes to disk, etc.) then use
SCHED_OTHER and let the system manage CPU utilization for you.
If you are concerned with event response time (latency) then use
SCHED_FIFO. If you have a small number of threads, consider isolating a CPU socket and moving your threads onto that socket's cores so that there are no other threads competing for time on the cores.