System/Business Analyst

This book enumerates and explains Red Hat Enterprise Linux 6 performance features at a high level, providing enough information on how subsystems perform for specific workloads (both by default and when optimized). The level of detail used in describing Red Hat Enterprise Linux 6 performance features helps potential customers and sales engineers understand the suitability of this platform in providing resource-intensive services at an acceptable level.

The Performance Tuning Guide also provides links to more detailed documentation on each feature whenever possible. At that detail level, readers can understand these performance features enough to form a high-level strategy in deploying and optimizing Red Hat Enterprise Linux 6. This allows readers to both develop and evaluate infrastructure proposals.

This feature-focused level of documentation is suitable for readers with a high-level understanding of Linux subsystems and enterprise-level networks.

System Administrator

The procedures enumerated in this book are suitable for system administrators with RHCE ^[1] skill level (or its equivalent, that is, 3-5 years experience in deploying and managing Linux). The Performance Tuning Guide aims to provide as much detail as possible about the effects of each configuration; this means describing any performance trade-offs that may occur.

The underlying skill in performance tuning lies not in knowing how to analyze and tune a subsystem. Rather, a system administrator adept at performance tuning knows how to balance and optimize a Red Hat Enterprise Linux 6 system for a specific purpose. This means also knowing which trade-offs and performance penalties are acceptable when attempting to implement a configuration designed to boost a specific subsystem's performance.

Communication

Horizontal scalability requires many tasks to be performed simultaneously (in parallel). As such, these tasks must have interprocess communication to coordinate their work. Further, a platform with horizontal scalability should be able to share tasks across multiple systems.

Storage

Storage via local disks is not sufficient in addressing the requirements of horizontal scalability. Some form of distributed or shared storage is needed, one with a layer of abstraction that allows a single storage volume's capacity to grow seamlessly with the addition of new storage hardware.

Management

The most important duty in distributed computing is the management layer. This management layer coordinates all software and hardware components, efficiently managing communication, storage, and the usage of shared resources.

Network monitoring scripts (in examples/network)

nettop.stp

Every 5 seconds, prints a list of processes (process identifier and command) with the number of packets sent and received and the amount of data sent and received by the process during that interval.

socket-trace.stp

Instruments each of the functions in the Linux kernel's net/socket.c file, and prints trace data.

tcp_connections.stp

Prints information for each new incoming TCP connection accepted by the system. The information includes the UID, the command accepting the connection, the process identifier of the command, the port the connection is on, and the IP address of the originator of the request.

dropwatch.stp

Every 5 seconds, prints the number of socket buffers freed at locations in the kernel. Use the --all-modules option to see symbolic names.

Storage monitoring scripts (in examples/io)

disktop.stp

Checks the status of reading/writing disk every 5 seconds and outputs the top ten entries during that period.

iotime.stp

Prints the amount of time spent on read and write operations, and the number of bytes read and written.

traceio.stp

Prints the top ten executables based on cumulative I/O traffic observed, every second.

traceio2.stp

Prints the executable name and process identifier as reads and writes to the specified device occur.

inodewatch.stp

Prints the executable name and process identifier each time a read or write occurs to the specified inode on the specified major/minor device.

inodewatch2.stp

Prints the executable name, process identifier, and attributes each time the attributes are changed on the specified inode on the specified major/minor device.

perf stat

This command provides overall statistics for common performance events, including instructions executed and clock cycles consumed. You can use the option flags to gather statistics on events other than the default measurement events. As of Red Hat Enterprise Linux 6.4, it is possible to use perf stat to filter monitoring based on one or more specified control groups (cgroups). For further information, read the man page: man perf-stat.

perf record

This command records performance data into a file which can be later analyzed using perf report. For further details, read the man page: man perf-record.

As of Red Hat Enterprise Linux 6.6, the -b and -j options are provided to allow statistical sampling of taken branches. The -b option samples any branches taken, while the -j option can be adjusted to sample branches of different types, such as user level or kernel level branches.

perf report

This command reads the performance data from a file and analyzes the recorded data. For further details, read the man page: man perf-report.

perf list

This command lists the events available on a particular machine. These events will vary based on the performance monitoring hardware and the software configuration of the system. For further information, read the man page: man perf-list.

perf mem

Available as of Red Hat Enterprise Linux 6.6. This command profiles the frequency of each type of memory access operation performed by a specified application or command. This allows a user to see the frequency of load and store operations performed by the profiled application. For further information, read the man page: man perf-mem.

perf top

This command performs a similar function to the top tool. It generates and displays a performance counter profile in realtime. For further information, read the man page: man perf-top.

perf trace

This command performs a similar function to the strace tool. It monitors the system calls used by a specified thread or process and all signals received by that application. Additional trace targets are available; refer to the man page for a full list.

performance

The processor is unwilling to sacrifice any performance for the sake of saving energy. This is the default value.

normal

The processor tolerates minor performance compromises for potentially significant energy savings. This is a reasonable setting for most desktops and servers.

powersave

The processor accepts potentially significant hits to performance in order to maximise energy efficiency.

pkg

The processor package number.

core

The processor core number.

CPU

The Linux CPU (logical processor) number.

%c0

The percentage of the interval for which the CPU retired instructions.

GHz

The average clock speed while the CPU was in the c0 state. When this number is higher than the value in TSC, the CPU is in turbo mode.

TSC

The average clock speed over the course of the entire interval. When this number is lower than the value in TSC, the CPU is in turbo mode.

%c1, %c3, and %c6

The percentage of the interval for which the processor was in the c1, c3, or c6 state, respectively.

%pc3 or %pc6

The percentage of the interval for which the processor was in the pc3 or pc6 state, respectively.

Default Tracking Categories

numa_hit

The number of attempted allocations to this node that were successful.

numa_miss

The number of attempted allocations to another node that were allocated on this node because of low memory on the intended node. Each numa_miss event has a corresponding numa_foreign event on another node.

numa_foreign

The number of allocations initially intended for this node that were allocated to another node instead. Each numa_foreign event has a corresponding numa_miss event on another node.

interleave_hit

The number of attempted interleave policy allocations to this node that were successful.

local_node

The number of times a process on this node successfully allocated memory on this node.

other_node

The number of times a process on another node allocated memory on this node.

-c

Horizontally condenses the displayed table of information. This is useful on systems with a large number of NUMA nodes, but column width and inter-column spacing are somewhat unpredictable. When this option is used, the amount of memory is rounded to the nearest megabyte.

-m

Displays system-wide memory usage information on a per-node basis, similar to the information found in /proc/meminfo.

-n

Displays the same information as the original numastat command (numa_hit, numa_miss, numa_foreign, interleave_hit, local_node, and other_node), with an updated format, using megabytes as the unit of measurement.

-p pattern

Displays per-node memory information for the specified pattern. If the value for pattern is comprised of digits, numastat assumes that it is a numerical process identifier. Otherwise, numastat searches process command lines for the specified pattern.

Command line arguments entered after the value of the -p option are assumed to be additional patterns for which to filter. Additional patterns expand, rather than narrow, the filter.

-s

Sorts the displayed data in descending order so that the biggest memory consumers (according to the total column) are listed first.

Optionally, you can specify a node, and the table will be sorted according to the node column. When using this option, the node value must follow the -s option immediately, as shown here:

numastat -s2

Do not include white space between the option and its value.

-v

Displays more verbose information. Namely, process information for multiple processes will display detailed information for each process.

-V

Displays numastat version information.

-z

Omits table rows and columns with only zero values from the displayed information. Note that some near-zero values that are rounded to zero for display purposes will not be omitted from the displayed output.

SCHED_FIFO

This policy is also referred to as static priority scheduling, because it defines a fixed priority (between 1 and 99) for each thread. The scheduler scans a list of SCHED_FIFO threads in priority order and schedules the highest priority thread that is ready to run. This thread runs until it blocks, exits, or is preempted by a higher priority thread that is ready to run.

Even the lowest priority realtime thread will be scheduled ahead of any thread with a non-realtime policy; if only one realtime thread exists, the SCHED_FIFO priority value does not matter.

SCHED_RR

A round-robin variant of the SCHED_FIFO policy. SCHED_RR threads are also given a fixed priority between 1 and 99. However, threads with the same priority are scheduled round-robin style within a certain quantum, or time slice. The sched_rr_get_interval(2) system call returns the value of the time slice, but the duration of the time slice cannot be set by a user. This policy is useful if you need multiple thread to run at the same priority.

/proc/sys/kernel/sched_rt_period_us

Defines the time period to be considered one hundred percent of CPU bandwidth, in microseconds ('us' being the closest equivalent to 'µs' in plain text). The default value is 1000000µs, or 1 second.

/proc/sys/kernel/sched_rt_runtime_us

Defines the time period to be devoted to running realtime threads, in microseconds ('us' being the closest equivalent to 'µs' in plain text). The default value is 950000µs, or 0.95 seconds.

SCHED_OTHER, or SCHED_NORMAL

The default scheduling policy. This policy uses the Completely Fair Scheduler (CFS) to provide fair access periods for all threads using this policy. CFS establishes a dynamic priority list partly based on the niceness value of each process thread. (Refer to the Deployment Guide for more details about this parameter and the /proc file system.) This gives users some indirect level of control over process priority, but the dynamic priority list can only be directly changed by the CFS.

CPU pinning

Virtual guests can be bound to run on a specific socket in order to optimize local cache use and remove the need for expensive inter-socket communications and remote memory access.

transparent hugepages (THP)

With THP enabled, the system automatically performs NUMA-aware memory allocation requests for large contiguous amounts of memory, reducing both lock contention and the number of translation lookaside buffer (TLB) memory management operations required and yielding a performance increase of up to 20% in virtual guests.

kernel-based I/O implementation

The virtual guest I/O subsystem is now implemented in the kernel, greatly reducing the expense of inter-node communication and memory access by avoiding a significant amount of context switching, and synchronization and communication overhead.

Huge pages kernel options

hugepages

Defines the number of persistent huge pages configured in the kernel at boot time. The default value is 0. It is only possible to allocate (or deallocate) huge pages if there are sufficient physically contiguous free pages in the system. Pages reserved by this parameter cannot be used for other purposes.

Default size huge pages can be dynamically allocated or deallocated by changing the value of the /proc/sys/vm/nr_hugepages file.

In a NUMA system, huge pages assigned with this parameter are divided equally between nodes. You can assign huge pages to specific nodes at runtime by changing the value of the node's /sys/devices/system/node/node_id/hugepages/hugepages-1048576kB/nr_hugepages file.

For more information, read the relevant kernel documentation, which is installed in /usr/share/doc/kernel-doc-kernel_version/Documentation/vm/hugetlbpage.txt by default. This documentation is available only if the kernel-doc package is installed.

hugepagesz

Defines the size of persistent huge pages configured in the kernel at boot time. Valid values are 2 MB and 1 GB. The default value is 2 MB.

default_hugepagesz

Defines the default size of persistent huge pages configured in the kernel at boot time. Valid values are 2 MB and 1 GB. The default value is 2 MB.

--leak-check

When enabled, Memcheck searches for memory leaks when the client program finishes. The default value is summary, which outputs the number of leaks found. Other possible values are yes and full, both of which give details of each individual leak, and no, which disables memory leak checking.

--undef-value-errors

When enabled (set to yes), Memcheck reports errors when undefined values are used. When disabled (set to no), undefined value errors are not reported. This is enabled by default. Disabling it speeds up Memcheck slightly.

--ignore-ranges

Allows the user to specify one or more ranges that Memcheck should ignore when checking for addressability. Multiple ranges are delimited by commas, for example, --ignore-ranges=0xPP-0xQQ,0xRR-0xSS.

--I1

Specifies the size, associativity, and line size of the first-level instruction cache, separated by commas: --I1=size,associativity,line size.

--D1

Specifies the size, associativity, and line size of the first-level data cache, separated by commas: --D1=size,associativity,line size.

--LL

Specifies the size, associativity, and line size of the last-level cache, separated by commas: --LL=size,associativity,line size.

--cache-sim

Enables or disables the collection of cache access and miss counts. The default value is yes (enabled).

Note that disabling both this and --branch-sim leaves Cachegrind with no information to collect.

--branch-sim

Enables or disables the collection of branch instruction and misprediction counts. This is set to no (disabled) by default, since it slows Cachegrind by approximately 25 per-cent.

Note that disabling both this and --cache-sim leaves Cachegrind with no information to collect.

--heap

Specifies whether to perform heap profiling. The default value is yes. Heap profiling can be disabled by setting this option to no.

--heap-admin

Specifies the number of bytes per block to use for administration when heap profiling is enabled. The default value is 8 bytes per block.

--stacks

Specifies whether to perform stack profiling. The default value is no (disabled). To enable stack profiling, set this option to yes, but be aware that doing so will greatly slow Massif. Also note that Massif assumes that the main stack has size zero at start-up in order to better indicate the size of the stack portion over which the program being profiled has control.

--time-unit

Specifies the unit of time used for the profiling. There are three valid values for this option: instructions executed (i), the default value, which is useful in most cases; real time (ms, in milliseconds), which can be useful in certain instances; and bytes allocated/deallocated on the heap and/or stack (B), which is useful for very short-run programs, and for testing purposes, because it is the most reproducible across different machines. This option is useful when graphing Massif output with ms_print.

Tunables

back_seek_max

Backward seeks are typically bad for performance, as they can incur greater delays in repositioning the heads than forward seeks do. However, CFQ will still perform them, if they are small enough. This tunable controls the maximum distance in KB the I/O scheduler will allow backward seeks. The default is 16 KB.

back_seek_penalty

Because of the inefficiency of backward seeks, a penalty is associated with each one. The penalty is a multiplier; for example, consider a disk head position at 1024KB. Assume there are two requests in the queue, one at 1008KB and another at 1040KB. The two requests are equidistant from the current head position. However, after applying the back seek penalty (default: 2), the request at the later position on disk is now twice as close as the earlier request. Thus, the head will move forward.

fifo_expire_async

This tunable controls how long an async (buffered write) request can go unserviced. After the expiration time (in milliseconds), a single starved async request will be moved to the dispatch list. The default is 250 ms.

fifo_expire_sync

This is the same as the fifo_expire_async tunable, for for synchronous (read and O_DIRECT write) requests. The default is 125 ms.

group_idle

When set, CFQ will idle on the last process issuing I/O in a cgroup. This should be set to 1 when using proportional weight I/O cgroups and setting slice_idle to 0 (typically done on fast storage).

group_isolation

If group isolation is enabled (set to 1), it provides a stronger isolation between groups at the expense of throughput. Generally speaking, if group isolation is disabled, fairness is provided for sequential workloads only. Enabling group isolation provides fairness for both sequential and random workloads. The default value is 0 (disabled). Refer to Documentation/cgroups/blkio-controller.txt for further information.

low_latency

When low latency is enabled (set to 1), CFQ attempts to provide a maximum wait time of 300 ms for each process issuing I/O on a device. This favors fairness over throughput. Disabling low latency (setting it to 0) ignores target latency, allowing each process in the system to get a full time slice. Low latency is enabled by default.

quantum

The quantum controls the number of I/Os that CFQ will send to the storage at a time, essentially limiting the device queue depth. By default, this is set to 8. The storage may support much deeper queue depths, but increasing quantum will also have a negative impact on latency, especially in the presence of large sequential write workloads.

slice_async

This tunable controls the time slice allotted to each process issuing asynchronous (buffered write) I/O. By default it is set to 40 ms.

slice_idle

This specifies how long CFQ should idle while waiting for further requests. The default value in Red Hat Enterprise Linux 6.1 and earlier is 8 ms. In Red Hat Enterprise Linux 6.2 and later, the default value is 0. The zero value improves the throughput of external RAID storage by removing all idling at the queue and service tree level. However, a zero value can degrade throughput on internal non-RAID storage, because it increases the overall number of seeks. For non-RAID storage, we recommend a slice_idle value that is greater than 0.

slice_sync

This tunable dictates the time slice allotted to a process issuing synchronous (read or direct write) I/O. The default is 100 ms.

Tunables

fifo_batch

This determines the number of reads or writes to issue in a single batch. The default is 16. Setting this to a higher value may result in better throughput, but will also increase latency.

front_merges

You can set this tunable to 0 if you know your workload will never generate front merges. Unless you have measured the overhead of this check, it is advisable to leave it at its default setting (1).

read_expire

This tunable allows you to set the number of milliseconds in which a read request should be serviced. By default, this is set to 500 ms (half a second).

write_expire

This tunable allows you to set the number of milliseconds in which a write request should be serviced. By default, this is set to 5000 ms (five seconds).

writes_starved

This tunable controls how many read batches can be processed before processing a single write batch. The higher this is set, the more preference is given to reads.

/sys/block/sdX/queue tunables

add_random

In some cases, the overhead of I/O events contributing to the entropy pool for /dev/random is measurable. In such cases, it may be desirable to set this value to 0.

max_sectors_kb

By default, the maximum request size sent to disk is 512 KB. This tunable can be used to either raise or lower that value. The minimum value is limited by the logical block size; the maximum value is limited by max_hw_sectors_kb. There are some SSDs which perform worse when I/O sizes exceed the internal erase block size. In such cases, it is recommended to tune max_hw_sectors_kb down to the erase block size. You can test for this using an I/O generator such as iozone or aio-stress, varying the record size from, for example, 512 bytes to 1 MB.

nomerges

This tunable is primarily a debugging aid. Most workloads benefit from request merging (even on faster storage such as SSDs). In some cases, however, it is desirable to disable merging, such as when you want to see how many IOPS a storage back-end can process without disabling read-ahead or performing random I/O.

nr_requests

Each request queue has a limit on the total number of request descriptors that can be allocated for each of read and write I/Os. By default, the number is 128, meaning 128 reads and 128 writes can be queued at a time before putting a process to sleep. The process put to sleep is the next to try to allocate a request, not necessarily the process that has allocated all of the available requests.

If you have a latency-sensitive application, then you should consider lowering the value of nr_requests in your request queue and limiting the command queue depth on the storage to a low number (even as low as 1), so that writeback I/O cannot allocate all of the available request descriptors and fill up the device queue with write I/O. Once nr_requests have been allocated, all other processes attempting to perform I/O will be put to sleep to wait for requests to become available. This makes things more fair, as the requests are then distributed in a round-robin fashion (instead of letting one process consume them all in rapid succession). Note that this is only a problem when using the deadline or noop schedulers, as the default CFQ configuration protects against this situation.

optimal_io_size

In some circumstances, the underlying storage will report an optimal I/O size. This is most common in hardware and software RAID, where the optimal I/O size is the stripe size. If this value is reported, applications should issue I/O aligned to and in multiples of the optimal I/O size whenever possible.

read_ahead_kb

The operating system can detect when an application is reading data sequentially from a file or from disk. In such cases, it performs an intelligent read-ahead algorithm, whereby more data than is requested by the user is read from disk. Thus, when the user next attempts to read a block of data, it will already by in the operating system's page cache. The potential down side to this is that the operating system can read more data from disk than necessary, which occupies space in the page cache until it is evicted because of high memory pressure. Having multiple processes doing false read-ahead would increase memory pressure in this circumstance.

For device mapper devices, it is often a good idea to increase the value of read_ahead_kb to a large number, such as 8192. The reason is that a device mapper device is often made up of multiple underlying devices. Setting this value to the default (128 KB) multiplied by the number of devices you are mapping is a good starting point for tuning.

rotational

Traditional hard disks have been rotational (made up of spinning platters). SSDs, however, are not. Most SSDs will advertise this properly. If, however, you come across a device that does not advertise this flag properly, it may be necessary to set rotational to 0 manually; when rotational is disabled, the I/O elevator does not use logic that is meant to reduce seeks, since there is little penalty for seek operations on non-rotational media.

rq_affinity

I/O completions can be processed on a different CPU from the one that issued the I/O. Setting rq_affinity to 1 causes the kernel to deliver completions to the CPU on which the I/O was issued. This can improve CPU data caching effectiveness.

Inode table initialization

For very large file systems, the mkfs.ext4 process can take a very long time to initialize all inode tables in the file system. This process can be deferred with the -E lazy_itable_init=1 option. If this is used, kernel processes will continue to initialize the file system after it is mounted. The rate at which this initialization occurs can be controlled with the -o init_itable=n option for the mount command, where the amount of time spent performing this background initialization is roughly 1/n. The default value for n is 10.

Auto-fsync behavior

Because some applications do not always properly fsync() after renaming an existing file, or truncating and rewriting, ext4 defaults to automatic syncing of files after replace-via-rename and replace-via-truncate operations. This behavior is largely consistent with older ext3 filesystem behavior. However, fsync() operations can be time consuming, so if this automatic behavior is not required, use the -o noauto_da_alloc option with the mount command to disable it. This will mean that the application must explicitly use fsync() to ensure data persistence.

Journal I/O priority

By default, journal commit I/O is given a slightly higher priority than normal I/O. This priority can be controlled with the journal_ioprio=n option of the mount command. The default value is 3. Valid values range from 0 to 7, with 0 being the highest priority I/O.

Input traffic

You can help prevent queue overruns by slowing down input traffic. This can be achieved by filtering, reducing the number of joined multicast groups, lowering broadcast traffic, and the like.

Queue length

Alternatively, you can also increase the queue length. This involves increasing the number of buffers in a specified queue to whatever maximum the driver will allow. To do so, edit the rx/tx ring parameters of ethX using:

ethtool --set-ring ethX

Append the appropriate rx or tx values to the aforementioned command. For more information, refer to man ethtool.

Device weight

You can also increase the rate at which a queue is drained. To do this, adjust the NIC's device weight accordingly. This attribute refers to the maximum number of frames that the NIC can receive before the softirq context has to yield the CPU and reschedule itself. It is controlled by the /proc/sys/net/core/dev_weight variable.

Input traffic

The first option is to slow down input traffic by configuring the rate at which the queue fills. To do so, either filter frames or pre-emptively drop them. You can also slow down input traffic by lowering the NIC's device weight^[6].

Queue depth

You can also avoid socket queue overruns by increasing the queue depth. To do so, increase the value of either the rmem_default kernel parameter or the SO_RCVBUF socket option. For more information on both, refer to Section 8.2, “Optimized Network Settings”.

Application call frequency

Whenever possible, optimize the application to perform calls more frequently. This involves modifying or reconfiguring the network application to perform more frequent POSIX calls (such as recv, read). In turn, this allows an application to drain the queue faster.