The resolver is a set of routines in the C library that provides access to the Internet Domain Name System (DNS). The resolver configuration file contains information that is read by the resolver routines the first time they are invoked by a process. The file is designed to be human readable and contains a list of keywords with values that provide various types of resolver information³. The /etc/resolv.conf file for this reference environment consists of two configuration options: nameserver and search. The search option is used to search for a host name that is part of a particular domain. The nameserver option is the IP address of the name server the system oracle1 must query. If more than one nameserver is listed, the resolver library queries them in order. An example of the /etc/resolv.conf file used on the reference environment is shown below.

# cat /etc/resolv.conf
# Generated by NetworkManager
search e2e.bos.redhat.com
nameserver 10.19.114.2

3: Linux man pages - man resolv.conf

The public network configuration consists of two network interfaces bonded together to provide high availability. The example below shows how to bond physical interfaces em1 and em2 with a bond device labeled bond0.

Check the status of Network Manager:

# systemctl status NetworkManager.service
\u25cf NetworkManager.service - Network Manager
   Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-07-18 02:29:13 UTC; 2 days ago
     Docs: man:NetworkManager(8)
 Main PID: 2038 (NetworkManager)
   CGroup: /system.slice/NetworkManager.service
           \u251c\u25002038 /usr/sbin/NetworkManager --no-daemon
           \u2514\u25002140 /sbin/dhclient -d -q -sf /usr/libexec/nm-dhcp-helper -pf /var/run/dhclient-em1.pid -lf /var/lib/NetworkManager/dhclie...

On each node within the environment:

Create a channel bonding interface:

# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
NAME=bond0
BONDING_MASTER=yes
BOOTPROTO=static
IPADDR=10.19.114.44
PREFIX=23
GATEWAY=10.19.115.254
BONDING_OPTS="mode=1 miimon=100 primary=em1"
ONBOOT=yes

On each node within the environment:

Create em1 and em2 as slave interfaces:

# cat /etc/sysconfig/network-scripts/ifcfg-em1
DEVICE=em1
HWADDR="44:a8:42:af:58:66"
ONBOOT=yes
IPV6INIT=no
PEERROUTES=yes
SLAVE=yes
BOOTPROTO="none"
MASTER=bond0

# cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
HWADDR="44:a8:42:af:58:67"
ONBOOT=yes
IPV6INIT=no
PEERROUTES=yes
SLAVE=yes
BOOTPROTO="none"
MASTER=bond0

On each node within the environment:

Restart the network service

# systemctl restart network.service

To ensure NetworkManager is aware of the changes issue the command:

# nmcli con reload

Once the bond0 device is configured on each node, use the ping command to verify connectivity as follows: On node one labeled oracle1,

# ping 10.19.114.48
PING 10.19.114.48 (10.19.114.48) 56(84) bytes of data.
64 bytes from 10.19.114.48: icmp_seq=1 ttl=64 time=0.417 ms

On node two labeled oracle2,

# ping 10.19.114.44
PING 10.19.114.44 (10.19.114.44) 56(84) bytes of data.
64 bytes from 10.19.114.44: icmp_seq=1 ttl=64 time=0.417 ms

SCAN provides a single name in which a client server can use to connect to a particular Oracle database. The main benefit of SCAN is the ability to keep a client connection string the same even if changes within the Oracle RAC Database environment occur, such as adding or removing of nodes within the cluster. The reason this works is because every client connection sends a request to the SCAN Listener, then reroutes the traffic to an available VIP Listener within the Oracle RAC cluster to establish a database connection. The setup of SCAN requires the creation of a single name, no longer than 15 characters in length not including the domain suffix, resolving to three IP addresses using a round-robin algorithm from the DNS server. SCAN must reside in the same subnet as the public network within the Oracle RAC Database cluster and be resolvable without the domain suffix. Within the reference environment, the domain is e2e.bos.redhat.com and SCAN name is db-oracle-scan.

An example DNS entry for the SCAN is as follows:

db-oracle-scan IN A 10.19.115.60
           IN A 10.19.115.73
           IN A 10.19.115.75

An example of the DNS entry for the SCAN to enable reverse lookups is as follows:

60 IN PTR db-oracle-scan.e2e.bos.redhat.com
73 IN PTR db-oracle-scan.e2e.bos.redhat.com
75 IN PTR db-oracle-scan.e2e.bos.redhat.com

On each node within the Oracle RAC cluster, verify the SCAN configuration within the DNS server is setup properly using the nslookup and host command as follows:

# nslookup db-oracle-scan
Server:     10.19.114.2
Address:    10.19.114.2#53

Name:   db-oracle-scan.e2e.bos.redhat.com
Address: 10.19.115.75
Name:   db-oracle-scan.e2e.bos.redhat.com
Address: 10.19.115.60
Name:   db-oracle-scan.e2e.bos.redhat.com
Address: 10.19.115.73

On each node within the Oracle RAC cluster, verify the SCAN configuration reverse lookup is setup properly using the nslookup and host command as follows:

# host db-oracle-scan
db-oracle-scan.e2e.bos.redhat.com has address 10.19.115.73
db-oracle-scan.e2e.bos.redhat.com has address 10.19.115.75
db-oracle-scan.e2e.bos.redhat.com has address 10.19.115.60

# nslookup 10.19.115.75
Server:     10.19.114.2
Address:    10.19.114.2#53

75.115.19.10.in-addr.arpa   name = db-oracle-scan.e2e.bos.redhat.com.

Repeat the above step for the reverse lookup on the remaining IP addresses used for the SCAN.

For more information on SCAN, please refer to Oracle’s documentation³

3: Oracle Single Client Access Name (SCAN) Whitepaper

The virtual IP is an IP address assigned to each node within an Oracle RAC Database environment with the IP address residing in the public subnet. During the installation of the Oracle Grid Infrastructure, each VIP Listener registers with every SCAN Listener. The reason is because when a client sends a request, the SCAN Listener routes the incoming traffic to one of the VIP Listeners within the Oracle RAC Database cluster. If a client connection string uses the VIP to talk directly to the VIP Listener (as done in prior versions), every time changes to the Oracle RAC Database environment are made, such as adding or removing nodes within the cluster, the client connection string would require updating. Due to this, Oracle recommends always using the SCAN for the client connection string.

An example DNS entry for the VIPs is as follows:

db-oracle1-vip IN A 10.19.115.40
db-oracle2-vip IN A 10.19.115.41

On each node within the Oracle RAC cluster, verify the VIP address for oracle1-vip and oracle2-vip within the DNS server is setup properly using the nslookup and host command. An example of checking oracle1-vip can be seen below.

# nslookup oracle1-vip
Server: 10.19.114.2
Address: 10.19.114.2#53
Name: oracle1.e2e.bos.redhat.com
Address: 10.19.115.40

# host oracle1-vip
oracle1-vip.e2e.bos.redhat.com has address 10.19.115.40

An example of the DNS entry for the SCAN to enable reverse lookups is as follows:

40 IN PTR oracle1-vip.e2e.bos.redhat.com
41 IN PTR oracle2-vip.e2e.bos.redhat.com

On each node within the Oracle RAC Database cluster, verify the VIP address reverse lookup for both VIP addresses (10.19.115.40 and 10.19.115.41) is setup properly using the nslookup and host command. An example is shown using VIP address 10.19.115.40 below.

# nslookup 10.19.115.40
Server: 10.19.114.2
Address: 10.19.114.2#53
40.115.19.10.in-addr.arpa name = oracle1-vip.e2e.bos.redhat.com.

# host 10.19.115.40
41.115.19.10.in-addr.arpa domain name pointer oracle-1-vip.e2e.bos.redhat.com.

The private network configuration consists of two network interfaces em3 and em4. The private network provides interconnect communication between all the nodes in the cluster. This is accomplished via Oracle’s Redundant Interconnect, also known as Highly Available Internet Protocol (HAIP), that allows the Oracle Grid Infrastructure to activate and load balance traffic on up to four Ethernet devices for private interconnect communication. The example below shows how to setup physical interfaces em3 and em4 to be used with HAIP.

On each node, set em3 and em4 for private interconnect traffic. An example below:

# cat /etc/sysconfig/network-scripts/ifcfg-p3p1
DEVICE=em3
HWADDR="44:a8:42:af:58:68"
ONBOOT=yes
IPV6INIT=no
BOOTPROTO=static
IPADDR=192.11.1.51
PREFIX=24
MTU=9000

# cat /etc/sysconfig/network-scripts/ifcfg-p3p2
DEVICE=em4
HWADDR="44:a8:42:af:58:69"
ONBOOT=yes
IPV6INIT=no
BOOTPROTO=static
IPADDR=192.12.1.51
PREFIX=24
MTU=9000

ifdown all the interfaces if status was UP during changing the config files.

Restart all interfaces using:

# nmcli con reload

Ensure all private Ethernet interfaces are set to different subnets on each node. If different subnets are not used and connectivity is lost, this can cause a node reboot within the cluster. For the reference environment, subnets 192.11.1.0/24 and 192.12.1.0/24 are used on each node within the Oracle RAC Database cluster.

Verify connectivity on each node using the ping command.

On node one labeled oracle1,

# ping 192.11.1.51
# ping 192.12.1.51

On node two labeled oracle2,

# ping 192.11.1.50
# ping 192.12.1.50

The following section only applies to environments taking advantage of iSCSI storage. If not using an iSCSI storage array, please skip to the following section Section 3.3, “OS Configuration”.

The iSCSI network configuration consists of two network interfaces em3 and em4. Set em3 and em4 for iSCSI traffic. An example below:

# cat /etc/sysconfig/network-scripts/ifcfg-em3
DEVICE=em3
HWADDR="44:a8:42:af:52:61"
ONBOOT=yes
IPV6INIT=no
BOOTPROTO=static
IPADDR=172.17.114.250
PREFIX=24
MTU=9000

# cat /etc/sysconfig/network-scripts/ifcfg-em4
DEVICE=em4
HWADDR="44:a8:42:af:52:62"
ONBOOT=yes
IPV6INIT=no
BOOTPROTO=static
IPADDR=172.17.114.251
PREFIX=24
MTU=9000

Stop and start the network interface

ifdown p3p1; ifdown p3p2; ifup p3p1; ifup p3p2

Verify connectivity on each node using the ping command.

# ping <Equallogic Group IP>

The subscription-manager command registers a system to the Red Hat Network (RHN) and manages the subscription entitlements for a system. The --help option specifies on the command line to query the command for the available options. If the --help option is issued along with a command directive, then options available for the specific command directive are listed.

To use Red Hat Subscription Management for providing packages to a system, the system must first register with the service. In order to register a system, use the subscription-manager command and pass the register command directive. If the --username and --password options are specified, then the command does not prompt for the RHN Network authentication credentials.

An example of registering a system using subscription-manager is shown below.

# subscription-manager register --username [User] --password '[Password]'
The system has been registered with id: abcd1234-ab12-ab12-ab12-481ba8187f60

After a system is registered, it must be attached to an entitlement pool. For the purposes of this reference environment, the Red Hat Enterprise Linux Server is the pool chosen. Identify and subscribe to the Red Hat Enterprise Linux Server entitlement pool, the following command directives are required.

# subscription-manager list --available | grep -A8 "Red Hat Enterprise Linux
Server"
Subscription Name: Red Hat Enterprise Linux Server, Standard (8 sockets)
(Unlimited guests)
Provides: Red Hat Beta
 Oracle Java (for RHEL Server)
 Red Hat Enterprise Linux Server
 Red Hat Software Collections Beta (for RHEL Server)
SKU: RH0186633
Contract: 10541483
Pool ID: <poolid>
Available: 47
Suggested: 1
Service Level: STANDARD
Service Type: L1-L3

# subscription-manager attach --pool <pool_id>
Successfully attached a subscription for: Red Hat Enterprise Linux Server,
Standard (8 sockets) (Unlimited guests)

The Red Hat Enterprise Linux supplementary repository is part of subscribing to the Red Hat Enterprise Linux Server entitlement pool, however, it is disabled by default. Enable the supplementary repository via the subscription-manager command.

# subscription-manager repos --enable=rhel-7-server-optional-rpms
Repo 'rhel-7-server-optional-rpms' is enabled for this system.

For more information on the use of Red Hat Subscription Manager, please visit the Red Hat Subscription management documentation⁴.

4: Red Hat Subscription Management

The chronyd — is a daemon for synchronisation of the system clock. It can synchronise the clock with NTP servers, reference clocks (e.g. a GPS receiver), and manual input using wristwatch and keyboard via chronyd. It can also operate as an NTPv4 (RFC 5905) server and peer to provide a time service to other computers in the network⁵.

5: chronyd - chronyd daemon man page – man chronyd (8)

In order to configure the chronyd daemon, on each node follow the instructions below.

A specific set of packages is required to properly deploy Oracle RAC Database 12c Release 2 on Red Hat Enterprise Linux 7. The number of installed packages required varies depending on whether a default or minimal installation of Red Hat Enterprise Linux 7 (x86_64) is performed. For the purposes of this reference environment, a minimal Red Hat Enterprise Linux 7 installation is performed to reduce the number of installed packages. A sample kickstart file as been provided within Appendix H, Sample Kickstart File. Red Hat Enterprise Linux 7 installation requires the following group packages:

Oracle Grid Infrastructure and Oracle Database 12c Release 2 required x86_64 RPM packages.

After the installation of Red Hat Enterprise Linux 7 is completed, create a file, req-rpm.txt, that contains the name of each RPM package listed above on a separate line. For simplicity, this req-rpm.txt file is included in Appendix D, Oracle Database Package Requirements Text File.

Within each node:

Use the yum package manager to install the packages and any of their dependencies with the following command:

# yum install `awk '{print $1}' ./req-rpm.txt`

A minimum installation of Red Hat Enterprise Linux 7 does not install the X Window System server package, but only the required X11 client libraries. In order to run the Oracle Universal Installer (OUI), a system with the X Window System server package installed is required.

Using a system with X Window System installed, ssh into each Oracle RAC Database server with the -Y option to ensure trusted X11 forwarding is set. The command is as follows:

# ssh -Y oracle1.e2e.bos.redhat.com

Alternatively, if a system with the X Window System server package is unavailable, install the X Window System server package directly on node one of the Oracle RAC cluster.

SELinux is an implementation of a mandatory access control (MAC) mechanism developed by the National Security Agency (NSA). The purpose of SELinux is to apply rules on files and processes based on defined policies. When policies are appropriately defined, a system running SELinux enhances application security by determining if an action from a particular process should be granted thus protecting against vulnerabilities within a system. The implementation of Red Hat Enterprise Linux 7 enables SELinux by default and appropriately sets it to the default setting of ENFORCING.

It is highly recommended that SELinux be kept in ENFORCING mode when running Oracle RAC Database 12c Release 2.

Verify that SELinux is running and set to ENFORCING:

As the root user on each node,

# sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

If the system is running in PERMISSIVE or DISABLED mode, modify the /etc/selinux/config file and set SELinux to enforcing as shown below.

SELINUX=enforcing

The modification of the /etc/selinux/config file takes effect after a reboot. To change the setting of SELinux immediately without a reboot, run the following command:

# setenforce 1

For more information on Security­ Enhanced Linux, please visit the  Red Hat Enterprise Linux 7 Security ­Enhanced Linux User Guide

Firewall access and restrictions play a critical role in securing your Oracle RAC Database 12c Release 2 environment. It is not uncommon for corporations to be running hardware based firewalls to protect their corporate networks. Due to this, enabling firewall may not be required. However, this reference environment demonstrates how to successfully implement firewall settings for an Oracle RAC Database environment. The firewall rules described below only apply to the public network. Oracle recommends the private network should not have any firewall rules as this can cause issues with the installation of Oracle RAC Database, as well as, disruption with the Oracle RAC Database private interconnect. It is highly recommended that the private network be isolated and communicate only between nodes locally. Red Hat Enterprise Linux 7 introduces the use of firewalld, a dynamic firewall daemon, instead of the traditional iptables service. firewalld works by assigning network zones to assign a level of trust to a network and its associated connections and interfaces⁶. The key difference and advantage of firewalld over the iptables service is that it does not require flushing of old firewall rules to apply the new firewall rules. firewalld changes the settings during runtime without losing existing connections⁶. With the implementation of firewalld, the iptables service configuration file /etc/sysconfig/iptables does not exist. It is recommended that the firewall settings be configured to permit access to the Oracle RAC Database network ports only from authorized database or database-management clients. For example, in order to allow access to a specific database client with an IP address of 10.19.142.54 and to make requests to the database server via SQL*Net using Oracle’s TNS (Transparent Network Substrate) Listener (default port of 1521), the following permanent firewall rule within the public zone must be added to the firewalld configuration.

On all nodes within the Oracle RAC cluster unless otherwise specified,

# firewall-cmd --permanent --zone=public --add-rich-rule=”rule family=”ipv4”
source address=”10.19.142.54” port protocol=”tcp” port=”1521” accept”
success

Likewise, if a particular database client with an IP address of 10.19.142.54 required access to the web-based EM Express that uses the default port of 5500, the following firewall rich rule must be added using the firewall-cmd command.

# firewall-cmd --permanent --zone=public --add-rich-rule=”rule family=”ipv4”
source address=”10.19.142.54” port protocol=”tcp” port=”5500” accept”
success

Ensure the firewall allows all traffic from the private network by accepting all traffic (trusted) from the private Ethernet interfaces em3 and em4 from all nodes within the Oracle RAC cluster. It is highly recommended that the private network be isolated and communicate only between nodes locally.

# firewall-cmd --permanent -–zone=trusted -–change-interface=em3
success
# firewall-cmd --permanent -–zone=trusted –-change-interface=em4
success

The following rules are added to satisfy the Oracle Installer’s prerequisites. Once Oracle installation is complete, the following rules can be removed. Steps for removal shown upon the completion of the Oracle RAC Database installation in Section 6.1, “Removal of firewalld Trusted Source Address”

On node one of the Oracle RAC cluster, add the public source IP of all remaining nodes. This reference environment only adds the public IP of node two of the Oracle RAC cluster as it is a two-node Oracle RAC Database environment.

# firewall-cmd -–permanent --zone=trusted -–add-source=10.19.114.48/23

On node two of the Oracle RAC cluster, add the public source IP of all remaining nodes. This reference environment only adds the public IP of node one of the Oracle RAC cluster as it is a two-node Oracle RAC Database environment.

# firewall-cmd -–permanent --zone=trusted -–add-source=10.19.114.44/23

Once the rules have been added, run the following command to activate:

# systemctl restart firewalld.service

To verify the port 1521 & 5500 has been added and database client with IP address of 10.19.142.54 has been properly added, run the following command:

# firewall-cmd --zone=public --list-all
public (default, active)
 interfaces: bond0 em1 em2
 sources:
 services: dhcpv6-client ssh
 ports:
 masquerade: no
 forward-ports:
 icmp-blocks:
 rich rules:
rule family="ipv4" source address="10.19.142.54" port port="1521"
protocol="tcp" accept
rule family="ipv4" source address="10.19.142.54" port port="5500"
protocol="tcp" accept

To verify the firewall rules being applied to the trusted zone for the private Ethernet interfaces, and temporarily for the source public IP run the following command:

Example of oracle1

# firewall-cmd --zone=trusted --list-all
trusted (active)
 interfaces: em3 em4
 sources: 10.19.114.48/23
 services:
 ports:
 masquerade: no
 forward-ports:
 icmp-blocks:
 rich rules:

6: Red Hat Enterprise Linux 7 – Using Firewalls

The following sections regarding virtual memory, shared memory, semaphores, network ports, I/O synchronous requests, file handles, and kernel panic on OOPS parameters provide a detailed explanation of what these parameters are and their effect in an Oracle deployment. It is recommended to read carefully each parameter for a better understanding on how to tweak a specific environment for a particular workload.

Each section provides the manual steps to tweaking the parameters. With that said, if looking to tweak parameters immediately, Section 3.4.6, “Optimizing Database Storage using Automatic System Tuning” covers setting the parameters using the oracle-tuned-profile.

Tuning virtual memory requires the modification of five kernel parameters that affect the rate that virtual memory is used within Oracle RAC databases.

A brief description⁷ and recommended settings for the virtual memory parameters, as well as, the definition of dirty data are described below.

SWAPPINESS⁷ - Starting with Red Hat Enterprise Linux 6.4 and above, the definition of swappiness has changed. Swappiness is defined as a value from 0 to 100 that controls the degree to which the system favors anonymous memory or the page cache. A high value improves file-system performance, while aggressively swapping less active processes out of memory. A low value avoids swapping processes out of memory, that usually decreases latency, at the cost of I/O performance. The default value is 60.

DIRTY DATA – Dirty data is data that has been modified and held in the page cache for performance benefits. Once the data is flushed to disk, the data is clean.

DIRTY_RATIO⁷ – Contains, as a percentage of total system memory, the number of pages at which a process that is generating disk writes will itself start writing out dirty data. The default value is 20. The recommended value is between 40 and 80. The reasoning behind increasing the value from the standard Oracle 15 recommendation to a value between 40 and 80 is because dirty ratio defines the maximum percentage of total memory that be can be filled with dirty pages before user processes are forced to write dirty buffers themselves during their time slice instead of being allowed to do more writes. All processes are blocked for writes when this occurs due to synchronous I/O, not just the processes that filled the write buffers. This can cause what is perceived as unfair behavior where a single process can hog all the I/O on a system. As the value of dirty_ratio is increased, it is less likely that all processes will be blocked due to synchronous I/O, however, this allows for more data to be sitting in memory that has yet to be written to disk.

DIRTY_BACKGROUND_RATIO⁷ – Contains, as a percentage of total system memory, the number of pages that the background write back daemon will start writing out dirty data. The Oracle recommended value is 3.

DIRTY_EXPIRE_CENTISECS⁷ - Defines when dirty in-memory data is old enough to be eligible for writeout. The default value is 3000, expressed in hundredths of a second. The Oracle recommended value is 500.

DIRTY_WRITEBACK_CENTISECS⁷ - Defines the interval of when writes of dirty in-memory data are written out to disk. The default value is 500, expressed in hundredths of a second. The Oracle recommended value is 100.

Create a file labeled 98-oracle-kernel.conf within /etc/sysctl.d/

# vi 98-oracle-kernel.conf
vm.swappiness = 1
vm.dirty_background_ratio = 3
vm.dirty_ratio = 80
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100

For the changes to take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

7: RHEL7 Kernel Documentation (requires package kernel-doc to be installed) - /usr/share/doc/kernel-doc-3.10.0/Documentation/sysctl/vm.txt

Shared memory allows processes to communicate with each other by placing regions of memory into memory segments. In the case of Oracle, shared memory segments are used by the System Global Area (SGA) to store incoming data and control information. The size of Oracle’s SGA impacts the amount of shared memory pages and shared memory segments to be set within a system. By default, Red Hat Enterprise Linux 7 provides a large amount of shared memory pages and segments. However, the appropriate allocation for a system depends on the size of the SGA within an Oracle database instance. In order to allocate the appropriate amount of shared memory pages and shared memory segments for a system running an Oracle database, the kernel parameters SHMMAX, SHMALL, and SHMMNI must be set.

SHMMAX – is the maximum size in bytes of a single shared memory segment

SHMALL – is the maximum total amount of shared memory pages

SHMMNI – is the maximum total amount of shared memory segments

A default installation of Red Hat Enterprise Linux 7.0 x86_64 provides a maximum size of a single shared memory segment, SHMMAX, to 4294967295 bytes, equivalent to 4 GB -1 byte. This value is important since it regulates the largest possible size of one single Oracle SGA shared memory segment. If the Oracle SGA is larger than the value specified by SHMMAX (default 4 GB-1 byte), then Oracle is required to create multiple smaller shared memory segments to completely fit Oracle’s SGA. This can cause a significant performance penalty, especially in NUMA environments. In an optimal NUMA configuration, a single shared memory segment for Oracle’s SGA is created on each NUMA node. If SHMMAX is not properly sized and creates multiple shared memory segments, SHMMAX limitations may keep the system from evenly distributing the shared memory segments across each NUMA node.

Starting with Red Hat Enterprise Linux 7.1 and above, SHMMAX default value is set to 18446744073692774399 bytes, equivalent to roughly 18 petabytes. Due to this, there is no need to calculate SHMMAX because of the very large size already provided. It is recommended to use the value set in Red Hat Enterprise Linux 7.1 and above because the value is purposely set higher than the architectural memory limits to ensure that any Oracle SGA value set within an Oracle database instance may fit in one single shared memory segment.

The value of SHMMAX can be confirmed via the command:

# sysctl kernel.shmmax

The next step is to determine the maximum amount of shared memory pages (SHMALL) in a system by capturing system’s page size in bytes. The following command can be used to obtain the system page size.

# getconf PAGE_SIZE
4096

A default installation of Red Hat Enterprise Linux 7.0 x86_64 provides a SHMALL value of 268435456 pages, the equivalent of 1 TB in system pages. This is determined by the following formula:

SHMALL IN BYTES * PAGE_SIZE

Starting with Red Hat Enterprise Linux 7.1 and above, SHMALL default value is 18446744073692774399 pages, the same value set to SHMMAX.

The value of SHMALL can be confirmed via the command:

# sysctl kernel.shmall

To ensure an adequate amount of memory pages are allocated to a single Oracle SGA, it is recommended that the value of SHMALL be set to the at least the value using the following formula:

SHMMAX IN BYTES / PAGE_SIZE

Since the default value of SHMALL in Red Hat Enterprise Linux 7.1 and above is 18446744073692774399 pages, and the minimum recommended value by Oracle for SHMALL is 1073741824, the larger default value is kept.

SHMMNI is the maximum total amount of shared memory segments. A default installation of Red Hat Enterprise Linux 7 x86_64 provides a SHMMNI default value of 4096. By Red Hat Enterprise Linux 7 optimizing the SHMMAX value with one shared memory segment per Oracle SGA, this parameter reflects the maximum number of Oracle and ASM instances that can be started on a system. Oracle recommends the value of SHMMNI to be left at the default value of 4096.

Prior to Red Hat Enterprise Linux 7.1, changes to the kernel parameters were required. However, with the new SHMMAX, SHMALL, and SHMMNI defaults no changes are made.

Red Hat Enterprise Linux 7 provides semaphores for synchronization of information between processes. The kernel parameter sem is composed of four parameters:

SEMMSL – is defined as the maximum number of semaphores per semaphore set

SEMMNI – is defined as the maximum number of semaphore sets for the entire system

SEMMNS – is defined as the total number of semaphores for the entire system

SEMOPM – is defined as the total number of semaphore operations performed per semop system call.

The following line is required within the /etc/sysctl.d/98-oracle-kernel.conf file to provide default values for semaphores for Oracle:

kernel.sem = 250 32000 100 128

For the changes to take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

The values above are sufficient for most environments and no tweaking should be necessary. However, the following describes how these values can be optimized and should be set when the defaults don’t suffice.

Example errors:

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpcreates

Semaphores are used by Oracle for internal locking of SGA structures. Sizing of semaphores directly depends on only the PROCESSES parameter of the instance(s) running on the system. The number of semaphores to be defined in a set should be set to a value that minimizes the waste of semaphores.

For example, say our environment consists of two Oracle instances with the PROCESSES set to 300 for database one and 600 for database two. With SEMMSL set at 250 (default), the first database requires 2 sets. The first set is 250 semaphores but an additional 50 semaphores is required thus an additional SEMMSL set is required thus wasting 200 semaphores. Our 2nd instance requires 3 sets, set one 250 semaphores, set two 250 semaphores, giving us a total of 500, but an additional 100 semaphores is required thus adding an additional SEMMSSL set wasting 150 semaphores. A better value of SEMMSL in this particular case would be 150. With SEMMSL set at 150, the first database requires two sets (wasting zero semaphores), our second instance requires four sets (wasting zero semaphores). This is an ideal example, and most likely some semaphore wastage is expected and okay as semaphores in general consume small amounts of memory. As more databases are created in an environment, these calculations may get complicated. In the end, the goal is to limit semaphore waste.

Regarding SEMMNI, SEMMNI should be set high enough for proper amount of sets to be available on the system. Using the value of SEMMSL, one can determine max amount of SEMMNI required. Round up to the nearest power of 2.

SEMMNI = SEMMNS/SEMMSL

Oracle requires 2x value of PROCESSES in the init.ora parameter for semaphores (SEMMNS value) on startup of the database, then half of those semaphores are released. To properly size SEMMNS, one must know the sum of all PROCESSES set across all instances on the host. SEMMNS should best be set higher than SEMMNI*SEMMSL value (this is how we get 32000 for default value 250*128)

SEMOP is calculated using the total SEMMNI divided by SEMMSL. In the default scenario that is 3200/250 = 128

Oracle recommends that the ephemeral default port range be set starting at 9000 to 65500. This ensures that all well known ports used by Oracle and other applications are avoided. To set the ephemeral port range, modify the /etc/sysctl.d/98-oracle-kernel.conf file and add the following line:

net.ipv4.ip_local_port_range = 9000 65500

For the changes to take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

Optimizing the network settings for the default and maximum buffers for the application sockets in Oracle is done by setting static sizes to RMEM and WMEM. The RMEM parameter represents the receive buffer size, while the WMEM represents the send buffer size. The recommended values by Oracle are configured within the /etc/sysctl.conf file.

net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576

For the changes to take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

On each node within the Oracle RAC Database cluster, set the value of NOZEROCONF to yes within the /etc/sysconfig/network file. Setting NOZEROCONF ensures that the route 169.254.0.0/16 is not added to the routing table.

NOZEROCONF=yes

From the Red Hat Customer Portal article: https://access.redhat.com/solutions/25463,

The Avahi website defines Avahi as: 'a system which facilities service discovery on a local network. This helps to plug the laptop or computer into a network and instantly be able to view other people who you can chat with, find printers to print to, or find files being shared…​' Avahidaemon (on by default on Red Hat Enterprise Linux) can interfere with Oracle RAC’s multicast heartbeat causing the application-layer interface to assume it has been disconnected on a node and reboot the node. It is not recommended to remove the package due to its many dependencies. The avahi libraries are being used by many packages on a system. On each node within the Oracle RAC Database cluster, stop and disable the avahi services run the following commands:

# systemctl stop avahi-dnsconfd
# systemctl stop avahi-daemon
Warning: Stopping avahi-daemon, but it can still be activated by:
 avahi-daemon.socket

To keep the avahi services off persistently across reboots, on each node run the following:

# systemctl disable avahi-dnsconfd
# systemctl disable avahi-daemon
rm '/etc/systemd/system/dbus-org.freedesktop.Avahi.service'
rm '/etc/systemd/system/multi-user.target.wants/avahi-daemon.service'
rm '/etc/systemd/system/sockets.target.wants/avahi-daemon.socket'

The kernel parameter FS.AIO-MAX-NR sets the maximum number of current asynchronous I/O requests. Oracle recommends setting the value to 1048576. In order to add FS-AIO-MAX-NR to 1048576, modify the /etc/sysctl.d/98-oracle-kernel.conf file on each node of the Oracle RAC cluster as follows:

fs.aio-max-nr = 1048576

In order for the changes take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

The kernel parameter FS.FILE-MAX sets the maximum number of open file handles assigned to the Red Hat Enterprise Linux 7 operating system. Oracle recommends that for each Oracle RAC database instance found within a system, allocate 512*PROCESSSES in addition to the open file handles already assigned to the Red Hat Enterprise Linux 7 operating system. PROCESSES within a database instance refers to the maximum number of processes that can be concurrently connected to the Oracle RAC database by the oracle user. The default value for PROCESSES is 2560 for Oracle RAC Database 12c Release 2. To properly calculate the FS.FILE-MAX for a system, first identify the current FS.FILE-MAX allocated to the system via the following command:

sysctl fs.file-max

Next, add all the PROCESSES together from each Oracle RAC database instance found within the system and multiple by 512 as seen in the following command.

# echo “512 * 2560” | bc

SQL> show parameter processes;
NAME                             TYPE VALUE
------------------------------------ -----------
processes                        integer 2560

Finally, add the current FS.FILE-MAX value with the new value found from multiplying 512*PROCESSES to attain the new FS.FILE-MAX value.

While the value of the FS.FILE-MAX parameter varies upon every environment, this reference environment uses the default value within Red Hat Enterprise Linux 7.4 (9784283). Oracle recommends a value no smaller than 6815744. In order to modify the value of FS.FILE-MAX, add to the_/etc/sysctl.d/98-oracle-kernel.conf_ file as follows:

fs.file-max = <value>

In order for the changes take effect immediately, run the following command:

# sysctl -p etc/sysctl.d/98-oracle-kernel.conf

In order for the changes take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf

Red Hat Enterprise Linux 7 defaults to the use of Strict Reverse Path filtering. The reason strict mode is the default is to prevent IP spoofing from Distributed Denial-of-service (DDos) attacks. However, having strict mode enabled on the private interconnect of an Oracle RAC database cluster may cause disruption of interconnect communication. It is recommended to set the RP_FILTER from strict mode to loose mode. Loosening the security on the private Ethernet interfaces should not be of concern as best practices recommend for an isolated private network that can only communicate between nodes specifically for Oracle’s private interconnect.

To satisfy the Oracle Installer prerequisite, add the following modifications to the /etc/sysctl.d/98-oracle-kernel.conf on each node of the Oracle RAC cluster as follows:

net.ipv4.conf.em3.rp_filter = 2
net.ipv4.conf.em4.rp_filter = 2

In addition to the above, please include the following modification within the 98-oracle-kernel.conf file located within the /etc/sysctl.d/ directory. The 98-oracle-kernel.conf file can be found within Appendix E, Kernel Parameters (98-oracle-kernel.conf)

In order for the changes take effect immediately, run the following command on each node of the Oracle RAC cluster:

# sysctl -p /etc/sysctl.d/98-oracle-kernel.conf
[Output Appreciated ...]
net.ipv4.conf.em3.rp_filter = 2
net.ipv4.conf.em4.rp_filter = 2

# sysctl -p //etc/sysctl.d/98-oracle-kernel.conf
net.ipv4.conf.em3.rp_filter = 2
net.ipv4.conf.em4.rp_filter = 2

Prior to the installation of Oracle RAC Database 12c Release 2, Oracle recommends the creation of a grid user for the Oracle Grid Infrastructure and an oracle user for the Oracle RAC Database software installed on the system.

For the purposes of this reference environment, the Oracle Grid Infrastructure owner is the user grid and the Oracle Database software owner is the user oracle. Each user is designated different groups to handle specific roles based on the software installed. However, the creation of separate users requires that both the oracle user and the grid user have a common primary group, the Oracle central inventory group (OINSTALL).

The following are the recommended system groups created for the installation of the Oracle RAC Database and part of the oracle user.

OSDBA group (DBA) – determines OS user accounts with DBA privileges

OSOPER group (OPER) – an optional group created to assign limited DBA privileges (SYSOPER privilege) to particular OS user accounts

OSBACKUPDBA group (BACKUPDBA) – an optional group created to assign limited administrative privileges (SYSBACKUP privilege) to a user for database backup and recovery

OSDGDBA group (DGDBA) – an optional group created to assign limited administrative privileges (SYSDG privilege) to a user for administering and monitoring Oracle Data Guard

OSKMDBA group (KMDBA) – an optional group created to assign limited administrative privileges (SYSKM privilege) to a user for encryption key management when using Oracle Wallet Manager

OSRACDBA group (RACDBA privilege) - grants the SYSRAC privileges to perform administrative tasks on an Oracle RAC cluster.

The following are the recommended system groups created for the installation of the Oracle Grid Infrastructure and part of the grid user:

OSDBA group (ASMDBA privilege) – provides administrative access to Oracle ASM instances

OSASM group (ASMADMIN privilege) – provides administrative access for storage files via the SYSASM privilege

OSOPER group (ASMOPER privilege) – an optional group created to assign limited DBA privileges with regards to ASM to particular OS user accounts

OSRACDBA group (RACDBA privilege) - grants the SYSRAC privileges to perform administrative tasks on an Oracle RAC cluster.

As the root user on each node, create the following user accounts, groups, and group assignments using a consistent UID and GID assignments across your organization:

# groupadd --gid 54321 oinstall
# groupadd --gid 54322 dba
# groupadd --gid 54323 asmdba
# groupadd --gid 54324 asmoper
# groupadd --gid 54325 asmadmin
# groupadd --gid 54326 oper
# groupadd --gid 54327 backupdba
# groupadd --gid 54328 dgdba
# groupadd --gid 54329 kmdba
# groupadd --gid 54330 racdba
# useradd --uid 54321 --gid oinstall --groups dba,oper,asmdba,racdba,\
> backupdba,dgdba,kmdba oracle
# passwd oracle
# useradd --uid 54322 --gid oinstall --groups dba,asmadmin,asmdba,asmoper,\
> racdba grid
# passwd grid

Verify the oracle and grid user on each Oracle RAC database cluster node correctly displays the appropriate primary and supplementary groups via the commands:

# id oracle
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba),
54323(asmdba),54326(oper),54327(backupdba),54328(dgdba),54329(kmdba),
54330(racdba)

# id grid
uid=54322(grid) gid=54321(oinstall) groups=54321(oinstall),54322(dba),
54323(asmdba),54324(asmoper),54325(asmadmin),54330(racdba)

Oracle recommends the following settings for the soft and hard limits for the number of open file descriptors (nofile), number of processes (nproc), and size of the stack segment (stack) allowed by each user respectively. The purpose of setting these limits is to prevent a system wide crash that could be caused if an application, such as Oracle, were allowed to exhaust all of the OS resources under an extremely heavy workload.

On each node, create a file labeled 99-grid-oracle-limits.conf within /etc/security/limits.d/ as follows:

# touch /etc/security/limits.d/99-grid-oracle-limits.conf

Within the /etc/security/limits.d/99-grid-oracle-limits.conf file, add the following soft and hard limits for the oracle and grid user:

oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768

Due to Bug 1597142116, the soft limit of nproc is not adjusted at runtime by the Oracle database. Due to this, if the nproc limit is reached, the Oracle database may become unstable and not be able to fork additional processes. A high enough value for the maximum number of concurrent threads for the given workload must be set, or use the hard limit value of 16384 as done above if in doubt.

8: What order are limits files in the limits.d directory read in?

As the root user on each node, create a shell script labeled oracle-grid.sh within /etc/profile.d/ to create the ulimits for the oracle and grid user. The contents of the oracle-grid.sh script:

#Setting the appropriate ulimits for oracle and grid user
if [ $USER = "oracle" ]; then
 if [ $SHELL = "/bin/ksh" ]; then
 ulimit -u 16384
 ulimit -n 65536
 else
 ulimit -u 16384 -n 65536
 fi
fi
if [ $USER = "grid" ]; then
 if [ $SHELL = "/bin/ksh" ]; then
 ulimit -u 16384
 ulimit -n 65536
 else
 ulimit -u 16384 -n 65536
 fi
fi

As oracle and grid user, verify the ULIMIT values by running the following command:

# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 385878
max locked memory (kbytes, -l) 14854144
max memory size (kbytes, -m) unlimited
open files (-n) 65536
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 16384
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

This section applies to users taking advantage of iSCSI storage. If not using iSCSI storage, please skip to section Section 3.4.3, “Device Mapper Multipath”.

For security purposes, CHAP (Challenge-Handshake Authentication Protocol) is used to validate the identity of the node(s) connecting to it. The process includes creating a secret username and password to authenticate on each node(s). The details on enabling CHAP within the iSCSI storage itself may vary depending on the vendor. Within the Dell EqualLogic PS Array the steps are as follows:

Once the CHAP user is created within the iSCSI storage array, the following steps are required for each Oracle node(s).

The following section provides steps in connecting the Dell EqualLogic iSCSI volumes to be used for the Oracle installation.

As the root user on each Oracle node,

Device mapper multipath provides the ability to aggregate multiple I/O paths to a newly created device mapper mapping to achieve high availability, I/O load balancing, and persistent naming. The following procedures provide the best practices to installing and configuring device mapper multipath devices.

The following instructions are required on each node within the Oracle RAC Database 12c cluster.

On the first node of the Oracle RAC Database cluster, create a partition for each device mapper volume (ocrvote1, ocrvote2, ocrvote3, db1,db2,fra,redo) using parted as displayed below for device db1.

# parted /dev/mapper/db1 mklabel gpt mkpart primary "1 -1"
Information: You may need to update /etc/fstab.

Once the partition is created, a newly created device mapper device is created as db1p1.

# ls -l /dev/mapper/db1p1
lrwxrwxrwx. 1 root root 8 Apr 16 15:15 /dev/mapper/db1p1 -> ../dm-11

On all the nodes in the Oracle RAC Database cluster, run:

# kpartx -a /dev/mapper/db1

If the following kpartx command does not add the p1 suffix to the partitions ending in a number, reboot all the nodes in the Oracle RAC Database cluster.

The configuration of Oracle ASM requires the use of either udev rules, Oracle ASMLib or Oracle ASM Filter Driver.

The following table provides key considerations between udev rules, Oracle ASMLib and Oracle ASM Filter Driver (ASMFD).

This reference architecture takes advantage of Red Hat’s native device manager udev rules as the method of choice for configuring Oracle ASM disks. For more information on Oracle ASM Filter Driver and installation method, visit: Administering Oracle ASM Filter Driver

The tuned package in Red Hat Enterprise Linux 7 is recommended for automatically tuning the system for common workloads via the use of profiles. Each profile is tailored for different workload scenarios such as: throughput performance, balanced, & high network throughput.

In order to simplify the tuning process for Oracle databases, the creation of a custom oracle profile labeled tuned-profiles-oracle resides in the rhel-7-server-optional-rpms repository. The tuned-profiles-oracle profile uses the throughput performance profile as its foundation and additionally sets all the different parameters mentioned in previous sections of this reference architecture and disables Transparent HugePages (THP) for Oracle databases workload environments.

For more information on why THP is disabled, see Section 4.5, “Enabling HugePages”. Table 3.8, “Profile Tuned Profile Comparsion” provides details between the balanced profile, throughput-performance profile, and the custom profile tuned-profiles-oracle.

The following procedures provide the steps that are required to install, enable, and select the tuned-profiles-oracle profile.

On each node within the Oracle RAC Database cluster, as the root user,

# systemctl stop tuned.service
# systemctl disable tuned.service

#kernel.shmmax = 4398046511104
#kernel.shmall = 1073741824
#kernel.shmmni = 4096
#fs.file-max = 6815744
#kernel.panic_on_oops = 1

#
# tuned configuration
#

[main]
summary=Optimize for Oracle RDBMS
include=throughput-performance

[sysctl]
vm.swappiness = 1
vm.dirty_background_ratio = 3
vm.dirty_ratio = 80
vm.dirty_expire_centisecs = 500
vm.dirty_writeback_centisecs = 100
#kernel.shmmax = 4398046511104
#kernel.shmall = 1073741824
#kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
#fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
#kernel.panic_on_oops = 1
net.ipv4.conf.em3.rp_filter = 2
net.ipv4.conf.em4.rp_filter = 2

[vm]
transparent_hugepages=never

# systemctl restart tuned.service