17.2. Configuring Very Large Memory (VLM)

The following example shows how to use the RAM disk ramfs to allocate 8 GB of shared memory for the Oracle 10g database buffer cache on a 32 bit Red Hat Enterprise Linux 3, 4 or 5 systems (hugemem kernel). If this setup is performed on a server that does not have enough RAM, then Linux will appear to hang and the kernel will automatically start killing processes due to memory shortage (ramfs is not swappable). Furthermore, ramfs is not backed by Huge Pages and therefore the Huge Pages pool should not be increased for database buffers, see Chapter 6, Swap Space. In fact, if there are too many Huge Pages allocated, then there may not be enough memory for ramfs.
Since ramfs is not swappable, it is by default only usable by root. If you put too much on a ramfs file system, you can easily hang the system. To mount the ramfs file system and to make it usable for the Oracle account, execute: 
# umount /dev/shm 
# mount -t ramfs ramfs /dev/shm 
# chown oracle:dba /dev/shm
When Oracle starts it will create a file in the /dev/shm directory that corresponds to the extended buffer cache. Ensure to add the above lines to /etc/rc.local. If ointall is the primary group of the Oracle account, use chown oracle:oinstall /dev/shm instead. For security reasons you do not want to give anyone write access to the shared memory file system. Having write access to the ramfs file system allows you to allocate and pin a large chunk of memory in RAM. In fact, you can kill a machine by allocating too much memory in the ramfs file system.
To enable VLM, set the Oracle parameter use_indirect_data_buffers to true: 
use_indirect_data_buffers=true
For 10g R1 and R2 databases it is important to convert DB_CACHE_SIZE and DB_xK_CACHE_SIZE parameters to DB_BLOCK_BUFFERS, and to remove SGA_TARGET if set. Otherwise you will get errors like these: 
ORA-00385: cannot enable Very Large Memory with new buffer cache parameters
Here is an example for configuring a 8 GB buffer cache for a 10g R2 database with Red Hat Enterprise Linux 3, 4 or 5 hugemem kernels: 
use_indirect_data_buffers=true 
db_block_size=8192 
db_block_buffers=1048576 
shared_pool_size=2831155200
Note that shmmax needs to be increased for shared_pool_size to fit into the System V shared memory. In fact, it should be slightly larger than the SGA size. Since shared_pool_size is less than 3 GB in this example, shmmax does not need to be larger than 3GB. The 8 GB indirect buffer cache will be in the RAM disk and hence it does not have to be accounted for in shmmax. On a 32 bit system the shmmax kernel parameter cannot be larger than 4GB, see also Section 7.3, “Setting SHMALL Parameter”.
In order to allow oracle processes to lock more memory into its address space for the VLM window size, the ulimit parameter memlock must be changed for oracle. Ensure to set memlock in /etc/security/limits.conf to 3145728: 
oracle           soft    memlock         3145728
oracle           hard    memlock         3145728
Login as Oracle again and check max locked memory limit: 
$ ulimit -l
3145728
If it is not working after a ssh log in, then you may have to set the SSH parameter UsePrivilegeSeparation, see Chapter 22, Setting Shell Limits for Your Oracle User.
If memlock is not set or too small, you will get error messages similar to this one: 
ORA-27103: internal error
Linux Error: 11: Resource temporarily unavailable
Now try to start the database. Note the database start up can take a while. Also, the sqlplus banner or show sga may not accurately reflect the actual SGA size in older Oracle versions.
The 8GB file for the database buffer cache can be seen in the ramfs shared memory file system: 
$ ls -al /dev/shm
total 120 
drwxr-xr-x    1 oracle   dba             0    Nov 20 16:29  . 
drwxr-xr-x   22 root     root       118784    Nov 20 16:25  .. 
-rw-r-----    1 oracle   dba    8589934592    Nov 20 16:30  ora_orcl_458754 
$
If the shared pool size is configured too large, you will get error messages similar to this one: 
ORA-27103: internal error
Linux Error: 12: Cannot allocate memory