How can I collect more information from a failure of an oracledb, orainstance, or oralistener resource in a RHEL 5 or 6 High Availability cluster?

Solution Verified - Updated -

Environment

  • Red Hat Enterprise Linux (RHEL) 5 or 6 with the High Availability Add On
  • rgmanager
  • One or more oracledb, orainstance, or oralistener resources from which extra verbosity is desired, such as if that resource is failing an operation

Issue

  • My oracledb resource is failing to start, but I don't get anything useful in the logs. Is there any way to get more verbose output?
  • If I try to start an oracledb or orainstance resource via rg_test, I see no more output on the command line than I normally do when enabling the service with clusvcadm -e. How can I determine why the operation is failing?
  • Where does verbose output from the Oracle High Availability resource agents go?
  • How can I generate debug logging for Oracle resources?

Root Cause

On more recent releases of resource-agents or rgmanager, the Oracle-related resource-agents (oracledb, orainstance, oralistener) have a modified logging method implemented that causes them to send their more verbose output to the daemon syslog facility using the logger utility. This mechanism is used for most outputs from the agent, including for the output from Oracle-specific processes and utilities that may give useful information about why an operation is failing or why something is not behaving as expected. The end result is that unless daemon.* is captured in syslog, then those messages may not be seen anywhere and certain failures or behaviors may be left unexplained.

The common reaction to such failures is to turn to rg_test to run the resource operation manually to get extra verbosity on the command line, but with these Oracle resource agents and the newer package releases, this generally won't produce any extra useful information, because those log messages are still going to daemon instead of to the command line as would occur with other resource types. As such, on these newer package releases, its best to capture that daemon output to a file, so that extra information is available for diagnosing behaviors that warrant a closer look.

Diagnostic Steps

When using an oracledb, orainstance, or oralistener agent, additional output from Oracle processes or from the resource agent itself can be obtained in one of two ways, depending on the release of High Availability packages deployed on the node:

Determining which method to use

The specific method to use depends on the RHEL release and High Availability package releases:

RHEL 6
RHEL 5

Capturing output from the daemon syslog facility to a log file

  1. Add an entry to /etc/rsyslog.conf (RHEL 6) or /etc/syslog.conf (RHEL 5) to capture daemon.* to the desired log file. To send this output to /var/log/messages, simply add daemon.* to the list of facilities going to that file on the line that should already exist:

    *.info;mail.none;authpriv.none;cron.none;daemon.*     /var/log/messages

    Or to log these messages to a separate file, create a new line pointing to the desired file path:

    daemon.*   /var/log/daemon.log

    NOTE: Many other applications use the daemon sysylog facility, so this may capture more than just the desired Oracle resource agent output.

  2. Now restart the rsyslogd daemon (RHEL 6) or syslogd daemon (RHEL 5) to apply the changes after saving the configuration:

    RHEL 6:

    # service rsyslog restart

    RHEL 5:

    # service syslog restart

  3. Now attempt to execute the operation for which extra verbosity is needed, using clusvcadm, Conga, or rg_test

    NOTE: The below rg_test method can still be used on the above mentioned releases, but the output on the command line will not be any more verbose than when starting with rgmanager. In other words, even if rg_test is used to test resources, the above syslog capture method should be implemented if extra verbosity is desired.

Using rg_test to manually execute an agent operation and get verbose output

Like with other resource agents, rg_test can be useful to start a resource and see the output on the command line. However, in those releases noted above for which capturing daemon syslog output is recommended, rg_test will actually not show much more verbose output on the command line than it would when starting through rgmanager with clusvcadm or Conga. However, on older releases, rg_test can still be useful.

For these releases, follow these instructions for using rg_test to perform the desired operation manually and see what it prints out.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

Comments