Device LUNs Name Changing Every OS reboot

Latest response

Please find what we found on the Oracle RAC installation.
• Every time we reboot the server the device name it keep changing
o example: /dev/sda present device name assignment after reboot /dev/sddg
• And this server is connected to EMC storage using power path v6.0
* we are using FCOE interface for SAN connectivity

For this issue we can put the rdloaddriver=hpsa,lpfc as a kernel parameter, so the local hard drive will get the first assignment.

Please we need an advice if this solution is sufficient or correct to resolve this issue without data corruption."

Responses

Device names are not guaranteed to be persistent. You can address filesystems in /etc/fstab by Label or UUID, you can use LVM to have device-mapper name devices consistently, and PowerPath should be naming your SAN devices consistently.

We use the UUIDs on our SAN-attached storage, and that's a real saver. When our network guys decide they need to change my storage to a new switch/port, it would change the device. We have had no problems ever since we began using the UUID for the mount in /etc/fstab.

I have not worked with PowerPath (on Linux) - but, this may still be relevant. If powerpath utilizes the /etc/lvm.conf, you can customize (or optimize) for your specific array and have a script to name the devices whatever you want.

getuid_callout      "/usr/local/sbin/myscsi_id.sh %n"

In my environment, the WWN gets sent to a perl script which generates output based on a dictionary which maps an HDS serial number to a human-readable name and the LUN number.
360060e800571c900000071c9000005f7
71c9 = HDS01
70a7 = HDS02
In this case, the WWN multipath -ll -v2 would return
HDS01-05f7 dm-5 HITACHI,OPEN-V

EMC's PowerPath should be maintaining a persistent mapping between the /dev/emcpowera (powerb, powerc, etc.) pseudo-devices and their corresponding back-end LUNs. So on one boot, "/dev/emcpowera" maps to /dev/sdg (and /dev/sdh, and maybe more if you have more paths), which is really "LUN 36" on the array; the next boot, LUN 36 might end up on /dev/sdx and /dev/sdy, but "/dev/emcpowera" should still point to it. Your Oracle ASM config should be set to only use the /dev/emcpower* devices, and then it will get consistent device mappings.

Our Oracle servers (RHEL 5.x + PowerPath 5.x + Oracle 11gR2 on ASM) have been rebooted probably a dozen times over the 3 years they've been in production, haven't had a problem with device/LUN mapping yet.

James,
Is the following an example of how to modify /etc/sysconfig/oracleasm?

ORACLEASM_SCANEXCLUDE=sd
ORACLEASM_SCANORDER=emcpower

http://www.oracle.com/technetwork/topics/linux/multipath-097959.html

EDIT: changed my phrasing of my question and provided a link to supporting doc.

As mentioned above, UUID/LABEL are reliable way to mount them in fstab. You can use DM-multipath and have unique names for each of those devices. multipath device name do not change across reboot.

James Nauer addressed this in a more subtle way, but I will try to respectfully be more blunt - you are doing it wrong.

The /dev/sd* device names are expected to change. When using a multi-pathing solution such as EMC's PowerPath or RHEL's native device-mapper-multipath, you should NEVER attempt to use the /dev/sd* device names. You can cause data corruption by doing so, as well as removing the redundancy of having multiple paths to the device for reliability and performance.

These multi-path solutions provide logical device names to the LUNs to provide 1) persistant device naming and 2) redundant access paths to the physical LUN. EMC PowerPath will provide logical device names that look like /dev/emcpowera which might point to example devices such as /dev/sdaa, /dev/sdab, /dev/sdac & /dev/sdad. There when when you read/write to /dev/emcpowera, that operation has 4 possible paths it could take. The number of devices depends upon the configuration of the backend SAN network, array and number of HBA interfaces.

In the case of RHEL's native device-mapper-multipath, the logical device will look something like /dev/mapper/mpatha. Both EMC PowerPath and Native Multi-Path perform equally well, though you will hear all kinds of bias and debates for each. It's all hogwash, they both perform very well and we have lots of customers using both very successfully. One advantage of Native Multi-Path is that you don't have to worry about updating to match the kernel. The "yum update kernel device-mapper-multipath" command will update both automatically.

Have you consulted our Oracle Recommend Practices Guides? I highly recommend them to properly tune your Oracle DB/RAC environment.

2014 - Deploying Oracle Database 12c on Red Hat Enterprise Linux 7- Recommended Practices
https://access.redhat.com/articles/1282303

2014 - Deploying Oracle Database 12c on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/725843

2013 - Deploying Oracle Database 11g R2 on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/395013

2013 - Deploying Oracle RAC 11g R2 Database on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/479093

Areas to focus on would include:

  • use latest 6.6 errata (scheduler and THP improvements)
  • multipath configuration
    *for performance use:

    • irqbalance (on by default)
    • numad
    • tuned ("tune-adm profile enterprise-storage")
    • configure hugepage sizes using the script in the guides

Having fun tuning your environment!!!

In addition to my previous comment, you can also use native udev rules in stead of the asmlib kernel module to provide persistant naming, owner, permissions to the devices to present to Oracle ASM. Again, there is no performance benefit either way, contrary to the bias you might read.

Native udev rules simply put a little more burden on the sysadmin to update the udev rules with the WWID any time a LUN is added/removed, but the benefit is again not having to deal with a third party kernel module update each time you update the kernel. They seem scary at first, but once you've done one or two, they are pretty easy. The guides above cover this process very well.

Primary downside to udev is that, once you get used to using them, when you can't use them (e.g., you move to a virtualized solution where your hypervisor doesn't know how to do UDIDs), you're kind of left scratching your head on the going back to the other options.

I will absolutely endorse Terry's recommendation of reviewing and following the best-practices doc. I have yet to work in a RAC environment that did not present some challenges along the way. It is best to follow a supported implementation plan if you will need to rely on the vendors for support when those challenges do arise.

The following is a video that demonstrates, basically in real-time, the advantages that can be realized with using tuned
https://access.redhat.com/videos/898563

1. Set up multipath like below

defaults {
    find_multipaths yes
    user_friendly_names yes
}

# make your luns to human readable name
multipaths {
  multipath {
      wwid          360002ac000000000000000020000f8cc
        alias           lun1g001
    }
  multipath {
      wwid          360002ac000000000000000030000f8cc
        alias           lun1g002
    }

2. make your luns to raw device
/etc/udev/rules.d/60-raw.rules

ACTION=="add|change", ENV{DM_NAME}=="lun1g001", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add|change", ENV{DM_NAME}=="lun1g002", RUN+="/bin/raw /dev/raw/raw2 %N"

3. optionally set devices permission.
/etc/udev/rules.d/12-dm-permission.rules

ENV{DM_UUID}=="mpath-?*", OWNER:="oracle", GROUP:="dba", MODE:="660"

4. Use your raw device to oracle rac raw device.

We where having pretty much same issue. We did converted server from P to V and attached raw disks from san to Virtual machines and decided to move from raw disks to vmdk's. And this is where all our issues got started. So what we did to resolve this issue is we created udev rules in the system. I had a discussion here where some experts helped me to create a rules. As soon as I will find the thread I will post it here.

Here it is : https://access.redhat.com/discussions/876913

Close

Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.