Device LUNs Name Changing Every OS reboot
Please find what we found on the Oracle RAC installation.
• Every time we reboot the server the device name it keep changing
o example: /dev/sda present device name assignment after reboot /dev/sddg
• And this server is connected to EMC storage using power path v6.0
* we are using FCOE interface for SAN connectivity
For this issue we can put the rdloaddriver=hpsa,lpfc as a kernel parameter, so the local hard drive will get the first assignment.
Please we need an advice if this solution is sufficient or correct to resolve this issue without data corruption."
Responses
I have not worked with PowerPath (on Linux) - but, this may still be relevant. If powerpath utilizes the /etc/lvm.conf, you can customize (or optimize) for your specific array and have a script to name the devices whatever you want.
getuid_callout "/usr/local/sbin/myscsi_id.sh %n"
In my environment, the WWN gets sent to a perl script which generates output based on a dictionary which maps an HDS serial number to a human-readable name and the LUN number.
360060e800571c900000071c9000005f7
71c9 = HDS01
70a7 = HDS02
In this case, the WWN multipath -ll -v2 would return
HDS01-05f7 dm-5 HITACHI,OPEN-V
EMC's PowerPath should be maintaining a persistent mapping between the /dev/emcpowera (powerb, powerc, etc.) pseudo-devices and their corresponding back-end LUNs. So on one boot, "/dev/emcpowera" maps to /dev/sdg (and /dev/sdh, and maybe more if you have more paths), which is really "LUN 36" on the array; the next boot, LUN 36 might end up on /dev/sdx and /dev/sdy, but "/dev/emcpowera" should still point to it. Your Oracle ASM config should be set to only use the /dev/emcpower* devices, and then it will get consistent device mappings.
Our Oracle servers (RHEL 5.x + PowerPath 5.x + Oracle 11gR2 on ASM) have been rebooted probably a dozen times over the 3 years they've been in production, haven't had a problem with device/LUN mapping yet.
As mentioned above, UUID/LABEL are reliable way to mount them in fstab. You can use DM-multipath and have unique names for each of those devices. multipath device name do not change across reboot.
James Nauer addressed this in a more subtle way, but I will try to respectfully be more blunt - you are doing it wrong.
The /dev/sd* device names are expected to change. When using a multi-pathing solution such as EMC's PowerPath or RHEL's native device-mapper-multipath, you should NEVER attempt to use the /dev/sd* device names. You can cause data corruption by doing so, as well as removing the redundancy of having multiple paths to the device for reliability and performance.
These multi-path solutions provide logical device names to the LUNs to provide 1) persistant device naming and 2) redundant access paths to the physical LUN. EMC PowerPath will provide logical device names that look like /dev/emcpowera which might point to example devices such as /dev/sdaa, /dev/sdab, /dev/sdac & /dev/sdad. There when when you read/write to /dev/emcpowera, that operation has 4 possible paths it could take. The number of devices depends upon the configuration of the backend SAN network, array and number of HBA interfaces.
In the case of RHEL's native device-mapper-multipath, the logical device will look something like /dev/mapper/mpatha. Both EMC PowerPath and Native Multi-Path perform equally well, though you will hear all kinds of bias and debates for each. It's all hogwash, they both perform very well and we have lots of customers using both very successfully. One advantage of Native Multi-Path is that you don't have to worry about updating to match the kernel. The "yum update kernel device-mapper-multipath" command will update both automatically.
Have you consulted our Oracle Recommend Practices Guides? I highly recommend them to properly tune your Oracle DB/RAC environment.
2014 - Deploying Oracle Database 12c on Red Hat Enterprise Linux 7- Recommended Practices
https://access.redhat.com/articles/12823032014 - Deploying Oracle Database 12c on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/7258432013 - Deploying Oracle Database 11g R2 on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/3950132013 - Deploying Oracle RAC 11g R2 Database on Red Hat Enterprise Linux 6 - Recommended Practices
https://access.redhat.com/articles/479093
Areas to focus on would include:
- use latest 6.6 errata (scheduler and THP improvements)
- multipath configuration
*for performance use:- irqbalance (on by default)
- numad
- tuned ("tune-adm profile enterprise-storage")
- configure hugepage sizes using the script in the guides
Having fun tuning your environment!!!
In addition to my previous comment, you can also use native udev rules in stead of the asmlib kernel module to provide persistant naming, owner, permissions to the devices to present to Oracle ASM. Again, there is no performance benefit either way, contrary to the bias you might read.
Native udev rules simply put a little more burden on the sysadmin to update the udev rules with the WWID any time a LUN is added/removed, but the benefit is again not having to deal with a third party kernel module update each time you update the kernel. They seem scary at first, but once you've done one or two, they are pretty easy. The guides above cover this process very well.
I will absolutely endorse Terry's recommendation of reviewing and following the best-practices doc. I have yet to work in a RAC environment that did not present some challenges along the way. It is best to follow a supported implementation plan if you will need to rely on the vendors for support when those challenges do arise.
The following is a video that demonstrates, basically in real-time, the advantages that can be realized with using tuned
https://access.redhat.com/videos/898563
1. Set up multipath like below
defaults {
find_multipaths yes
user_friendly_names yes
}
# make your luns to human readable name
multipaths {
multipath {
wwid 360002ac000000000000000020000f8cc
alias lun1g001
}
multipath {
wwid 360002ac000000000000000030000f8cc
alias lun1g002
}
2. make your luns to raw device
/etc/udev/rules.d/60-raw.rules
ACTION=="add|change", ENV{DM_NAME}=="lun1g001", RUN+="/bin/raw /dev/raw/raw1 %N"
ACTION=="add|change", ENV{DM_NAME}=="lun1g002", RUN+="/bin/raw /dev/raw/raw2 %N"
3. optionally set devices permission.
/etc/udev/rules.d/12-dm-permission.rules
ENV{DM_UUID}=="mpath-?*", OWNER:="oracle", GROUP:="dba", MODE:="660"
4. Use your raw device to oracle rac raw device.
We where having pretty much same issue. We did converted server from P to V and attached raw disks from san to Virtual machines and decided to move from raw disks to vmdk's. And this is where all our issues got started. So what we did to resolve this issue is we created udev rules in the system. I had a discussion here where some experts helped me to create a rules. As soon as I will find the thread I will post it here.
Welcome! Check out the Getting Started with Red Hat page for quick tours and guides for common tasks.
