Recently I was tasked to install Oracle RAC 11.2.0.3 on RHEL 6 (kernel 2.6.32-xx) using HP P4000 as our iSCSI shared storage solution. One would assume that Oracle ASMLib would be available to configure Oracle ASM devices but to my surprise realized that Oracle has now decided to yank support for it in RHEL 6.x releases, BUT offer ASMLib for OEL!
Therein begun my journey into unix system admin udev and multipath utilities to manual configure ASM devices. It took a lot of digging to figure it out and I thought that documenting the steps might make it easier for somebody out there facing the same issue.
HP P4000 has a unique solution that allows us to create volumes across multiple storage servers using Network RAID. I have yet to test out how well it performs in our case since we are sharing the same network switch to interconnect 2 storage servers and the connection to 2 database servers. More on that later…
We created 2 volumes:
1. CRS (OCR and Voting Disk)
2. DATA (datafiles, redologfiles, etc).
If using FRA, then I presume you would want to create a 3rd volume, but that was not needed in my case. Since each DB Server serves as ISCSI “initiator” that needs to connect to shared storage volume, defined as “target”, we found the ISCSI Initiatorname as follows:
# more /etc/iscsi/initiatorname.iscsi InitiatorName=iqn.1994-05.com.redhat:xxxxxxxxxxxx
and input the above when adding Servers in HP P4000 CMC Software.
ISCSI Discovery / Setup
From each DB Server, we need to “discover” the “targets”. Targets are the 2 volumes created on the shared storage device HP P4000 using Network RAID 10.
- Login as root
- Define two paths to reach the target (for redundancy).
# iscsiadm -m iface -I oraeth0 –op=new
New interface oraeth0 added
# iscsiadm -m iface -I oraeth2 –op=new
New interface oraeth2 added
# iscsiadm -m iface -I oraeth0 –op=update -n iface.net_ifacename -v eth0
oraeth0 updated.
# iscsiadm -m iface -I oraeth2 –op=update -n iface.net_ifacename -v eth2
oraeth2 updated.
3. Discover the targets using both interfaces defined above on HP P4000 VIP (10.116.49.88):
# iscsiadm -m discovery -t st -p 10.116.49.88 -I oraeth0 -I oraeth2
10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:285:data
10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs
10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:285:data
10.116.49.88:3260,1 iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs
4. Enable automatic login to targets on reboot (copy iqn…. name from above):
# iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:285:data -p 10.116.49.88 –op update -n node.startup -v automatic
# iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs -p 10.116.49.88 –op update -n node.startup -v automatic
5. Manual Login to targets
#iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs -p 10.116.49.88 -l
#iscsiadm -m node -T iqn.2003-10.com.lefthandnetworks:rac-mg:285:data -p 10.116.49.88 -l
Multipath
Set Alias and multipath policy in /etc/multipath.conf file
- Determine the /dev/sd* device name that is mapped to target.
# ls -l /dev/disk/by-path | grep ip
total 0
lrwxrwxrwx 1 root root 9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs-lun-0 -> ../../sde
lrwxrwxrwx 1 root root 9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:285:data-lun-0 -> ../../sdh
- The above displays sdh and sde as devices that are mapped to CRS and DATA volumes exposed as LUN 0.
- Create partition on sdh device
# fdisk /dev/sdh
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x76993a12.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-52216, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-52216, default 52216):
Using default value 52216
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
- Create partition on sde device
fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xdf7be671.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won’t be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to
switch off the mode (command ‘c’) and change display units to
sectors (command ‘u’).
Command (m for help): n
Command action
e extended
p primary partition (1-4)
1
Invalid partition number for type `1′
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1014, default 1):
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1014, default 1014):
Using default value 1014
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
- Determine the target and uuid of sdh device:
# ls -l /dev/disk/by-path |grep sdh
lrwxrwxrwx 1 root root 9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:285:data-lun-0 -> ../../sdh
# scsi_id –whitelisted /dev/sdh
36000eb3262f406b2000000000000011d
Note target data uuid is 36000eb3262f406b2000000000000011d
- Determine the target and uuid of sde device:
# ls -l /dev/disk/by-path |grep sde
lrwxrwxrwx 1 root root 9 Jul 19 20:43 ip-10.116.49.88:3260-iscsi-iqn.2003-10.com.lefthandnetworks:rac-mg:283:crs-lun-0 -> ../../sde
# scsi_id –whitelisted /dev/sde
36000eb3262f406b2000000000000011b
Note target crs uuid is 36000eb3262f406b2000000000000011b
- Enter the above values into /etc/multipath and give friendly aliases to uuid.
vi /etc/multipathd.conf
Uncomment and modify as follows:
blacklist {
devnode “^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*”
devnode “^hd[a-z]”
}
multipaths {
multipath {
wwid 36000eb3262f406b2000000000000011b
alias crs
path_grouping_policy multibus
path_selector “round-robin 0”
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 36000eb3262f406b2000000000000011d
alias data
path_grouping_policy multibus
path_selector “round-robin 0”
failback immediate
rr_weight priorities
no_path_retry 5
}
}
- Restart multipathd service.
service multipathd restart
- Check if multipath is setup properly:
# multipath -ll
crs (36000eb3262f406b2000000000000011b) dm-2 LEFTHAND,iSCSIDisk
size=3.0G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 66:0:0:0 sdc 8:32 active ready running
|- 65:0:0:0 sdb 8:16 active ready running
|- 67:0:0:0 sdd 8:48 active ready running
`- 68:0:0:0 sde 8:64 active ready running
data (36000eb3262f406b2000000000000011d) dm-3 LEFTHAND,iSCSIDisk
size=400G features=’1 queue_if_no_path’ hwhandler=’0′ wp=rw
`-+- policy=’round-robin 0′ prio=1 status=active
|- 70:0:0:0 sdf 8:80 active ready running
|- 71:0:0:0 sdg 8:96 active ready running
|- 72:0:0:0 sdh 8:112 active ready running
`- 69:0:0:0 sdi 8:128 active ready running
- Determine partitioned alias for target device
# dmsetup ls |grep p1
datap1 (253, 5)
crsp1 (253, 4)
- Edit /etc/udev/rules.d/12-dm-permissions.rules file to set the above device ownership as oracle:dba
# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
ENV{DM_NAME}==”crsp1″, OWNER:=”oracle”, GROUP:=”dba”, MODE:=”660″, SYMLINK+=”iscsi/oraasm-$env{DM_NAME}”
ENV{DM_NAME}==”datap1″, SYMLINK+=”iscsi/oraasm-$env{DM_NAME}”, OWNER:=”oracle”, GROUP:=”dba”, MODE:=”660″
- Restart multipathd service.
service multipathd restart
- Check if permissions are correct
# ls -l /dev/dm*
brw-rw—- 1 root disk 253, 0 Jul 16 19:40 /dev/dm-0
brw-rw—- 1 root disk 253, 1 Jul 16 19:40 /dev/dm-1
brw-rw—- 1 root disk 253, 2 Jul 19 21:47 /dev/dm-2
brw-rw—- 1 root disk 253, 3 Jul 19 21:47 /dev/dm-3
brw-rw—- 1 oracle dba 253, 4 Jul 19 21:47 /dev/dm-4
brw-rw—- 1 oracle dba 253, 5 Jul 19 21:47 /dev/dm-5
Note: Multipath /dev/dm-4 and /dev/dm-5 devices are owned by oracle:dba, which point to CRS and DATA partitioned volumes.
- Check if oracle ASM symbolic links exists for multipath devices.
# ls -l /dev/iscsi/*
lrwxrwxrwx 1 root root 7 Jul 19 21:47 /dev/iscsi/oraasm-crsp1 -> ../dm-4
lrwxrwxrwx 1 root root 7 Jul 19 21:47 /dev/iscsi/oraasm-datap1 -> ../dm-5
murtazahabib 