Installation Notes on Install Oracle RAC with Openfiler on ESXi 5.5
Oracle provides a very good guide ("Build Your Own Oracle RAC 11g Cluster on Oracle Linux and iSCSI") to install 2 Oracle 11g R2 RAC nodes with a Openfiler 2.3 as shared storage. The guide is based on using cheap hardware to setup a lab environment.
Using VMware provides a even lower cost to setup a lab. Since VMware allows VM cloning, I can saved my time to setup RAC node 2 by copying node 1. The below document is my log on setup the environment.
Follow all steps in the instruction until step 15.
There is a bug on the /home/grid/.bash_profile provided. Edit the below lines:
There is a bug on the /home/oracle/.bash_profile provided. Edit the below lines:
Skip step 16.
In order to pass runcluvfy.sh, we need to add "dba" as supplementary group of user "grid".
In order to pass runcluvfy.sh, we need to enable ntpd.
Follow all steps in the instruction until step 18 ( Install and Configure ASMLib 2.0)
Shutdown racnode1 and clone racnode1 to racnod2
Edit line 29 of /home/grid/.bash_profile
Edit line 28 of /home/oracle/.bash_profile
Using Xming to login to remote system
The above command allows Xming to accept connection from any IP.
Generate keys for SSH passwordless login
For the DB creation
If you failed to properly setup the 3 DNS A record for the cluster hostname (racnode-cluster-scan) during OUI installation, you will end up with only one scan listener. To add back the other two Scan Listener, we can follow the instruction here.
- Follow the Oracle's instruction, except the setup of node 2 (we will clone Node 1 later).
- Create a VM, using 3GB RAM, 40GB harddisk (thin provision)
- Install Oracle Linux and all the stuff
- Changed to VMX LAN card and install VMWare Tools.
- Reconfigure all LAN because of change LAN card (need to reboot to check if the new LAN card is "activate" automatically during boot up. Otherwise, please configure the LAN card at system-config.network.
- For the Openfiler
- 1GB RAM, 4GB harddisk for the system
- I have create another 20GB virtual harddisk for data.
- Configure all iSCSI stuff per instruction
- if you cannot list the iSCSI target at racnodes, try these method
- Openfiler won’t accept connections; check /etc/initiators.deny for an entry denying access from ALL. Just comment it out and save the file. Take care: Openfiler will regenerate this file from time to time… (reference)
- use this method 2.
- The openfiler shall returns only 3 SCSI targets by command iscsiadm -m discovery -t st -p 192.168.2.x. If iscsi added 6 devices, add the following line at /etc/initiators.deny
- ALL 192.168.1.0/24
- Remark line 24 and line 27
- Remark line 23 and line 26
- chkconfig ntpd on
- edit /etc/sysconfig/ntpd. Add option "-x " before "-u". Details at here.
- service ntpd start
- Clone racnode1 to racnode2. You may use VMware Converter Standalone 5.5 to clone. It supports ESXi to ESXi VM clone.
- We need to modify the below items for the clone
- Hostname and IP addresses (via system-config-network inside GUI)
- Hostname and IP addresses (via system-config-network inside GUI
- Edit IP addresses of both network interface
- Delete old NIC profile
- Update the hostname
- Make sure the ethernet NICs use the same name (i.e. eth0 and eth1) across both nodes.
- A new unique name for the iscsi initaitor (the iSCSI client). Edit the file:
- nano /etc/iscsi/initiatorname.iscsi
- Change +ASM1 to +ASM2
- Change racdb1 to racdb2
- In my case, I receives "Xlib: connection to "[my_windows_ip]:0.0" refused by server" error when try to run xterm.
- For the connection to be successful, start Xming with the below command:
- "C:\Program Files\Xming\Xming.exe" :0 -clipboard -multiwindow -ac
- At racnode2, delete the ~/.ssh
- re-run /usr/bin/ssh-keygen -t dsa
- before DB creation, we need to reset ASMSNMP password
- su - grid
- sqlplus / as sysadm
- alter user asmsnmp identified by password;