Installing 11gR2 Real Application Clusters on Oracle Enterprise Linux 4 x86-64 (64 bit)

Installing Oracle Grid Infrastrcture
This document provides a step by step guide to, installing Oracle 11gR2 Real Application clusters on Oracle Enterprise Linux 4, x86-64 (64 Bit).
Keep in mind, this is a test installation and hence does not follow some of the best practices for installing oracle real application clusters (eg: use a separate o/s user for oracle grid infrastructure, have at least 3 voting disks etc…).

Important Documents

Grid infrastructure installation guide for Linux
Real Application Clusters Installation guide for Linux and Unix
Clusterware Administration and Deployment Guide
Real Application Clusters Administration and Deployment Guide

Some new concepts in 11gR2 Rac

Oracle clusterware and ASM now are installed into the Same Oracle Home, and is now called the grid infrastructure install.
Raw devices are no longer supported for use for anything (Read oracle cluster registry, voting disk, asm disks), for new installs.
OCR and Voting disk can now be stored in ASM, or a certified cluster file system.
The redundancy level of your ASM diskgroup (That you choose to place voting disk on) determines the number of voting disks you can have.
You can place

Only One voting disk on an ASM diskgroup configured as external redundancy
Only Three voting disks on an ASM diskgroup configured as normal redundancy
Only Five voting disks on an ASM diskgroup configured as high redundancy

The contents of the voting disks are automatically backed up into the OCR
ACFS (Asm cluster file system) is only supported on Oracle Enterprise Linux 5 (And RHEL5), not on OEL4.
There is a new service called cluster time synchronization service that can keep the clocks on all the servers in the cluster synchronized (In case you dont have network time protocol (ntp) configured)
Single Client Access Name (SCAN), is a hostname in the DNS server that will resolve to 3 (or at least one) ip addresses in your public network. This hostname is to be used by client applications to connect to the database (As opposed to the vip hostnames you were using in 10g and 11gr1). SCAN provides location independence to the client connections connecting to the database. SCAN makes node additions and removals transparent to the client application (meaning you dont have to edit your tnsnames.ora entries every time you add or remove a node from the cluster).
Oracle Grid Naming Service (GNS), provides a mechanism to make the allocation and removal of VIP addresses a dynamic process (Using dynamic Ip addresses).
Intelligent Platform Management Interface (IPMI) integration, provides a new mechanism to fence server’s in the cluster, when the server is not responding.
The installer can now check the O/S requirements, report on the requirements that are not met, and give you fixup scripts to fix some of them (like setting kernel parameters).
The installer can also help you setup SSH between the cluster nodes.
There is anew deinstall utility that cleans up a existing or failed install.
There is a new Instantaneous problem detection OS tool. This tool is the nextgen oswatcher.
And the list goes on and on.
Listed below are some of the top hardware and software requirements for the install. Please refer to the grid infrastructure install guide for all the pre-requisites.

Hardware requirements

Please refer to the install guide for all the pre-requisites.
One or more servers
Shared Disk storage (SAN, NAS)
GigE or higher network switch (For the private interconnect)
Atleast One GigE or higher Network interface card for the private interconnect
Atleast One Network interface card for the public interconnect
IP address requirements

One SCAN name (That resolves to 3 ip addresses) for the cluster
For each Node

1 public IP address
1 private IP address
1 VIP IP address

The SCAN, Public IP addresses and the VIP should be in the same subnet.

Operating System requirements

Amoung other requirements (Please see the install documents for all the requirements)
RHEL4 (or OEL4), update 7 or higher
RHEL5 (or OEL5), update 2 or higher

Software requirements list for x86-64 linux platforms
Kernel parameters
If you are part of the Redhat Linux Network, or Oracle Unbreakable linux network, then you can get the oracle-validated rpm. This rpm sets up all the (or most of it in my case) required rpm’s and kernel parameters for you. You can use up2date to install this package. This really simplifies the setup steps that you need to perform.
Setup SSH between the cluster nodes.
Setup ntp deamon for clock synchronization on all the nodes. Remember to use the -x switch, or else cluvfy will always call this out and declare failure.
Configure ASMLIB

Install the asm packages oracleasm-support oracleasmlib oracleasm
Run oracleasm configure on each node to configure asmlib
Run oracleasm createdisk to create the needed ASM disks from one node
Run oracleasm scandisks to mount the disks on all the other nodes.

Download Software for Installation

Download Oracle Grid Infrastructure Software
Download Oracle Database Software – Part 1
Download Oracle Database Software – Part 2

Grid Infrastructure Installation

Unzip the grid infrastructure software into a staging directory on one of the server’s.
login as the oracle user.
cd <software stage home>/grid
./runInstaller

In the screen above, choose to “Install and configure grid infrastructure for a cluster”, click Next.

Choose a “Typical” installation, Click Next.

The above screen will be displayed with a default scan name, and just the details regarding the node from which you are running the installer.
Click on the Add button and add the hostname and virtual ip name for the rest of the nodes in the cluster.
The SCAN Name has to be a valid hostname that is setup in DNS which resolves to 3 ip addresses (atleast 1) in a round robin fashion.
Replace the default shown in the screen above with a valid SCAN you will be using.

If you have not configured ssh connectivity, you can click on “SSH connectivity” and configure ssh.

You can click on “Identify network interfaces” to check and/or change, the interfaces being used for the public and private interfaces.

Click Ok to return to the Cluster Configuration screen.
Click Next

In the screen above, choose the location for Oracle Base, the Home directory for the Grid infrastructure software, the passwords for the ASM instance and the group to be used as the OSASM group. Click Next to continue.

In the screen above, choose the disk group name to use for the first ASM diskgroup created, and which disks to use. The ASM instance needs to be created so that the voting disk and cluster registry can be placed on ASM. In my case i had configured the disks using ASMLIB before i started the grid infrastructure install. In my case the disks are created as Raid 10 on the array, so i chose external redundancy. This means that there will be only one voting disk created. Click Next to continue.

In the “Prerequisite checks” screen above, the installer runs through and checks all the pre-requisites (kernel parameters, ssh, ntp, packages). It displays the pre-requisites that are not being met. Some of these it is able to help us fix (like kernel parameters). For the one’s it can fix it will give you a script to run from /tmp as root.

In my case although i had used the oracle-validated package, there were some packages missing (i386 versions of lib-aio, unixODBC and unixODBC-devel), and some kernel parameters to be fixed.

I fixed them and ran the Check again, which resulted in the above two remaining. Since I know that i have enough swap space and I had not used the -x flag for the ntp setup, i choose to Ignore All and continue.

In the above screen, click finish to continue with the install.

Finally it will prompt you with the screen above to run the scripts as root.
Make sure that you run them one at a time, one node at a time.
Once the scripts are run to completion successfully, click on the Ok button.