Solaris 10 + IPMP + Oracle RAC

I recently installed a 2-node RAC cluster using the following configuration:

Operating System: Solaris 10 (SPARC-64)
Oracle Clusterware: 11.1.0.6
Oracle ASM: 11.1.0.6
Oracle RDBMS: 10.2.0.3

Because the servers had 4 network interface cards, I asked the system administrators to configure IPMP on the Virtual IP and Private Interconnect interfaces.

The Clusterware, ASM, and RDBMS installations went as planned. However, when we tried restarting the ASM instance, it would take several minutes before coming up. While it started, I ran a ptree on the racgimon process and found that it was hanging on the “sh -c /usr/sbin/arp -a | /usr/xpg4/bin/grep SP” command. It took awhile to sort out, but I was finally able to put together enough blog posts and Metalink notes to figure out what needed to be done.

  1. Collect the hostname, VIP, and private interconnect aliases and IP addresses for each RAC node from /etc/hosts.
  2. Collect network interface information on each node, identifying which interfaces are part of each IPMP group.
    ifconfig -a
  3. Identify which interfaces nodeapps is using on each node.
    srvctl config nodeapps -n <hostname>
  4. Update the nodeapps interfaces as necessary.
    srvctl modify nodeapps -n <hostname> -A <ip_ddress>/<subnet_mask>/<ipmp_interface1>\|<ipmp_interface2>
  5. Identify the OCR private interface(s).
    oifcfg getif
  6. Delete the OCR private interface(s).
    oifcfg delif -global <if_name>
  7. Set the CLUSTER_INTERCONNECTS parameter in each ASM and database instance pfile/spile.
    CLUSTER_INTERCONNECTS='ip_address'
    or
    alter system set cluster_interconnects='ip_address' scope=spfile sid='SID1';
  8. Restart all services on each node.
  9. Verify that each database and ASM instance is using the appropriate Private Interconnect.
    select * from gv$cluster_interconnects;

The ASM startup will now take a fraction of the time it was taking before, and the correct interconnect IP address will be used.

References
Metalink Note 283107.1 – Configuring Solaris IP Multipathing (IPMP) for the Oracle 10g VIP
Metalink Note 368464.1 – How to Setup IPMP as Cluster Interconnect

One thought on “Solaris 10 + IPMP + Oracle RAC

  1. KARTIK VASHISHTA

    Thanks for this useful information regarding use of IPMP in cluster interconnects. IPMP helps minimize errant failovers, and, helps ensure redundancy. This isb certainly at no additional charge(IPMP comes with Solaris 10).

    Do you have notes of oracle installation on a cluster…I intend to learn this and any help/tips will not be unappreciated.

Leave a Reply

Your email address will not be published. Required fields are marked *