ksvcreate: Process(m000) creation failed. Error in Alert log due to DNS ip change.

Today on one of our 10g (10.2.0.1, don’t ask !) production databases  running on AIX 5.3 we saw the following message in the alert log.

ksvcreate: Process(m000) creation failed

The database slowed down to a crawl. We couldn't login even SQLPLUS, even sqlplus -prelim couldn't bring up the instance. On starting the instance we got the below error.

ORA-00445: background process "PMON" did not start after 120 seconds

After googling around, we figured out it could be related to a DNS change. Our old DNS was commissioned and the new ones were not added to the resolv.conf file. After adding the new name servers. The instance started immediately without any issues. I'm still wondering what is the relation between the Oracle instance startup and the DNS ip's. Could this be a bug specific to 10.2.0.1 or is it something more generic.
Advertisements

Join AIX to DNS

# vi /etc/resolv.conf

— If file doesn’t exist, create it

Add the below entries into resolv.conf file of your DNS server and domain name

nameserver 127.0.0.1
domain yourdomainname.com

# nslookup nameofyourserver

Change SCAN ip’s Oracle RAC11gR2

SCAN (Single Client Access Name) is a new concept introduced in since 11gR2. It eliminated to have VIP )Virutal ip) of nodes in the tnsnames.ora file. Prior to release 11gR2 the TNSNAMES.ora file required to have all nodes entered in it for load balancing and failover. This created unnecessarily changes everytime a node was added or deleted. With 11gR2 Oracle created a listener runnin on top of the node listeners. This listener needs 3 ips to be configured on the DNS. And it creates a virtual hostname which can be used as a signle entry in the tnsnames. Its a very useful feature. But sometimes if you change the scan ips in your organization. The SCAN ip’s first need to be changed on your DNS by your network administrator, then the clusterware needs to be told of the changes. Heres how you do it.

Steps:

1. Check the current status SCN IP address in the DNS server.

$nslookup <scan-name>

$nslookup testrac-scan.abc.com

Server:         160.34.11.20
Address:        160.34.11.21#40

Name:   testrac-scan.abc.com
Address: 160.34.11.88
Name:   testrac-scan.abc.com
Address: 160.34.11.89
Name:   testrac-scan.abc.com
Address: 160.34.11.90

2. Check the current status SCAN-VIP in the resource file

# $GRID_HOME/bin/srvctl config scan
SCAN name: testrac-scan, Network: 1/10.101.10.0/255.255.255.0/eth4
SCAN VIP name: scan1, IP: /testrac-scan.abc.com/160.34.11.88
SCAN VIP name: scan2, IP: /testrac-scan.abc.com/160.34.11.89
SCAN VIP name: scan3, IP: /testrac-scan.abc.com/160.34.11.90

3. Request network admin to update NEW SCAN IP address in the DNS server.

Example:

Old SCAN IP:

160.34.11.88
160.34.11.89
160.34.11.90

new SCAN IP:

170.35.12.60
170.35.12.61
170.35.12.62

$nslookup testrac-scan.abc.com

Server:         170.35.12.20

Address:        170.35.12.21#40

Name:   testrac-scan.abc.com
Address: 170.35.12.60
Name:   testrac-scan.abc.com
Address: 170.35.12.61
Name:   testrac-scan.abc.com
Address: 170.35.12.62

4.CRS to update the SCAN VIP resources:

Note: current SCAN-VIP resource should be stopped before modifying the CRS resoruce file.

# $GRID_HOME/bin/srvctl stop scan_listener
# $GRID_HOME/bin/srvctl stop scan
# $GRID_HOME/bin/srvctl status scan
SCAN VIP scan1 is enabled
SCAN VIP scan1 is not running
SCAN VIP scan2 is enabled
SCAN VIP scan2 is not running
SCAN VIP scan3 is enabled
SCAN VIP scan3 is not running
# $GRID_HOME/bin/srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is not running
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is not running

# $GRID_HOME/bin/srvctl modify scan -n testrac-scan.abc.com

5. To verify that the change was successful

# $GRID_HOME/bin/srvctl config scan
SCAN name: sales-scan, Network: 1/1/10.101.10.0/255.255.255.0/eth4
SCAN VIP name: scan1, IP: /testrac-scan.abc.com/170.35.12.60
SCAN VIP name: scan2, IP: /testrac-scan.abc.com/170.35.12.61
SCAN VIP name: scan3, IP: /testrac-scan.abc.com/170.35.12.62

6. Start SCAN and the SCAN listener

# $GRID_HOME/bin/srvctl start scan
# $GRID_HOME/bin/srvctl start scan_listener