Oracle 12c Dataguard RAC Primary to RAC Standby with Dataguard Broker (on Exadata)

Setup for the Environment
——————-

Exadata X4-2 Quater Rack n Primary and Standby Site
Grid Version : 12.1.0.2
DB Home Version : 12.1.0.2

Primary, 2 Node RAC, db_unique_name = primeprd
Instance 2 : primeprd1
Instance 2 : primeprd2
TNS entry: PRIMEPRD, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEPRD =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = primaryhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd)
)
)

Standby, 2 Node RAC, db_unique_name= primedr
Instance 2 : primedr1
Instance 2 : primedr2
TNS ENTRY: PRIMEDR, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEDR =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = standbyhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr)
)
)

1. Enable Force Logging on the Database

 alter database force logging;

Database altered.

2.1 change the below parameter in Primary database.


ALTER SYSTEM SET db_recovery_file_dest='+RECOC1' scope=both sid='*';

System altered.

alter system set LOG_ARCHIVE_CONFIG='DG_CONFIG=(primeprd,primedr)' scope=both sid='*';

alter system set LOG_ARCHIVE_DEST_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=primeprd' scope=both sid='*';

alter system set LOG_ARCHIVE_DEST_2='SERVICE=primedr LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=primedr' scope=both sid='*';

alter system set log_archive_format='%t_%s_%r.arc' scope=spfile sid='*';

alter system set log_archive_max_processes=8 scope=both sid='*';

alter system set fal_server=primedr scope=both sid='*';

alter system set fal_client=primeprd scope=both sid='*';

alter system set standby_file_management=AUTO scope=both sid='*';

alter system set db_file_name_convert=('+DATAC1/primedr/', '+DATAC1/primeprd/', '+RECOC1/primedr/', '+RECOC1/primeprd/') scope=both sid='*';

alter system set log_file_name_convert=('+DATAC1/primedr/', '+DATAC1/primeprd/', '+RECOC1/primedr/', '+RECOC1/primeprd/') scope=both sid='*';


-- Query to check if parameters are set correctly --

set linesize 500 pages 0
col value for a90
col name for a50
select name, value
from v$parameter
where name in ('db_name','db_unique_name','log_archive_config', 'log_archive_dest_1','log_archive_dest_2',
'log_archive_dest_state_1','log_archive_dest_state_2', 'remote_login_passwordfile',
'log_archive_format','log_archive_max_processes','fal_server','db_file_name_convert',
'log_file_name_convert', 'standby_file_management');

select name, value
from v$parameter
where name in ('db_name','db_unique_name','log_archive_config', 'log_archive_dest_1','log_archive_dest_2',
'log_archive_dest_state_1','log_archive_dest_state_2', 'remote_login_passwordfile',
'log_archive_format','log_archive_max_processes','fal_server','db_file_name_convert',
'log_file_name_convert', 'standby_file_management');

3.1. verify the logfile using the below querys.

select group#,THREAD#,bytes/1024/1024,status,members from v$log;
select member from v$logfile;

3.2. Add the below logfile and standby logfile in primary database, make sure you have large redolog files. In this day and age, 100MB does not really cut it.

ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 1 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 2 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 4 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 5 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 6 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 7 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 8 ('+DATAC1','+RECOC1') SIZE 1G;

 select * from v$standby_log;

-- Add Standby Redolog files, number of standby redo log files = Redo logfile Groups + 2 --

ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 9 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 10 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 11 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 12 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 13 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 14 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 15 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 16 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 17 ('+DATAC1','+RECOC1') SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 18 ('+DATAC1','+RECOC1') SIZE 1G;

4. Take the rman backup using below command.

rman target / nocatalog
run
{
sql 'alter system archive log current';
backup as compressed backupset database plus archivelog format '/u03/rmanbkp/online/Primary_bkp_for_stndby_%U';
backup current controlfile for standby format '/u03/rmanbkp/online/stby.ctl';
sql 'alter system archive log current';
}

5.1. create the init file using the below command in primary database.

create pfile='/u03/rmanbkp/prime/pfile_for_standby.txt' from spfile;

— Primary PFILE —

primeprd1.__data_transfer_cache_size=0
primeprd2.__data_transfer_cache_size=0
primeprd1.__db_cache_size=7046430720
primeprd2.__db_cache_size=7147094016
primeprd1.__java_pool_size=67108864
primeprd2.__java_pool_size=67108864
primeprd1.__large_pool_size=301989888
primeprd2.__large_pool_size=301989888
primeprd2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primeprd1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primeprd1.__pga_aggregate_target=3221225472
primeprd2.__pga_aggregate_target=3221225472
primeprd1.__sga_target=9663676416
primeprd2.__sga_target=9663676416
primeprd1.__shared_io_pool_size=469762048
primeprd2.__shared_io_pool_size=469762048
primeprd1.__shared_pool_size=1644167168
primeprd2.__shared_pool_size=1577058304
primeprd1.__streams_pool_size=100663296
primeprd2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='+DATAC1/primedr/','+DATAC1/primeprd/','+RECOC1/primedr/','+RECOC1/primeprd/'
*.db_name='primeprd'
*.db_recovery_file_dest='+DATAC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primeprdXDB)'
*.fal_client='primeprd'
*.fal_server='primedr'
primeprd2.instance_number=2
primeprd1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primeprd'
*.log_archive_dest_2='service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='+DATAC1/primedr/','+DATAC1/primeprd/','+RECOC1/primedr/','+RECOC1/primeprd/'
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primeprd2.thread=2
primeprd1.thread=1
primeprd2.undo_tablespace='UNDOTBS2'
primeprd1.undo_tablespace='UNDOTBS1'

— DR PFILE —

primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump' # Create on Both Nodes
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
#*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='PRIMEPRD','PRIMEDR' ##### Changed for dataguard
*.db_unique_name='primedr' ##### Changed for dataguard
*.db_name='primeprd'
*.db_recovery_file_dest='+RECOC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)' ##### Changed for dataguard
*.fal_client='primedr' ##### Changed for dataguard
*.fal_server='primeprd' ##### Changed for dataguard
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr' ##### Changed for dataguard
*.log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd' ##### Changed for dataguard
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='PRIMEPRD','PRIMEDR' ##### Changed for dataguard
*.remote_listener='exadr-scan:1521' ##### Changed for dataguard
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace='UNDOTBS2'
primedr1.undo_tablespace='UNDOTBS1'

cp -ip pfile_for_standby.txt pfile_to_apply.ora

vi pfile_to_apply.ora

— Actual PFILE after modifying all parameters —

primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__oracle_base='/u03/app/oracle'#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest='/u03/app/oracle/admin/primeprd/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='12.1.0.2.0'
#*.control_files='+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763','+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763'
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest='+DATAC1'
*.db_domain=''
*.db_file_name_convert='PRIMEPRD','PRIMEDR'
*.db_unique_name='primedr'
*.db_name='primeprd'
*.db_recovery_file_dest='+RECOC1'
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest='/u03/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)'
*.fal_client='primedr'
*.fal_server='primeprd'
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config='dg_config=(primeprd,primedr)'
*.log_archive_dest_1='LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr'
*.log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd'
*.log_archive_dest_state_1='ENABLE'
*.log_archive_dest_state_2='ENABLE'
*.log_archive_format='log%t_%s_%r.arc'
*.log_file_name_convert='PRIMEPRD','PRIMEDR'
*.remote_listener='exadr-scan:1521'
*.nls_length_semantics='CHAR'
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile='exclusive'
*.resource_manager_cpu_allocation=48
*.resource_manager_plan='DEFAULT'
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management='AUTO'
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace='UNDOTBS2'
primedr1.undo_tablespace='UNDOTBS1'

6. Create same RMAN backup directory in DR and copy the backup Piece and init file from Production to DR server.

Also do not forget to create directory for adump in path ‘/u03/app/oracle/admin/primeprd/adump’

startup nomount pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';
rman target sys/*****@primeprd auxiliary /

Recovery Manager: Release 12.1.0.2.0 - Production on Tue Jun 28 10:05:31 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: PRIMEPRD (DBID=111111111111111)
connected to auxiliary database: PRIMEPRD (not mounted)

show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name PRIMEPRD are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO 'SBT_TAPE';
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '/u03/rmanbkp/prime/autobackup_control_file_%F';
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO '%F'; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '/u03/rmanbkp/prime';
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u03/app/oracle/product/12.1.0.2/payapps/dbs/snapcf_primeprd1.f'; # default

 DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK;

9. start the MRP process using the below command in SQL prompt.

ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

10. In case if you get error while receiving logging onto standby use the below command and create password files on both nodes of production and dr

— For Oracle 12c, do not create password files with this method. Go to section below for Dataguard Broker Setup

orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd2 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr2 force=y ignorecase=y

select * from v$pwfile_users;

Check by logging in from sqlplus from both production site and DR site from both nodes

From DR both nodes

sqlplus sys/*****@primeprd as sysdba

From Prod Both nodes

sqlplus sys/******@primedr as sysdba

Stop and start recovery again in DR

alter database recover managed standby database cancel;

alter database recover managed standby database using current logfile disconnect from session;

select process,status from v$managed_standby;

11. Create the spfile and add the newly created control files

show parameter control_file

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919, +RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923

Add the controlfile string in the pfile you created earlier called pfile_to_apply.ora

cd /u03/rmanbkp/prime/
vi pfile_to_apply.ora

*.control_files=’+DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919′,’+RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923′

shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.

startup nomount pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';

create spfile='+DATAC1/primedr/spfileprimedr.ora' from pfile='/u03/rmanbkp/prime/pfile_to_apply.ora';

shutdown immediate;

cd /u03/app/oracle/product/12.1.0.2/payapps/dbs/

vi initprimedr1.ora

–add only one line pointing to the SPFILE in ASM

spfile=’+DATAC1/primedr/spfileprimedr.ora’

startup mount pfile='/u03/app/oracle/product/12.1.0.2/payapps/dbs/initprimedr1.ora';

show parameter spfile

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATAC1/primedr/spfileprimedr.
ora

alter database recover managed standby database using current logfile disconnect from session;

12. Create cluster server using the below command.

srvctl add database -d primedr -o /u03/app/oracle/product/12.1.0.2/payapps -p +DATAC1/primedr/spfileprimedr.ora -r physical_standby -a DATAC1,RECOC1 -s MOUNT
srvctl add instance -d primedr -i primedr1 -n exadrdbadm01
srvctl add instance -d primedr -i primedr2 -n exadrdbadm02

srvctl add service -db onlndr -s onlnprod -preferred onlndr1,onlndr2
srvctl add service -db primedr -s primeprod -preferred primedr1,primedr2

srvctl config service -s onlnprod -db onlndr
srvctl config service -s primeprod -db primedr

srvctl stop database -d primedr

-- if gives error stop from sqlplus

srvctl start database -d primedr -o mount

— Start Recovery Again —

 alter database recover managed standby database using current logfile disconnect from session;

13.1. Add the below entries in both production nodes tnsnames.ora file as a oracle user. (Not required from ORacle 12c 12.1.0.2 onwards)

primeprd_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd_DGMGRL)
)
)

13.2. Add the below entries in both DR nodes tnsnames.ora file as a oracle user.

primedr_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr_DGMGRL)
)
)

13.3. Add the below entries in production node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd1)
)

13.4. Add the below entries in production node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd2)
)

13.5. Add the below entries in DR node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr1)
)

13.6. Add the below entries in DR node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr2)
)

13.7. Using the below command to restart the listener.

$ srvctl stop listener
$ srvctl start listener

14.1. Set the below parameter in primary database.

alter system set dg_broker_config_file1 = '+DATAC1/PRIMEPRD/dr1primeprd.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATAC1/PRIMEPRD/dr2primeprd.dat' scope=both sid='*';
alter system set dg_broker_start = true scope=both sid='*';

14.2. Set the below parameter in standby database.

alter system set dg_broker_config_file1 = '+DATAC1/PRIMEDR/dr1primedr.dat' scope=both sid='*';
alter system set dg_broker_config_file2 = '+DATAC1/PRIMEDR/dr2primedr.dat' scope=both sid='*';
alter system set dg_broker_start = true scope=both sid='*';

15.1. Configure the dataguard brocker using the below command from production database server

/home/oracle::11g- dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

Configuration "dg_primeprd" created with primary database "primeprd"

add database primedr as connect identifier is primedr maintained as physical;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.

remove configuration;
Removed configuration

Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

alter system set log_archive_dest_2='' scope=both sid='*';

Disable then Enable the broker parameter on both the Primary and Standby databases.

— Primary

alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';

— Standby

alter system set dg_broker_start=false scope=both sid='*';
alter system set dg_broker_start=true scope=both sid='*';

On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.

/home/oracle::11g-- dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

enable configuration;

Set back the log_archive_dest_2 on both production and dr

— In Production Database —

alter system set log_archive_dest_2='service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr' scope=both sid='*';

— In DR Database —

alter system set log_archive_dest_2='service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd' scope=both sid='*';

If in Broker you get below error

 show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database
Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 40 seconds ago)

Check in Broker Log File.

Mon Jul 04 14:41:46 2016
Error 1017 received logging on to the standby
------------------------------------------------------------
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

Failed to send message to site . Error code is ORA-01017.

Then re-create password files in ASM in 12c like below, the correct format is pwd

Primary

srvctl config database -d primeprd
orapwd file='+DATAC1/PRIMEPRD/pwdprimeprd' entries=10 dbuniquename=primeprd password=****** force=y ignorecase=y

Standby

srvctl config database -d primedr
orapwd file='+DATAC1/PRIMEDR/pwdprimedr' entries=10 dbuniquename=primedr password=****** force=y ignorecase=y

Check configuration now and the error should have vanished and the status is success

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 18 seconds ago)

— Try doing Switchover using DGMGRL Now —

 switchover to primedr
Performing switchover NOW, please wait...
Error: ORA-16644: apply instance not available

Failed.
Unable to switchover, primary database is still "primeprd"

edit database primedr set state='APPLY-ON' with apply instance='primedr1';

show database verbose primedr

Database - primedr

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 6.00 KByte/s
Active Apply Rate: 755.00 KByte/s
Maximum Apply Rate: 827.00 KByte/s
Real Time Query: OFF
Instance(s):
primedr1 (apply instance)
primedr2

switchover to primedr
Performing switchover NOW, please wait...
New primary database "primedr" is opening...
Oracle Clusterware is restarting database "primeprd" ...
Switchover succeeded, new primary is "primedr"

 show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr - Primary database
primeprd - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS (status updated 40 seconds ago)

edit database primeprd set state='APPLY-ON' with apply instance='primeprd1';

show database verbose primeprd

Database - primeprd

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 307.00 KByte/s
Active Apply Rate: 0 Byte/s
Maximum Apply Rate: 0 Byte/s
Real Time Query: OFF
Instance(s):
primeprd1 (apply instance)
primeprd2

DGMGRL

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr - Primary database
primeprd - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 55 seconds ago)

Connect to DR site broker to intiate switchover

switchover to primeprd ;
Performing switchover NOW, please wait...
Operation requires a connection to instance "primeprd1" on database "primeprd"
Connecting to instance "primeprd1"...
Connected as SYSDBA.
New primary database "primeprd" is opening...
Oracle Clusterware is restarting database "primedr" ...
Switchover succeeded, new primary is "primeprd

show configuration

Configuration - dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd - Primary database
primedr - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 11 seconds ago
Advertisements

Oracle Dataguard Broker 12c with password file in ASM

Configure 12c dataguard broker using the below command onPrimary database server

primeprd – > tnsnames for primary database

primedr-> tnsnames for standby database

/home/oracle::11g–>dgmgrl

DGMGRL> connect sys/********

DGMGRL> create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;

Configuration “dg_primeprd” created with primary database “primeprd”

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.
DGMGRL> remove configuration;
Removed configuration
Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

alter system set log_archive_dest_2=” scope=both sid=’*’;
Disable then Enable the broker parameter on both the Primary and Standby databases.

— Primary
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;

— Standby
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;

On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.

/home/oracle::11g–>dgmgrl

DGMGRL> connect sys/**********

DGMGRL> create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

DGMGRL> add database primedr as connect identifier is primedr maintained as physical;

DGMGRL> enable configuration;
If in Broker you get below error Error: ORA-16664: unable to receive the result from a database

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database
Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 40 seconds ago)

Check in Broker Log File.

Mon Jul 04 14:41:46 2016
Error 1017 received logging on to the standby
————————————————————
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

Failed to send message to site . Error code is ORA-01017.

Re-create password files in ASM in 12c like below, the correct format is pwd<dbuniquename>

Primary

srvctl config database -d primeprd

orapwd file=’+DATAC1/PRIMEPRD/pwdprimeprd’ entries=10 dbuniquename=primeprd password=*** force=y ignorecase=y
Standby

srvctl config database -d primedr
orapwd file=’+DATAC1/PRIMEDR/pwdprimedr’ entries=10 dbuniquename=primedr password=*** force=y ignorecase=y

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 18 seconds ago)

— Try Switchover Now —
DGMGRL> switchover to primedr
Performing switchover NOW, please wait…
Error: ORA-16644: apply instance not available

Failed.
Unable to switchover, primary database is still “primeprd”

DGMGRL> edit database primedr set state=’APPLY-ON’ with apply instance=’primedr1′;

DGMGRL> show database verbose primedr

Database – primedr

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 6.00 KByte/s
Active Apply Rate: 755.00 KByte/s
Maximum Apply Rate: 827.00 KByte/s
Real Time Query: OFF
Instance(s):
primedr1 (apply instance)
primedr2
DGMGRL> switchover to primedr
Performing switchover NOW, please wait…
New primary database “primedr” is opening…
Oracle Clusterware is restarting database “primeprd” …
Switchover succeeded, new primary is “primedr”
DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 40 seconds ago)
DGMGRL> edit database primeprd set state=’APPLY-ON’ with apply instance=’primeprd1′;

DGMGRL> show database verbose primeprd

Database – primeprd

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 307.00 KByte/s
Active Apply Rate: 0 Byte/s
Maximum Apply Rate: 0 Byte/s
Real Time Query: OFF
Instance(s):
primeprd1 (apply instance)
primeprd2

DGMGRL>
DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 55 seconds ago)

Connect to DR site broker to intiate switchover

DGMGRL> switchover to primeprd ;
Performing switchover NOW, please wait…
Operation requires a connection to instance “primeprd1” on database “primeprd”
Connecting to instance “primeprd1″…
Connected as SYSDBA.
New primary database “primeprd” is opening…
Oracle Clusterware is restarting database “primedr” …
Switchover succeeded, new primary is “primeprd

DGMGRL> show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 11 seconds ago

RMAN — ORA-19909: datafile 1 belongs to an orphan incarnation in a Standby Database.

We had an interesting scenario where we did a full rman refresh of a standby database becuase the archivelogs went missing and there was a gap. After doing the restore we got the below error in the RMAN recovery

rman nocatalog target sys/******@mw auxiliary /
connected to target database: MW (DBID=000000000)
using target database control file instead of recovery catalog
connected to auxiliary database: MW (not mounted)
RMAN> DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK DORECOVER;
The restore was succesful but the recovery gave below error

Oracle Error:
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01152: file 1 was not restored from a sufficiently old backup
ORA-01110: data file 1: ‘+DATAC1/mwdr/datafile/system.572.867661777’
After doing a lot of googling and searching support.oracle.com it turns out it has something to do with the
incarnation of databases. We checked the incarnation for both the Primary and Standby database.

Output for Primary Database
RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 CURRENT 925702 10-JUL-14
Output for Standby Database

RMAN> list incarnation of database;

List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 ORPHAN 925702 10-JUL-14
4 4 MW 1312370394 ORPHAN 20799166 23-OCT-14
3 3 MW 1312370394 CURRENT 22477563 30-OCT-14


Solution:

As we can see clearly from the output the primary database is currenly on a different incarnation from the standby database. The way to address it is to reset the incarnation on the standby database to match the primary incarnation.
RMAN> reset database to incarnation 2;

RMAN> list incarnation of database;
List of Database Incarnations
DB Key Inc Key DB Name DB ID STATUS Reset SCN Reset Time
——- ——- ——– —————- — ———- ———-
1 1 MW 1312370394 PARENT 1 24-AUG-13
2 2 MW 1312370394 CURRENT 925702 10-JUL-14
4 4 MW 1312370394 ORPHAN 20799166 23-OCT-14
3 3 MW 1312370394 ORPHAN 22477563 30-OCT-14
As you can now the incarnations match. Now do the recovery for the database again and the logs will apply and start shipping like normal again.

Failed Primary Conversion to Physical Standby

1) Query on old standby database (new primary)

SQL> SELECT TO_CHAR(STANDBY_BECAME_PRIMARY_SCN) FROM V$DATABASE;

TO_CHAR(STANDBY_BECAME_PRIMARY_SCN)
—————————————-
12496643189

2) Flash back the failed primary database.

SQL> shutdown immediate;

SQL> startup mount;

SQL> FLASHBACK DATABASE TO SCN 12496643189;

Flashback complete.

3) Convert old primary into standby database

SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

SQL> SHUTDOWN IMMEDIATE;

SQL> STARTUP MOUNT;

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;

4) Make log switch from New Primary

SQL> ALTER SYSTEM ARCHIVE LOG CURRENT;

RAC to Single Instance Standby Database Setup (Oracle 11gR2 Physical Standby)

Primary is a 2 node-rac cluster with instances swx1 and swx2 running on ASM. A service called swxscan is created for client sides. We will use this service name for the tnsnames.ora.

The standby instance will be called swxstandby and it will be running single instance on ASM as well.

This is a standard dataguard configuration but the only tricky part is the listeners on primary and DR, since in 11gR2 we have the Grid Infrastructure running under Grid user and RDBMS under oracle (provided you followed Oracle’s OFA, which by the way is the best way to do things imho)

Also take care for the user permissions on DR server for the raw devices. Since grid user owns the raw devices the group which grid is part of should also have oracle user as its member.

Do we will begin the procedure now. First we will do steps on Production RAC and then do the remaining on Standby server. I am not going to use active duplicate over the network. The reason for this is that in a production environment where size of your database is genereally above 500gb. If your bandwidth is not good enough it might take days to duplicate the database over network. Its better to use the old duplicate command and copy the rman backups over using a tape or any restore method at your disposal. What I do is backup it up to tape and then restore on DR site.

———————-

Environment

——————–
Primary Database 2 Node RAC
1 Node Names: swxracnode1, swxracnode2
2 DB Name: SWX
3 DB Unique Name: SWX
4 DB Version: 11.2.0.3
5 Grid Infrastructure (CRS + ASM).
6 SCAN settings in /etc/hosts file, so SCAN listener only running on one node (Swxracnode2 for this demo)
7 ASM Diskgroups: +DATA, +FRA
8 CRS installed with ‘Grid’ User, RDBMS installed with ‘Oracle’ user

Standby Database Single Instance:
1 ASM Diskgroup : +DATA, +FRA (For datafiles and FRA)
2 DB Unique Name: SWXSTANDBY
3 DB Name: SWX
4 CRS(for single instance) installed with ‘Grid’ User, RDBMS installed with ‘Oracle’ user

—————————
PREPARATIONS AT PRIMARY SITE
—————————-

GI_HOME= /grid/11.2.0/grid
ORACLE_HOME=/oracle/app/oracle/product/11.2.0/db_1

CRS User= Grid
RDBMS User= Oracle

1. Force logging on primary.
Login to any one node and run below sql command

alter database force logging;

2. Check for Online Redo Logfiles(ORL) on nodes and Create Standby Redo Logs (SRL) on the Primary RAC.

select * from gv$logfile;
select * from gv$log;

— ORL ADDITION —
Add atleast four groups per thread of size 100M;

ALTER DATABASE ADD LOGFILE THREAD 1
GROUP 1 (‘+DATA/swx/onlinelog/redo-01-01.log’,’+DATA/swx/onlinelog/redo01-02.log’) SIZE 100m,
GROUP 2 (‘+DATA/swx/onlinelog/redo-02-01.log’,’+DATA/swx/onlinelog/redo-02-02.log’) SIZE 100m;

ALTER DATABASE ADD LOGFILE THREAD 2
GROUP 3 (‘+DATA/swx/onlinelog/redo-03-01.log’,’+DATA/swx/onlinelog/redo-03-02.log’) SIZE 100m,
GROUP 4 (‘+DATA/swx/onlinelog/redo-04-01.log’,’+DATA/swx/onlinelog/redo-04-02.log’) SIZE 100m;

ALTER DATABASE ADD LOGFILE THREAD 1
GROUP 5 (‘+DATA/swx/onlinelog/redo-05-01.log’,’+DATA/swx/onlinelog/redo05-02.log’) SIZE 100m,
GROUP 6 (‘+DATA/swx/onlinelog/redo-06-01.log’,’+DATA/swx/onlinelog/redo-06-02.log’) SIZE 100m;

ALTER DATABASE ADD LOGFILE THREAD 2
GROUP 7 (‘+DATA/swx/onlinelog/redo-07-01.log’,’+DATA/swx/onlinelog/redo-07-02.log’) SIZE 100m,
GROUP 8 (‘+DATA/swx/onlinelog/redo-08-01.log’,’+DATA/swx/onlinelog/redo-08-02.log’) SIZE 100m;

ALTER DATABASE ADD LOGFILE THREAD 1
GROUP 9 (‘+DATA/swx/onlinelog/redo-09-01.log’,’+DATA/swx/onlinelog/redo-09-02.log’) SIZE 100m,
GROUP 10 (‘+DATA/swx/onlinelog/redo-10-01.log’,’+DATA/swx/onlinelog/redo-10-02.log’) SIZE 100m;

ALTER DATABASE ADD LOGFILE THREAD 2
GROUP 11 (‘+DATA/swx/onlinelog/redo-11-01.log’,’+DATA/swx/onlinelog/redo-11-02.log’) SIZE 100m,
GROUP 12 (‘+DATA/swx/onlinelog/redo-12-01.log’,’+DATA/swx/onlinelog/redo-12-02.log’) SIZE 100m;

Create Standby Redo Logs (SRL) related to the online redo log (ORL) files in the database. As there are three redo log groups with two members each we will create four members for each thread. Note that OMF (db_create_file_dest=’+DATA’) are used and the SRL are created with the same size as ORL.

(maximum # of logfiles +1) * maximum # of threads

For example in setup using two online log files for each thread. Thus, the number of standby redo logs should be (2 + 1) * 2 = 6. That is, one more standby redo log file for each thread.

3. Set DB_RECOVERY_FILE_DEST_SIZE, DB_RECOVERY_FILE_DEST and ADD the SRL’s

SQL> alter system set DB_RECOVERY_FILE_DEST_SIZE=400G scope=both sid=’*’;

SQL> alter system set DB_RECOVERY_FILE_DEST=’+FRA’ scope=both sid=’*’;

alter database add standby logfile thread 1 group 7 size 104857600;

alter database add standby logfile thread 1 group 8 size 104857600;

alter database add standby logfile thread 1 group 9 size 104857600;

alter database add standby logfile thread 1 group 10 size 104857600;

alter database add standby logfile thread 2 group 11 size 104857600;

alter database add standby logfile thread 2 group 12 size 104857600;

alter database add standby logfile thread 2 group 13 size 104857600;

alter database add standby logfile thread 2 group 14 size 104857600;

4. Bring Database in Archivelog mode

alter system set log_archive_dest_1=’location=+fra’ scope=both sid=’*’;

srvctl stop database -d swx

### log into both nodes

sqlplus “/as sysdba”

startup mount

alter database archivelog;

SQL> archive log list;

Database log mode Archive Mode
Automatic archival Enabled
Archive destination +FRA
Oldest online log sequence 1625
Next log sequence to archive 1626
Current log sequence 1626

SQL> alter database open;

5. Configure Listener.ora (GRID user) and Tnsnames.ora (ORACLE user) for both nodes on Primary Side
Configure tnsnames alias and a static registration with the GI listeners for both primary and standby database and compliant with Data Guard Broker. With Oracle 11gR2 since the SCAN Listener was introduced the modification is made to both listeners in GI $OH and to the tnsnames.ora in RDBMS $OH. Reload the listeners.

— Listener.ora on both Nodes of Primary RAC as Grid User–

LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))
# line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S
CAN3)))) # line added by Agent
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S
CAN2)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_S
CAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by
Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON # line added by
Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON # line added by
Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent

SWXSTANDBY =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxracdr)(PORT = 1521))
)
)

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swx)
(SID_NAME = swx)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swxstandby)
(SID_NAME = swxstandby)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swx_DGMGRL)
(SID_NAME = swx)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swxstandby_DGMGRL)
(SID_NAME = swxstandby)
)
)

## As grid user in $ORACLE_HOME (which is the Grid Home) do the below on both nodes.Start and Stop all scan listeners on both nodes

lsnrctl stop LISTENER_SCAN1
lsnrctl stop LISTENER_SCAN2
lsnrctl stop LISTENER_SCAN3

lsnrctl start LISTENER_SCAN1
lsnrctl start LISTENER_SCAN2
lsnrctl start LISTENER_SCAN3

— TNSNAMES.ora on both Nodes of Primary RAC as Oracle user —

SWXSTANDBY =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxracdr)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = swxstandby)
)
)

SWX =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxscan)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = swx)
)
)

## tnsping ping swx, swxstandby as Oracle user to check ### If not pingabe check if DNS is configured on your server

6. Create password files for the Nodes as Oracle user.

User format as orapwswx1 for node 1
User format as orapwswx2 for node 2

Create the password file in folder $ORACLE_HOME/dbs on all nodes.

orapwd file=$ORACLE_HOME/dbs/orapwsswx1 ignorecase=y password=xxxxxxxx force=y
orapwd file=$ORACLE_HOME/dbs/orapwsswx2 ignorecase=y password=xxxxxxxx force=y

–ignore case is important parameter here since from 11gR2 onwards passwords are case-sensitive

PING[ARC2]: Heartbeat failed to connect to standby ‘swx’. Error is 16191.
Error 1017 received logging on to the standby

These are common errors associated with password files. If you encounter the above errors, recreate the password files.

7. Create initialization parameter file and add Standby database parameters to it.

We will not use Active duplication over the network. Since in a production environment it is sometimes impractical; where size of database can easily exceed 1TB. The method to duplicate via RMAN and copy the backup to Standby database server is a tried and tested method and works beautifully.

Login to sqlplus on one of the nodes.

show parameter spfile;

create pfile=’/backup/swxPRIMARY.ora’ from spfile;

Ensure you remote_listener parameter is set to your scan listener

*.remote_listener=’swxscan.domain.com:1521′

Add the below parameters to the new swxPRIMARY.ora file

#########################STANDBY PARAMETERS########################
*.DB_UNIQUE_NAME=’swx’
*.db_file_name_convert=’+DATA/SWXSTANDBY/’,’+DATA/SWX/’,’+FRA/SWXSTANDBY/’,’+FRA/SWX/’
*.log_archive_config=’dg_config=(swxstandby,swx)’
*.log_archive_dest_1=’LOCATION=+FRA valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=swx’
*.log_archive_dest_2=’service=swxstandby LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=swxstandby’
*.LOG_ARCHIVE_DEST_STATE_1=’ENABLE’
*.LOG_ARCHIVE_DEST_STATE_2=’ENABLE’
*.LOG_ARCHIVE_FORMAT=’log%t_%s_%r.arc’
*.log_file_name_convert=’+DATA/SWXSTANDBY/’,’+DATA/SWX/’,’+FRA/SWXSTANDBY/’,’+FRA/SWX/’
*.remote_login_passwordfile=’exclusive’
*.STANDBY_FILE_MANAGEMENT=’AUTO’
*.fal_client=’SWX’
*.fal_server=’SWXSTANDBY’
*.sec_case_sensitive_logon=FALSE
########################################

FULL PARAMETER FILE OF THE PRIMARY DATABASE

swx2.__db_cache_size=4294967296
swx1.__db_cache_size=3892314112
swx2.__java_pool_size=16777216
swx1.__java_pool_size=33554432
swx2.__large_pool_size=33554432
swx1.__large_pool_size=50331648
swx2.__pga_aggregate_target=2147483648
swx1.__pga_aggregate_target=2147483648
swx2.__sga_target=6442450944
swx1.__sga_target=6442450944
swx2.__shared_io_pool_size=536870912
swx1.__shared_io_pool_size=536870912
swx2.__shared_pool_size=1526726656
swx1.__shared_pool_size=1862270976
swx2.__streams_pool_size=0
swx1.__streams_pool_size=33554432
*.audit_file_dest=’/oracle/app/oracle/admin/swx/adump’
*.audit_trail=’db’
*.cluster_database=true
*.compatible=’11.2.0.0.0′
*.control_files=’+DATA/swx/controlfile/current.265.773009443′
*.db_block_size=8192
*.db_create_file_dest=’+DATA’
*.db_domain=”
*.db_name=’swx’
*.db_recovery_file_dest_size=429496729600
*.db_recovery_file_dest=’+FRA’
*.diagnostic_dest=’/oracle/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=swxXDB)’
*.swx1.instance_number=1
*.swx2.instance_number=2
*.job_queue_processes=1000
*.log_archive_config=’dg_config=(swxstandby,swx)’
*.log_archive_dest_1=’LOCATION=+FRA valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=swx’
*.log_archive_dest_2=’service=swxstandby LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=swxstandby’
*.open_cursors=300
*.pga_aggregate_target=2147483648
*.processes=1000
*.remote_listener=’swxscan.qiibonline.com:1521′
*.remote_login_passwordfile=’exclusive’
*.sessions=1105
*.sga_target=6442450944
*.standby_file_management=’AUTO’
*.swx2.thread=2
*.swx1.thread=1
*.swx2.undo_tablespace=’UNDOTBS2′
*.swx1.undo_tablespace=’UNDOTBS1′
*.db_file_name_convert=’+DATA/SWXSTANDBY/’,’+DATA/SWX/’,’+FRA/SWXSTANDBY/’,’+FRA/SWX/’
*.log_file_name_convert=’+DATA/SWXSTANDBY/’,’+DATA/SWX/’,’+FRA/SWXSTANDBY/’,’+FRA/SWX/’
*.standby_file_management=auto
*.fal_server=’SWXSTANDBY’
*.fal_client=’SWX’

Bounce the database with the new parameter file

sqlplus “/as sysdba”

shutdown immediate;

— shutdown 2nd node also, you can use ‘srvctl stop database -d swx’ for this also

startup pfile=’/backup/swxPRIMARY.ora’;

create spfile from pfile=’/backup/swxPRIMARY.ora’;

shutdown immediate;

startup;

show parameter spfile;

Now we have completed the steps at Primary Side, only the duplicate command from RMAN is remaining. But let us first ensure we prepare the standby database before we duplicate and restore the database.

—————————
PREPARATIONS AT STANDBY SITE
—————————-

GI_HOME= /grid/11.2.0/grid
ORACLE_HOME=/oracle/app/oracle/product/11.2.0/db_1

CRS User= Grid
RDBMS User= Oracle

The installation for database is grid infrastrcuture utilizing ASM for the database. We have already created the HAS services and the RDBMS software was also installed. In ASM 2 diskgroups called +DATA, +FRA was created, same like production.

We will create a new database using the intialization parameter file we created before but we will add new parameters for the standby database. If your are doing this setup on windows then you will have to create a windows service first using ‘oradim’

1. Create listener.ora with Grid user

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = swxracdr)(PORT = 1521))
)
)

ADR_BASE_LISTENER = /grid/app

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent

SWXSTANDBY =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxracdr)(PORT = 1521))
)
)

SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swx)
(SID_NAME = swx)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swxstandby)
(SID_NAME = swxstandby)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swx_DGMGRL)
(SID_NAME = swx)
)
(SID_DESC =
(ORACLE_HOME= /grid/11.2.0/grid)
(GLOBAL_DBNAME = swxstandby_DGMGRL)
(SID_NAME = swxstandby)
)
)

Reload the listeners using lsnrctl reload command

2. Create tnsnames.ora file using Oracle user

SWX =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxscan.domain.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = swx)
)
)

SWXSTANDBY =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = swxracdr.domain.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = swxstandby)
)
)

3. Create Directory for adump audit_file_dest, also create directories in the ASM diskgroup using ASMCMD

— as oracle user —
mkdir -p /oracle/app/oracle/admin/swx/adump

asmcmd>

asmcmd> mkdir +DATA/SWXSTANDBY/
asmcmd> mkdir +DATA/SWX/
asmcmd> mkdir +FRA/SWXSTANDBY/
asmcmd> mkdir +FRA/SWX/

Also create any other directories you think you have missed in ASM or on the local filesystem.

4. Change intilization parameters in swxPRIMARY.ora file and name file as swxSTANDBY.ora

Since it is RAC to single instance dataguard the parameter cluster_database needs to be set to false. This is the primary difference between a single instance and a rac database.

#########################STANDBY DATABASE PARAMETERS########################
*.DB_UNIQUE_NAME=’SWXSTANDBY’
*.db_file_name_convert=’+DATA/SWX/’,’+DATA/SWXSTANDBY/’,’+FRA/SWX/’,’+FRA/SWXSTANDBY/’
*.log_archive_config=’dg_config=(swxstandby,swx)’
*.log_archive_dest_1=’LOCATION=+FRA valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=swxstandby’
*.log_archive_dest_2=’service=swx LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=swx’
*.LOG_ARCHIVE_DEST_STATE_1=’ENABLE’
*.LOG_ARCHIVE_DEST_STATE_2=’ENABLE’
*.LOG_ARCHIVE_FORMAT=’log%t_%s_%r.arc’
*.log_file_name_convert=’+DATA/SWX/’,’+DATA/SWXSTANDBY/’,’+FRA/SWX/’,’+FRA/SWXSTANDBY/’
*.remote_login_passwordfile=’exclusive’
*.STANDBY_FILE_MANAGEMENT=’AUTO’
*.fal_client=’SWXSTANDBY’
*.fal_server=’SWX’
*.cluster_database=false
*.sec_case_sensitive_logon=FALSE
########################################

FULL PARAMETER FILE swxSTANDBY.ora

swxstandby.__db_cache_size=3724541952
swxstandby.__java_pool_size=16777216
swxstandby.__large_pool_size=50331648
swxstandby.__pga_aggregate_target=2147483648
swxstandby.__sga_target=6442450944
swxstandby.__shared_io_pool_size=536870912
swxstandby.__shared_pool_size=2063597568
swxstandby.__streams_pool_size=16777216
*.audit_file_dest=’/oracle/app/oracle/admin/swx/adump’
*.cluster_database=false
*.compatible=’11.2.0.0.0′
*.control_files=’+DATA/swxstandby/controlfile/current.272.792584487’#Restore Controlfile
*.db_block_size=8192
*.db_create_file_dest=’+DATA’
*.db_create_online_log_dest_1=’+FRA’
*.db_domain=”
*.db_file_name_convert=’+DATA/SWX/’,’+DATA/SWXSTANDBY/’,’+FRA/SWX/’,’+FRA/SWXSTANDBY/’
*.db_name=’swx’
*.db_recovery_file_dest_size=429496729600
*.db_recovery_file_dest=’+FRA’
*.db_unique_name=’SWXSTANDBY’
*.diagnostic_dest=’/oracle/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=swxXDB)’
*.fal_client=’SWXSTANDBY’
*.fal_server=’SWX’
*.job_queue_processes=1000
*.log_archive_config=’dg_config=(swxstandby,swx)’
*.log_archive_dest_1=’LOCATION=+FRA valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=swxstandby’
*.log_archive_dest_2=’service=swx LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=swx’
*.log_file_name_convert=’+DATA/SWX/’,’+DATA/SWXSTANDBY/’,’+FRA/SWX/’,’+FRA/SWXSTANDBY/’
*.open_cursors=300
*.pga_aggregate_target=2147483648
*.processes=1000
*.remote_listener=’swxscan.qiibonline.com:1521′
*.remote_login_passwordfile=’exclusive’
*.sec_case_sensitive_logon=FALSE
*.sessions=1105
*.sga_target=6442450944
*.standby_file_management=’AUTO’
*.undo_tablespace=’UNDOTBS1′

5. Now startup the database in nomount mode using this pfile and create spfile from it.

sqlplus “/as sysdba”

startup nomount pfile=’/backup/swxSTANDBY.ora’;

create spfile from pfile=’/backup/swxSTANDBY.ora’;

shutdown immediate;

startup nomount;

6. Create password file for standby database

Create the password file in folder $ORACLE_HOME/dbs on all nodes and DR server as Oracle user

User format as orapwswxstandby for Standby database

orapwd file=$ORACLE_HOME/dbs/orapwswxstandby force=y ignorecase=y password=xxxxxx

To check password file use query from sql plus.

select * from v$pwfile_users;

USERNAME SYSDB SYSOP SYSAS
—————————— —– —– —–
SYS TRUE TRUE FALSE

—————————
DUPLICATING THE DATABASE FOR STANDBY FROM RMAN
—————————-

1. Before duplicating ensure that the mountpoint /backup exists on the DR server. This is the location where we will copy the backups to and restore from there.

RMAN > show all;

CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/backup/rmanbkp/SWX_Lev0_%s_%D_%M_%Y’;

RMAN> sql ‘alter system archive log current’;

RMAN> backup database include current controlfile for standby plus archivelog;

Copy the database to the DR server on folder ‘/backup/rmanbkp’ and now we will begin to duplicate

Login to RMAN from DR server from Oracle user like below

rman target sys/***@swx auxiliary sys/***

or

rman target sys/***@swx auxiliary /

or

rman target sys/***@swx auxiliary sys/****@swxstandby

$ rman target sys/*****@swx auxiliary /

Recovery Manager: Release 11.2.0.3.0 – Production on Tue Nov 20 15:39:01 2012

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: SWX (DBID=xxxxxx)
connected to auxiliary database: SWX (DBID=xxxxxx)

If you get some error use (UR=A) in tnsnames.ora file like

(SERVICE_NAME = swx) (UR=A)

Go to primary server and crosscheck archivelogs and backupsets and delete expired or obsolete objects and relogin to auxiliary.

RMAN> duplicate target database for standby nofilenamecheck dorecover;

2. Put the Standby into Managed standby mode:
ON standby server

shutdown immediate;

alter database mount standby database;

alter database recover managed standby database using current logfile disconnect from session;

Go to the Primary Server:

SQL> alter system switch logfile;
SQL> alter system archive log current;

Check if the logs are coming here and properly applying.

3. Verify the sync between primary and standby

From primary
sql > select max(sequence#) as “PRIMARY” from v$log_history;

From Standby

SQL > select max(sequence#) as “STANDBY”,applied from v$archived_log group by applied ;

SQL> Select process,status from v$managed_standby;

If there is error like below , it is generally due to password files and for 11gR2 the parameter sec_case_sensitive_logon

PING[ARC2]: Heartbeat failed to connect to standby ” Error is 16191.
Error 1017 received logging on to the standby

Recreate the password files and alter sec_case_sensitive_logon to false

Invoking a DR Standby Database

Overview

There are different methods you can use to invoke a standby database depending on the typical DR scenario you are facing.

Scenario for usage.

1. Database Switchover

Using this method you can switch backwards and forwards between the primary and DR servers (e.g. so that the primary can become DR and DR can become primary) without having to rebuild either environment.

This scenario can be used if both the Primary and Standby Servers are available so that you can connect to both environments simultaneously and perform a clean switchover.

2. Activating a Standby Database

This method will activate the Standby Database as the primary database recovering up to the point of the last log shipment. After activating a standby database as the primary the original primary server becomes obsolete and will need to be rebuilt as a standby database (e.g. you can not just switch the servers back to act as a primary or standby database and will need to rebuild the data-guard environments).

This scenario can be used if the primary server is not available and you need to force the standby database to become the primary.

3. Opening the standby Database in Read-Only Mode

This method involves stopping data-guard replication and opening the standby database in read-only mode for querying. The database can then be shutdown and reopened in standby mode so that data-guard replication can resume (assuming all required archive logs are still available)

This scenario can be used if you want to open the standby database temporarily for querying data and then wish to put the database back into standby mode.

The following section describes how to perform each of the above methods of invoking a standby database:

1. Database Switchover

Using this method you can switch backwards and forwards between the primary and DR servers (e.g. so that the primary can become DR and DR can become primary) without having to rebuild either environment:

On Primary Server:

SQL> alter database commit to switchover to standby;

This may cause the following error to be generated:

ERROR at line 1:

ORA-01093: ALTER DATABASE CLOSE only permitted with no sessions connected

If this does occur then restart the database, as below, before retrying the above command:

SQL> shutdown immediate

SQL> startup

SQL> shutdown immediate

SQL> startup nomount

SQL> alter database mount standby database;

SQL> alter database recover managed standby database disconnect;

The primary server is now configured as a DR standby database.

On DR Server:

SQL> alter database recover managed standby database cancel;

SQL> alter database commit to switchover to primary;

SQL> shutdown immediate

SQL> startup

The DR server is now configured as the primary database.

To switch back you just need to repeat the above process but the other way around (e.g. convert the DR database back to a standby and the primary database back to primary).

2. Activating a Standby Database

If the primary database is not available the standby database can be converted into the primary database as follows:

SQL> alter database recover managed standby database cancel;

SQL> alter database activate standby database;

SQL> shutdown immediate

SQL> startup

The original primary database is now obsolete and can be rebuilt as a standby database once it is available again.

3. Opening the Standby Database in Read Only Mode (Active Dataguard)

The standby database can be opened in read only mode for querying and then converted back into a standby database without affecting the primary.

On standby server:

SQL> alter database recover managed standby database cancel;

SQL> alter database open read only;

The standby database is now open and available for querying in read only mode.

To put the standby database back into standby mode:

SQL> shutdown immediate

SQL> startup nomount

SQL> alter database mount standby database;

SQL> alter database recover managed standby database disconnect;

=====================================================================================