Restore and Upgrade Oracle Database from 11gR2 to 12c from Exadata X4-2 to X7-2

Upgrade from Exadata X4-2 11g (11.2.0.4) to Exadata X7-2 12CR1 (12.1.0.2.0) : ASM to ASM

Oracle Database 11.2.0.3and 11.2.0.4 to Oracle 12.2.0.1 Direct upgrade is supported by Oracle. If you have Database lower than those version than you have to upgrade to 11.2.0.4 first and you can upgrade to 12cR2

In our Scenario We will Migrate and Upgrade from One Exadata machine (X4-2) to another Exadata machine (X7-2) from 11.2.0.4 to 12.1.0.2.0

Source Exadata Setup

———————-

RAC 2-Nodes

Hardware : X4-2

Image version: 12.1.2.3.5.170418

DB NAME : mob

DATABASE HOME VERSION: 11.2.0.4

GRID HOME VERSION : 12.1.0.2.0

ASM Diskgroups : +DATAC1, +RECOC1

Target Exadata Setup

—————–

RAC 2-Nodes

Hardware : X7-2

Image version: 18.1.4.0.0.180125.3

DB NAME: SWX

DATABASE HOME VERSION : 12.1.0.2.0

GRID HOME VERSION : 12.2.0.1.0

ASM Diskgroups : +DATAC1, +RECOC1

 

Presumptions

————

* Oracle Home (12.1.0.2.0) is installed on the Target Server with latest patches

* Both Exadata Servers are reachable from the Network to Copy the backup files

1. Take RMAN Backup of Source Database on X4-2 hardware

rman target / nocatalog

run

{

allocate channel swx1 DEVICE TYPE DISK ;

allocate channel swx2 DEVICE TYPE DISK ;

allocate channel swx3 DEVICE TYPE DISK ;

allocate channel swx4 DEVICE TYPE DISK ;

CONFIGURE CONTROLFILE AUTOBACKUP ON;

CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘/exabackups/rmanbkp/swx/autobackup_control_file_%F’;

CONFIGURE DEFAULT DEVICE TYPE TO DISK;

BACKUP DATABASE FORMAT ‘/exabackups/rmanbkp/swx/SWX_FULL_%s_%D_%M_%Y’ tag ‘SWX_DAILY_FULL_DB_BKP’;

BACKUP ARCHIVELOG ALL FORMAT ‘/exabackups/rmanbkp/swx/ARCH_%d_ALL_%s_%t’ tag ‘SWX_ARCH_LOG_BKP’;

}

SQL> create pfile=’/exabackups/rmanbkp/swx/swx_source.ora’ from spfile ;

2. Copy Backup from Source Server to Target Server

cd /exabackups/rmanbkp/swx

scp -r * oracle@qiibtstdbadm01.qiibonline.com:/u03/swx_today

3. Edit the Source pfile : swx_source.ora and change below parameters

Old pfile

———

swx1.__db_cache_size=9395240960

swx2.__db_cache_size=9395240960

swx1.__java_pool_size=33554432

swx2.__java_pool_size=33554432

swx1.__large_pool_size=67108864

swx2.__large_pool_size=67108864

swx2.__oracle_base=’/u02/app/oracle’#ORACLE_BASE set from environment

swx1.__oracle_base=’/u02/app/oracle’#ORACLE_BASE set from environment

swx1.__pga_aggregate_target=6006243328

swx2.__pga_aggregate_target=6006243328

swx1.__sga_target=11173625856

swx2.__sga_target=11173625856

swx1.__shared_io_pool_size=0

swx2.__shared_io_pool_size=0

swx1.__shared_pool_size=1577058304

swx2.__shared_pool_size=1577058304

swx1.__streams_pool_size=0

swx2.__streams_pool_size=0

*._client_enable_auto_unregister=TRUE

*.archive_lag_target=0

*.audit_file_dest=’/u02/app/oracle/admin/swx/adump’ ####### Change to Target Directory and Create teh Directory on OS ###########

*.audit_trail=’db’

*.cluster_database=TRUE ######## Set to FALSE ############

*.compatible=’11.2.0.4.0′ ######## Change to 12.1.0.2.0 ##########

*.control_files=’+datac1/swx/controlfile/current.346.853755085′,’+RECOC1/swx/controlfile/current.4208.853579009′ ########## Hash this #########

*.cpu_count=2

*.db_block_size=8192

*.db_create_file_dest=’+DATAC1′

*.db_create_online_log_dest_1=’+RECOC1′

*.db_domain=”

*.db_name=’swx’

*.db_recovery_file_dest=’+RECOC1′

*.db_recovery_file_dest_size=858993459200

*.diagnostic_dest=’/u02/app/oracle’ ########## Change to new target Oracle Base directory ############

*.dispatchers='(PROTOCOL=TCP) (SERVICE=swxXDB)’

swx2.instance_number=2

swx1.instance_number=1

*.java_jit_enabled=TRUE

*.log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=SWX’

*.log_archive_format=’%t_%s_%r.arc’

swx1.log_archive_format=’%t_%s_%r.arc’

swx2.log_archive_format=’%t_%s_%r.arc’

*.log_archive_max_processes=8

*.log_archive_min_succeed_dest=1

swx1.log_archive_trace=0

swx2.log_archive_trace=0

*.sga_max_size=17179869184

*.sga_target=17179869184

*.open_cursors=300

*.processes=1500

*.remote_listener=’exaprod-scan:1521′ ########## Change to SCAN listener of Target DB #########

*.remote_login_passwordfile=’exclusive’

*.resource_manager_cpu_allocation=48

*.resource_manager_plan=’DEFAULT_PLAN’

*.sessions=1655

*.standby_file_management=’AUTO’

swx2.thread=2

swx1.thread=1

swx1.undo_tablespace=’UNDOTBS1′

swx2.undo_tablespace=’UNDOTBS2′

*.instance_number=2

Save the parameters and save pfile as swx_target.ora

New pfile : swx_target.ora

———

swx1.__db_cache_size=9395240960

swx2.__db_cache_size=9395240960

swx1.__java_pool_size=33554432

swx2.__java_pool_size=33554432

swx1.__large_pool_size=67108864

swx2.__large_pool_size=67108864

swx2.__oracle_base=’/u02/app/oracle’#ORACLE_BASE set from environment

swx1.__oracle_base=’/u02/app/oracle’#ORACLE_BASE set from environment

swx1.__pga_aggregate_target=6006243328

swx2.__pga_aggregate_target=6006243328

swx1.__sga_target=11173625856

swx2.__sga_target=11173625856

swx1.__shared_io_pool_size=0

swx2.__shared_io_pool_size=0

swx1.__shared_pool_size=1577058304

swx2.__shared_pool_size=1577058304

swx1.__streams_pool_size=0

swx2.__streams_pool_size=0

*._client_enable_auto_unregister=TRUE

*.archive_lag_target=0

*.audit_file_dest=’/u03/app/oracle/admin/swx/adump’ ####### Change to Target Directory and Create teh Directory on OS ###########

*.audit_trail=’db’

*.cluster_database=FALSE ######## Set to FALSE ############

*.compatible=’12.1.0.2.0′ ######## Change to 12.1.0.2.0 ##########

#*.control_files=’+datac1/swx/controlfile/current.346.853755085′,’+RECOC1/swx/controlfile/current.4208.853579009′ ########## Hash this #########

*.cpu_count=2

*.db_block_size=8192

*.db_create_file_dest=’+DATAC1′

*.db_create_online_log_dest_1=’+RECOC1′

*.db_domain=”

*.db_name=’swx’

*.db_recovery_file_dest=’+RECOC1′

*.db_recovery_file_dest_size=858993459200

*.diagnostic_dest=’/u03/app/oracle’ ########## Change to new target Oracle Base directory ############

*.dispatchers='(PROTOCOL=TCP) (SERVICE=swxXDB)’

swx2.instance_number=2

swx1.instance_number=1

*.java_jit_enabled=TRUE

*.log_archive_dest_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=SWX’

*.log_archive_format=’%t_%s_%r.arc’

swx1.log_archive_format=’%t_%s_%r.arc’

swx2.log_archive_format=’%t_%s_%r.arc’

*.log_archive_max_processes=8

*.log_archive_min_succeed_dest=1

swx1.log_archive_trace=0

swx2.log_archive_trace=0

*.sga_max_size=17179869184

*.sga_target=17179869184

*.open_cursors=300

*.processes=1500

*.remote_listener=’qiibtst-scan:1521′ ########## Change to SCAN listener of Target DB #########

*.remote_login_passwordfile=’exclusive’

*.resource_manager_cpu_allocation=48

*.resource_manager_plan=’DEFAULT_PLAN’

*.sessions=1655

*.standby_file_management=’AUTO’

swx2.thread=2

swx1.thread=1

swx1.undo_tablespace=’UNDOTBS1′

swx2.undo_tablespace=’UNDOTBS2′

*.instance_number=2

4. Start the Instance on Target Database using swx_target.ora pfile

Make sure you export ORACLE_HOME and ORACLE_SID environment variables properly before beginning

SQL> startup nomount pfile=’/u03/swx_today/swx_source.ora’;

rman target /

RMAN> restore controlfile from ‘/u03/swx_today/control_file_SWX_853579011_20180215_4985_1’;

Starting restore at 18-MAR-18

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=1140 instance=swx device type=DISK

channel ORA_DISK_1: restoring control file

channel ORA_DISK_1: restore complete, elapsed time: 00:00:07

output file name=+RECOC1/SWX/CONTROLFILE/current.654.971100501

Finished restore at 18-MAR-18

RMAN> alter database mount;

RMAN> catalog start with ‘/u03/swx_today’;

Starting implicit crosscheck backup at 18-MAR-18

allocated channel: ORA_DISK_1

allocated channel: ORA_DISK_2

allocated channel: ORA_DISK_3

allocated channel: ORA_DISK_4

Crosschecked 158 objects

Finished implicit crosscheck backup at 18-MAR-18

Starting implicit crosscheck copy at 18-MAR-18

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

Crosschecked 4 objects

Finished implicit crosscheck copy at 18-MAR-18

searching for all files in the recovery area

cataloging files…

no files cataloged

searching for all files that match the pattern /u03/swx_today

List of Files Unknown to the Database

=====================================

File Name: /u03/swx_today/ARCH_SWX_ALL_4979_968178229

File Name: /u03/swx_today/control_file_SWX_853579011_20180215_4985_1

File Name: /u03/swx_today/SWX_FULL_4975_15_02_2018

File Name: /u03/swx_today/ARCH_SWX_ALL_4981_968178230

File Name: /u03/swx_today/ARCH_SWX_ALL_4978_968178229

File Name: /u03/swx_today/ARCH_SWX_ALL_4982_968178255

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-02

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-00

File Name: /u03/swx_today/SWX_FULL_4974_15_02_2018

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-01

File Name: /u03/swx_today/swx_source.ora

File Name: /u03/swx_today/SWX_FULL_4976_15_02_2018

File Name: /u03/swx_today/ARCH_SWX_ALL_4980_968178230

File Name: /u03/swx_today/ARCH_SWX_ALL_4983_968178258

Do you really want to catalog the above files (enter YES or NO)? YES

cataloging files…

cataloging done

List of Cataloged Files

=======================

File Name: /u03/swx_today/ARCH_SWX_ALL_4979_968178229

File Name: /u03/swx_today/control_file_SWX_853579011_20180215_4985_1

File Name: /u03/swx_today/SWX_FULL_4975_15_02_2018

File Name: /u03/swx_today/ARCH_SWX_ALL_4981_968178230

File Name: /u03/swx_today/ARCH_SWX_ALL_4978_968178229

File Name: /u03/swx_today/ARCH_SWX_ALL_4982_968178255

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-02

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-00

File Name: /u03/swx_today/SWX_FULL_4974_15_02_2018

File Name: /u03/swx_today/autobackup_control_file_c-133836929-20180215-01

File Name: /u03/swx_today/SWX_FULL_4976_15_02_2018

File Name: /u03/swx_today/ARCH_SWX_ALL_4980_968178230

File Name: /u03/swx_today/ARCH_SWX_ALL_4983_968178258

List of Files Which Were Not Cataloged

=======================================

File Name: /u03/swx_today/swx_source.ora

RMAN-07517: Reason: The file header is corrupted

RMAN> restore database;

Starting restore at 18-MAR-18

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: SID=26 instance=swx device type=SBT_TAPE

channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 7.1.0.0

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00003 to +DATAC1/swx/datafile/undotbs1.281.853578943

channel ORA_DISK_1: restoring datafile 00004 to +RECOC1/swx/datafile/audit_tbs.2213.871200459

channel ORA_DISK_1: reading from backup piece /u03/swx_today/SWX_FULL_4975_15_02_2018

channel ORA_DISK_2: starting datafile backup set restore

channel ORA_DISK_2: specifying datafile(s) to restore from backup set

channel ORA_DISK_2: restoring datafile 00001 to +DATAC1/swx/datafile/system.285.853578943

channel ORA_DISK_2: restoring datafile 00002 to +DATAC1/swx/datafile/sysaux.278.853578943

channel ORA_DISK_2: reading from backup piece /u03/swx_today/SWX_FULL_4976_15_02_2018

channel ORA_DISK_3: starting datafile backup set restore

channel ORA_DISK_3: specifying datafile(s) to restore from backup set

channel ORA_DISK_3: restoring datafile 00005 to +DATAC1/swx/datafile/undotbs2.351.853579027

channel ORA_DISK_3: restoring datafile 00006 to +DATAC1/swx/datafile/swx.345.853583217

channel ORA_DISK_3: reading from backup piece /u03/swx_today/SWX_FULL_4974_15_02_2018

channel ORA_DISK_1: piece handle=/u03/swx_today/SWX_FULL_4975_15_02_2018 tag=SWX_DAILY_FULL_DB_BKP

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:15

channel ORA_DISK_2: piece handle=/u03/swx_today/SWX_FULL_4976_15_02_2018 tag=SWX_DAILY_FULL_DB_BKP

channel ORA_DISK_2: restored backup piece 1

channel ORA_DISK_2: restore complete, elapsed time: 00:00:15

channel ORA_DISK_3: piece handle=/u03/swx_today/SWX_FULL_4974_15_02_2018 tag=SWX_DAILY_FULL_DB_BKP

channel ORA_DISK_3: restored backup piece 1

channel ORA_DISK_3: restore complete, elapsed time: 00:04:45

Finished restore at 18-MAR-18

RMAN> recover database;

Starting recover at 18-MAR-18

using channel ORA_DISK_1

using channel ORA_DISK_2

using channel ORA_DISK_3

using channel ORA_DISK_4

using channel ORA_SBT_TAPE_1

starting media recovery

channel ORA_DISK_1: starting archived log restore to default destination

channel ORA_DISK_1: restoring archived log

archived log thread=2 sequence=48891

channel ORA_DISK_1: restoring archived log

archived log thread=1 sequence=43454

channel ORA_DISK_1: restoring archived log

archived log thread=2 sequence=48892

channel ORA_DISK_1: restoring archived log

archived log thread=1 sequence=43455

channel ORA_DISK_1: reading from backup piece /u03/swx_today/ARCH_SWX_ALL_4983_968178258

channel ORA_DISK_1: piece handle=/u03/swx_today/ARCH_SWX_ALL_4983_968178258 tag=SWX_ARCH_LOG_BKP

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:01

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_1_seq_43454.645.971100923 thread=1 sequence=43454

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_2_seq_48891.642.971100923 thread=2 sequence=48891

channel default: deleting archived log(s)

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_2_seq_48891.642.971100923 RECID=238985 STAMP=971100923

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_2_seq_48892.639.971100923 thread=2 sequence=48892

channel default: deleting archived log(s)

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_1_seq_43454.645.971100923 RECID=238987 STAMP=971100923

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_1_seq_43455.648.971100923 thread=1 sequence=43455

channel default: deleting archived log(s)

archived log file name=+RECOC1/SWX/ARCHIVELOG/2018_03_18/thread_1_seq_43455.648.971100923 RECID=238988 STAMP=971100923

unable to find archived log

archived log thread=1 sequence=43456

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of recover command at 03/18/2018 14:15:25

RMAN-06054: media recovery requesting unknown archived log for thread 1 with sequence 43456 and starting SCN of 21582106266

RMAN> alter database open resetlogs;

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-00601: fatal error in recovery manager

RMAN-03004: fatal error during execution of command

ORA-03114: not connected to ORACLE

RMAN-03002: failure of sql statement command at 03/18/2018 14:15:42

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00904: “I”.”UNUSABLEBEGINNING#”: invalid identifier

Process ID: 136136

Session ID: 1137 Serial number: 14577

sqlplus “/as sysdba”

Change control file information with new location after restore in pfile and start database again +RECOC1/SWX/CONTROLFILE/current.654.971100501

Also change CLUSTER_DATABASE to false in pfile

If you get any error like below error then copy the file to the dbs folder as init.ora

ORA-01078: failure in processing system parameters

LRM-00109: could not open parameter file ‘/u03/app/oracle/product/12.1.0.2/dbhome_1/dbs/initswx.ora’

$ cp -p /u03/swx_today/swx_source.ora /u03/app/oracle/product/12.1.0.2/dbhome_1/dbs/initswx.ora

sqlplus “/as sysdba”

SQL> startup upgrade;

$ cd $ORACLE_HOME/rdbms/admin

$ORACLE_HOME/perl/bin/perl catctl.pl -n 4 catupgrd.sql

Argument list for [catctl.pl]

SQL Process Count n = 4

SQL PDB Process Count N = 0

Input Directory d = 0

Phase Logging Table t = 0

Log Dir l = 0

Script s = 0

Serial Run S = 0

Upgrade Mode active M = 0

Start Phase p = 0

End Phase P = 0

Log Id i = 0

Run in c = 0

Do not run in C = 0

Echo OFF e = 1

No Post Upgrade x = 0

Reverse Order r = 0

Open Mode Normal o = 0

Debug catcon.pm z = 0

Debug catctl.pl Z = 0

Display Phases y = 0

Child Process I = 0

catctl.pl version: 12.1.0.2.0

Oracle Base = /u03/app/oracle

Analyzing file catupgrd.sql

Log files in /u03/app/oracle/product/12.1.0.2/dbhome_1/rdbms/admin

catcon: ALL catcon-related output will be written to catupgrd_catcon_227787.lst

catcon: See catupgrd*.log files for output generated by scripts

catcon: See catupgrd_*.lst files for spool files, if any

Number of Cpus = 2

SQL Process Count = 4

——————————————————

Phases [0-73] Start Time:[2018_03_18 14:39:13]

——————————————————

Serial Phase #: 0 Files: 1 Time: 105s

Serial Phase #: 1 Files: 5 Time: 22s

Restart Phase #: 2 Files: 1 Time: 0s

Parallel Phase #: 3 Files: 18 Time: 4s

Restart Phase #: 4 Files: 1 Time: 0s

Serial Phase #: 5 Files: 5 Time: 10s

Serial Phase #: 6 Files: 1 Time: 9s

Serial Phase #: 7 Files: 4 Time: 4s

Restart Phase #: 8 Files: 1 Time: 0s

Parallel Phase #: 9 Files: 62 Time: 12s

Restart Phase #:10 Files: 1 Time: 0s

Serial Phase #:11 Files: 1 Time: 8s

Restart Phase #:12 Files: 1 Time: 0s

Parallel Phase #:13 Files: 91 Time: 4s

Restart Phase #:14 Files: 1 Time: 0s

Parallel Phase #:15 Files: 111 Time: 8s

Restart Phase #:16 Files: 1 Time: 0s

Serial Phase #:17 Files: 3 Time: 0s

Restart Phase #:18 Files: 1 Time: 0s

Parallel Phase #:19 Files: 32 Time: 8s

Restart Phase #:20 Files: 1 Time: 0s

Serial Phase #:21 Files: 3 Time: 4s

Restart Phase #:22 Files: 1 Time: 0s

Parallel Phase #:23 Files: 23 Time: 43s

Restart Phase #:24 Files: 1 Time: 0s

Parallel Phase #:25 Files: 11 Time: 17s

Restart Phase #:26 Files: 1 Time: 0s

Serial Phase #:27 Files: 1 Time: 0s

Restart Phase #:28 Files: 1 Time: 0s

Serial Phase #:30 Files: 1 Time: 0s

Serial Phase #:31 Files: 257 Time: 12s

Serial Phase #:32 Files: 1 Time: 0s

Restart Phase #:33 Files: 1 Time: 0s

Serial Phase #:34 Files: 1 Time: 3s

Restart Phase #:35 Files: 1 Time: 0s

Restart Phase #:36 Files: 1 Time: 0s

Serial Phase #:37 Files: 4 Time: 31s

Restart Phase #:38 Files: 1 Time: 0s

Parallel Phase #:39 Files: 13 Time: 27s

Restart Phase #:40 Files: 1 Time: 0s

Parallel Phase #:41 Files: 10 Time: 3s

Restart Phase #:42 Files: 1 Time: 0s

Serial Phase #:43 Files: 1 Time: 4s

Restart Phase #:44 Files: 1 Time: 0s

Serial Phase #:45 Files: 1 Time: 3s

Serial Phase #:46 Files: 1 Time: 1s

Restart Phase #:47 Files: 1 Time: 0s

Serial Phase #:48 Files: 1 Time: 82s

Restart Phase #:49 Files: 1 Time: 0s

Serial Phase #:50 Files: 1 Time: 22s

Restart Phase #:51 Files: 1 Time: 0s

Serial Phase #:52 Files: 1 Time: 16s

Restart Phase #:53 Files: 1 Time: 0s

Serial Phase #:54 Files: 1 Time: 192s

Restart Phase #:55 Files: 1 Time: 0s

Serial Phase #:56 Files: 1 Time: 37s

Restart Phase #:57 Files: 1 Time: 0s

Serial Phase #:58 Files: 1 Time: 69s

Restart Phase #:59 Files: 1 Time: 0s

Serial Phase #:60 Files: 1 Time: 295s

Restart Phase #:61 Files: 1 Time: 0s

Serial Phase #:62 Files: 1 Time: 24s

Restart Phase #:63 Files: 1 Time: 0s

Serial Phase #:64 Files: 1 Time: 0s

Serial Phase #:65 Files: 1 Time: 11s

Serial Phase #:66 Files: 1 Time: 25s

Serial Phase #:67 Files: 1 Time: 1s

Serial Phase #:68 Files: 1 Time: 0s

Serial Phase #:69 Files: 1 Time: 27s

——————————————————

Phases [0-73] End Time:[2018_03_18 14:58:16]

——————————————————

Grand Total Time: 1145s

*** WARNING: ERRORS FOUND DURING UPGRADE ***

Due to errors found during the upgrade process, the post

upgrade actions in catuppst.sql have not been automatically run.

*** THEREFORE THE DATABASE UPGRADE IS NOT YET COMPLETE ***

1. Evaluate the errors found in the upgrade logs

and determine the proper action.

2. Execute the post upgrade script as described in Chapter 3

of the Database Upgrade Guide.

REASON:

catuppst.sql unable to run in Database: swx Id: 0

ERRORS FOUND: during upgrade CATCTL ERROR COUNT=5

——————————————————

Identifier XDB 18-03-18 02:47:25 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [ORA-01917: user or role ‘ANONYMOUS’ does not exist ORA-06512: at line 7

ORA-06512: at line 15

]

STATEMENT = [declare

already_revoked exception;

pragma exception_init(already_revoked,-01927);

procedure revoke_inherit_privileges(user in varchar2) as

begin

execute immediate ‘revoke inherit privileges on user ‘||

dbms_assert.enquote_name(user)||’ from public’;

exception

when already_revoked then null;

end;

begin

revoke_inherit_privileges(‘xdb’);

revoke_inherit_privileges(‘anonymous’);

end;]

——————————————————

——————————————————

Identifier XDB 18-03-18 02:47:25 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [ORA-06512: at line 7 ORA-06512: at line 15

]

STATEMENT = [as above]

——————————————————

——————————————————

Identifier XDB 18-03-18 02:47:25 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [ORA-06512: at line 15]

STATEMENT = [as above]

——————————————————

——————————————————

Identifier ORDIM 18-03-18 02:50:44 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [ORA-20000: Oracle XML Database component not valid. Oracle XML Database must be installed and valid prior to Oracle Multimedia install, upgrade, downgrade, or patch.

ORA-06512: at line 3

]

STATEMENT = [begin

IF dbms_registry.is_valid(‘XDB’,dbms_registry.release_version) != 1 THEN

raise_application_error(-20000,

‘Oracle XML Database component not valid. ‘||

‘Oracle XML Database must be installed and valid prior to Oracle Multimedia install, upgrade, downgrade, or patch.’);

END IF;

end;]

——————————————————

——————————————————

Identifier ORDIM 18-03-18 02:50:44 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [installed and valid prior to Oracle Multimedia install, upgrade, downgrade, or patch.

ORA-06512: at line 3

]

STATEMENT = [as above]

——————————————————

——————————————————

Identifier ORDIM 18-03-18 02:50:44 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [patch. ORA-06512: at line 3

]

STATEMENT = [as above]

——————————————————

——————————————————

Identifier ORDIM 18-03-18 02:50:44 Script = /u03/app/oracle/product/12.1.0.2/dbhome_

ERROR = [ORA-06512: at line 3]

STATEMENT = [as above]

——————————————————

LOG FILES: (catupgrd*.log)

Upgrade Summary Report Located in:

/u03/app/oracle/product/12.1.0.2/dbhome_1/cfgtoollogs/swx/upgrade/upg_summary.log

Grand Total Upgrade Time: [0d:0h:19m:5s]

sqlplus “/as sysdba”

SQL> @utlrp.sql

SQL> select * from dba_registry;

Verify all components are valid

OWB OWB 11.2.0.4.0 VALID 18-MAR-2018 15:02:31 SERVER SYS OWBSYS DBMS_OWB.VALIDATE

EM Oracle Enterprise Manager 11.2.0.4.0 VALID 24-AUG-2013 11:58:27 SERVER SYS SYSMAN

AMD OLAP Catalog 11.2.0.4.0 OPTION OFF 18-MAR-2018 14:46:55 SERVER SYS OLAPSYS CWM2_OLAP_INSTALLER.VALIDATE_CWM2_INSTALL

SDO Spatial 12.1.0.2.0 VALID 18-MAR-2018 15:02:30 SERVER SYS MDSYS VALIDATE_SDO MDDATA,MDSYS,SPATIAL_CSW_ADMIN_USR,SPATIAL_WFS_ADMIN_USR

ORDIM Oracle Multimedia 12.1.0.2.0 VALID 18-MAR-2018 15:02:29 SERVER SYS ORDSYS VALIDATE_ORDIM ORDDATA,ORDPLUGINS,SI_INFORMTN_SCHEMA

XDB Oracle XML Database 12.1.0.2.0 VALID 18-MAR-2018 15:02:28 SERVER SYS XDB DBMS_REGXDB.VALIDATEXDB XS$NULL

CONTEXT Oracle Text 12.1.0.2.0 VALID 18-MAR-2018 15:02:27 SERVER SYS CTXSYS VALIDATE_CONTEXT

OWM Oracle Workspace Manager 12.1.0.2.0 VALID 18-MAR-2018 15:02:25 SERVER SYS WMSYS VALIDATE_OWM

CATALOG Oracle Database Catalog Views 12.1.0.2.0 VALID 18-MAR-2018 15:02:23 SERVER SYS SYS DBMS_REGISTRY_SYS.VALIDATE_CATALOG

CATPROC Oracle Database Packages and Types 12.1.0.2.0 VALID 18-MAR-2018 15:02:23 SERVER SYS SYS DBMS_REGISTRY_SYS.VALIDATE_CATPROC APPQOSSYS,DBSNMP,DIP,GSMADMIN_INTERNAL,ORACLE_OCM,OUTLN,SYSTEM

JAVAVM JServer JAVA Virtual Machine 12.1.0.2.0 VALID 18-MAR-2018 15:02:26 SERVER SYS SYS INITJVMAUX.VALIDATE_JAVAVM

COMP_ID COMP_NAME VERSION STATUS MODIFIED NAMESPACE CONTROL SCHEMA PROCEDURE STARTUP PARENT_ID OTHER_SCHEMAS

—————————— ————————————————————————————————————————————————————————————————————————————————————— —————————— ———– ——————– —————————— ——————————————————————————————————————————– ——————————————————————————————————————————– ————————————————————- ——– —————————— —————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————-

XML Oracle XDK 12.1.0.2.0 VALID 18-MAR-2018 15:02:27 SERVER SYS SYS XMLVALIDATE

CATJAVA Oracle Database Java Packages 12.1.0.2.0 VALID 18-MAR-2018 15:02:27 SERVER SYS SYS DBMS_REGISTRY_SYS.VALIDATE_CATJAVA

APS OLAP Analytic Workspace 12.1.0.2.0 VALID 18-MAR-2018 15:02:29 SERVER SYS SYS APS_VALIDATE

XOQ Oracle OLAP API 12.1.0.2.0 VALID 18-MAR-2018 15:02:30 SERVER SYS SYS XOQ_VALIDATE

RAC Oracle Real Application Clusters 12.1.0.2.0 VALID 18-MAR-2018 15:02:31 SERVER SYS SYS DBMS_CLUSTDB.VALIDATE

16 rows selected.

SQL>

Drop those Indexes and see the Magical Unicorn of Exadata do it’s Magic !

Now that I have got your attention with my sales pitch. Let me quote a sales pitch we have heard quite often from Exadata Sales rep “Drop the indexes on Exadata ; It is a magical unicorn and it will guarantee performance even without them”  Well, 50 years of computer science begs to differ. This Urban legend has to die for once and all. Indexes are computer science constructs and one of the basics of search optimization.

Hardware like Exadata cannot always be a solution to a Software problem. Exadata though a very well engineered system design to run Oracle database it cannot always compensate for wrong SQL logic. Recently we encountered a issue where a query running in a constant loop was doing lot of user I/O and hence consuming way too much CPU. We decided to dig a little further by getting the execution plan for the query.

Query

[code language=”sql”]
SELECT  t.*                          ,
        e.RETURNCODEMSG              ,
        r.RETURNCODE            AS R_RC  ,
        R.RETURNCODEDESC        AS R_RCD ,
        R.RETURNCODEPOVIDER     AS R_RCP ,
        R.RETURNCODEPOVIDERDESC AS R_RCPD
FROM
        (
                SELECT  *
                FROM
                        (
                                SELECT  ROWNUM rown ,
                                        t1.*
                                FROM
                                        (
                                                SELECT  TRAN.*
                                                FROM    transactions TRAN,
                                                        TRANSREVERSED RTRAN
                                                WHERE   RTRAN.LOGREFID(+)    =TRAN.ID
                                                        AND TRAN.UTIMESTAMP >= to_date(’26/10/2017′, ‘dd/MM/yyyy’)
                                                        AND TRAN.UTIMESTAMP <= to_date(’26/10/2017 23:59:59′,’dd/MM/yyyy HH24:MI:ss’)                                                         AND 1                =1                                                         AND TRAN.SERVICENAME =’STRINGTOSEARCH’                                                         AND TRAN.returncode !=’000004′                                                 ORDER BY TRAN.UTIMESTAMP DESC                                         )                                         t1                         )                         q1                 WHERE   q1.rown    >=1
                        AND q1.rown
        )
        t               ,
        TRANSREVERSED r ,
        esbreturncode e
WHERE   r.LOGREFID(+)      =t.ID
        AND e.returncode(+)=t.returncode
ORDER BY t.UTIMESTAMP DESC

[/code]

Now If you analyze the Query, it’s a complex one with Joins on two tables selecting data from table TRANSACTIONS and then doing an inner join on a column in table TRANSREVERSED. The predicate LOGREFID is used for this inner join. In the Explain plan we saw a full table scan being done on the TRANSREVERSED table. Even though the lookup table is small the result set via which it has to be compared is quite huge.

Execution Plan Pre-Index Ethos

[code language=”sql”]

<hr />
<p class="x_MsoNormal">| Id  | Operation                          | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |</p>
<p class="x_MsoNormal">——————————————————————————————————————–</p>
<p class="x_MsoNormal">|   0 | SELECT STATEMENT                   |               |    39 | 84786 | 17007   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|   1 |  SORT ORDER BY                     |               |    39 | 84786 | 17007   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|*  2 |   HASH JOIN OUTER                  |               |    39 | 84786 | 17006   (1)| 00:03:25 |       |       |</p>
<p class="x_MsoNormal">|*  3 |    HASH JOIN RIGHT OUTER           |               |    39 | 80106 |  9053   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   4 |     TABLE ACCESS STORAGE FULL      | ESBRETURNCODE |    76 |  2964 |     4   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|*  5 |     VIEW                           |               |    39 | 78585 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   6 |      COUNT                         |               |       |       |            |          |       |       |</p>
<p class="x_MsoNormal">|   7 |       VIEW                         |               |    39 | 78078 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|   8 |        SORT ORDER BY               |               |    39 | 12558 |  9049   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|*  9 |         HASH JOIN OUTER            |               |    39 | 12558 |  9048   (1)| 00:01:49 |       |       |</p>
<p class="x_MsoNormal">|  10 |          PARTITION RANGE SINGLE    |               |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|* 11 |           TABLE ACCESS STORAGE FULL| TRANSACTIONS  |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|  12 |          PARTITION RANGE ALL       |               | 16278 |   429K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  13 |           TABLE ACCESS STORAGE FULL| TRANSREVERSED | 16278 |   429K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  14 |    PARTITION RANGE ALL             |               | 16278 |  1907K|  7953   (1)| 00:01:36 |     1 |1048575|</p>
<p class="x_MsoNormal">|  15 |     TABLE ACCESS STORAGE FULL      | TRANSREVERSED | 16278 |  1907K|  7953   (1)| 00:01:36 |     1 |1048575|</p>

[/code]

 

After trying a couple of silly scenarios for tuning it like moving the Table to the flash cache permanently with FLASH_CACHE=KEEP option and trying to ask the developer to reduce the time span for which it searches the first table via field UTIMESTAMP (developers will never admit their code is crappy). Then we went back to the hammer and nails approach. “Why not create an index on the field instead of trying to put a nail through with a Jedi lightsaber !!”

So after a few hours of excersizing our grey cells we created a non-unique b*tree index on the field LOGREFID and rebuild the table stats. The performance of the query immediately saw improvement from consuming a helluva lot user I/O to almost nothing.  In case of heavy reads a missing index is almost always the silver bullet.

Execution Plan Post-Index Ethos

[code language=”sql”]
<p class="x_MsoNormal">———————————————————————————————————————-</p>
<p class="x_MsoNormal">| Id  | Operation                            | Name          | Rows  | Bytes | Cost (%CPU)| Time     | Pstart| Pstop |</p>
<p class="x_MsoNormal">———————————————————————————————————————-</p>
<p class="x_MsoNormal">|   0 | SELECT STATEMENT                     |               |    39 | 84786 |  1174   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|   1 |  SORT ORDER BY                       |               |    39 | 84786 |  1174   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|   2 |   NESTED LOOPS OUTER                 |               |    39 | 84786 |  1173   (1)| 00:00:15 |       |       |</p>
<p class="x_MsoNormal">|*  3 |    HASH JOIN RIGHT OUTER             |               |    39 | 80106 |  1127   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   4 |     TABLE ACCESS STORAGE FULL        | ESBRETURNCODE |    76 |  2964 |     4   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|*  5 |     VIEW                             |               |    39 | 78585 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   6 |      COUNT                           |               |       |       |            |          |       |       |</p>
<p class="x_MsoNormal">|   7 |       VIEW                           |               |    39 | 78078 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|   8 |        SORT ORDER BY                 |               |    39 | 12558 |  1123   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|*  9 |         HASH JOIN OUTER              |               |    39 | 12558 |  1122   (1)| 00:00:14 |       |       |</p>
<p class="x_MsoNormal">|  10 |          PARTITION RANGE SINGLE      |               |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|* 11 |           TABLE ACCESS STORAGE FULL  | TRANSACTIONS  |    39 | 11505 |  1095   (1)| 00:00:14 |    86 |    86 |</p>
<p class="x_MsoNormal">|  12 |          INDEX STORAGE FAST FULL SCAN| LOGREFID_IDX  | 16278 |   429K|    27   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">|  13 |    TABLE ACCESS BY GLOBAL INDEX ROWID| TRANSREVERSED |     1 |   120 |     2   (0)| 00:00:01 | ROWID | ROWID |</p>
<p class="x_MsoNormal">|* 14 |     INDEX RANGE SCAN                 | LOGREFID_IDX  |     1 |       |     1   (0)| 00:00:01 |       |       |</p>
<p class="x_MsoNormal">[/code]

Lets look at more graphical representation  from OEM ;  A picture speaks louder than a thousand words.

image001-2

/* The left-side mountain spikes are pre-index ethos and the right side hills are post. Self explanatory */

Heavy reads with high cost screams Index missing. Now this does not mean you go and start creating indexes in every SQL query you see a bit of reads. The Crux of the matter is that before you go ahead and blame your storage admin or disk subsystem or get new hardware. Doing a bit of the basics like looking at AWR reports, generating plans and applying good ol’ DBA intuition, it can do more good than throwing hardware at it. Hardware will never guarantee solution to a software problem.

P.S: We had the above issue on a densely packed Exadata X4-2 Quarter rack sharing 20+ instances. Disk I/O is the number one performance killer on a consolidated platform. Waiting on Disk I/O will always lead to higher CPU consumption.

 

 

Oracle 12c Transparent Data Encryption (TDE) on RAC Database with Physical Standby Database : Multiple Databases in Single Oracle Home

Oracle 12c TDE on RAC Database with Physical Standby RAC. The Oracle Database Home has Multiple Instances, due to which we have to use parameter $ORACLE_UNQNAME for ENCRYPTION_WALLET_LOCATION in sqlnet.ora file

In Oracle 12c ALTER SYSTEM and orapki has been deprecated and the correct method is to use ADMINISTER KEY MANAGEMENT commands. Many of the statements from the ALTER SYSTEM SQL statement correspond to the ADMINISTER KEY MANAGEMENT statement.

1. Create folder for wallets, since this is RAC cluster database ,it is better to have a shared filesystem. In our case it is ACFS
mkdir -p /ACFS/wallets/primeprd
mkdir -p /ACFS/wallets/onlnprd
chown -R oracle:oinstall /ACFS/wallets/primeprd
chown -R oracle:oinstall /ACFS/wallets/onlnprd
2. Create sqlnet.ora file with encryption wallet location, do on both nodes with oracle user
ENCRYPTION_WALLET_LOCATION =
(SOURCE = (METHOD = FILE) (METHOD_DATA =
(DIRECTORY=/ACFS/wallets/$ORACLE_UNQNAME/)))
3. Set DB Unique Name for the databases and verfiy the encryption wallet location, do on both nodes with oracle user

Also it is better to set the ORACLE_UNQNAME in the user profile

export ORACLE_UNQNAME=`$ORACLE_HOME/bin/srvctl config database |grep -w ${ORACLE_SID%?}`
echo $ORACLE_UNQNAME
srvctl setenv database -d primeprd -T “ORACLE_UNQNAME=primeprd”
select * from v$encryption_wallet;

export ORACLE_UNQNAME=`$ORACLE_HOME/bin/srvctl config database |grep -w ${ORACLE_SID%?}`
echo $ORACLE_UNQNAME
srvctl setenv database -d onlnprd -T “ORACLE_UNQNAME=onlnprd”
select * from v$encryption_wallet;

4. At Dr side, with Oracle user in DB home directory
mkdir -p /ACFS/wallets/primedr
mkdir -p /ACFS/wallets/onlndr
chown -R oracle:oinstall /ACFS/wallets/onlndr
chown -R oracle:oinstall /ACFS/wallets/primedr

Create sqlnet.ora file in DR side with encryption wallet location, do on both nodes with oracle user

ENCRYPTION_WALLET_LOCATION =
(SOURCE = (METHOD = FILE) (METHOD_DATA =
(DIRECTORY=/ACFS/wallets/$ORACLE_UNQNAME/)))

export ORACLE_UNQNAME=`$ORACLE_HOME/bin/srvctl config database |grep -w ${ORACLE_SID%?}`
echo $ORACLE_UNQNAME
srvctl setenv database -d onlndr -T “ORACLE_UNQNAME=onlndr”

export ORACLE_UNQNAME=`$ORACLE_HOME/bin/srvctl config database |grep -w ${ORACLE_SID%?}`
echo $ORACLE_UNQNAME
srvctl setenv database -d primedr -T “ORACLE_UNQNAME=primedr”

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED
0
SQL> administer key management create keystore ‘/ACFS/wallet/primeprd/’ identified by “Abc1234$#”;

keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
CLOSED UNKNOWN SINGLE UNDEFINED
0
SQL> administer key management set keystore open identified by Abc1234$#;

keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
OPEN_NO_MASTER_KEY PASSWORD SINGLE UNDEFINED
0
SQL> administer key management create key identified by Abc1234$# with backup;

keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
OPEN_NO_MASTER_KEY PASSWORD SINGLE UNDEFINED
0
SQL> select key_id from v$encryption_keys;

KEY_ID
——————————————————————————
AcVlvDzDNE8iv/rD38jpCtYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

SQL> administer key management use key ‘AcVlvDzDNE8iv/rD38jpCtYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA’ identified by Abc1234$# with backup;

keystore altered.

SQL> select key_id,activation_time from v$encryption_keys;

KEY_ID
——————————————————————————
ACTIVATION_TIME
—————————————————————————
AcVlvDzDNE8iv/rD38jpCtYAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
23-NOV-16 09.53.46.485930 AM +00:00
SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
OPEN PASSWORD SINGLE NO
0

primeprd1::12c–>srvctl stop database -d primeprd
primeprd1::12c–>srvctl start database -d primeprd
primeprd1::12c–>sqld

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
CLOSED UNKNOWN SINGLE UNDEFINED
0
SQL> administer key management set keystore open identified by Abc1234$#;

keystore altered.

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
OPEN PASSWORD SINGLE NO
0
— Create Auto Login for the Keystore —

SQL> ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE ‘/ACFS/wallet/primeprd/’ identified by Abc1234$#;

keystore altered.

primeprd1::12c–>srvctl stop database -d primeprd
primeprd1::12c–>srvctl start database -d primeprd
primeprd1::12c–>sqld

SQL*Plus: Release 12.1.0.2.0 Production on Wed Nov 23 13:13:30 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallet/primeprd/
OPEN AUTOLOGIN SINGLE NO
0
Copy the Encryption key Files to Standby Site as “Oracle” user

primeprd2::12c–>cd /ACFS/wallet/primeprd/
primeprd2::12c–>scp -r * oracle@exadrdbadm01:/ACFS/wallets/primedr

The authenticity of host ‘exadrdbadm01 (10.5.238.11)’ can’t be established.
RSA key fingerprint is c8:c0:69:80:1c:c8:ea:e3:6c:46:6b:c1:05:86:aa:c9.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added ‘exadrdbadm01,10.5.238.11’ (RSA) to the list of known hosts.
oracle@exadrdbadm01’s password:
cwallet.sso 100% 3893 3.8KB/s 00:00
ewallet_2016112309525312.p12 100% 2408 2.4KB/s 00:00
ewallet_2016112309534645.p12 100% 3656 3.6KB/s 00:00
ewallet.p12 100% 3848 3.8KB/s 00:00

— At DR Side on Both Nodes–
SQL> select * from v$encryption_wallet;
WRL_TYPE
——————–
WRL_PARAMETER
—————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————
STATUS WALLET_TYPE WALLET_OR FULLY_BAC CON_ID
—————————— ——————– ——— ——— ———-
FILE
/u03/app/oracle/admin/primedr/wallet
NOT_AVAILABLE UNKNOWN SINGLE UNDEFINED
/home/oracle::12c–>cd /u03/app/oracle/product/12.1.0.2/teststandby/network/admin/
/u03/app/oracle/product/12.1.0.2/teststandby/network/admin::12c–>vi sqlnet.ora
ENCRYPTION_WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /ACFS/wallets/primedr/$ORACLE_UNQNAME/)))

/u03/app/oracle/product/12.1.0.2/teststandby/network/admin::12c–>sqld

SQL*Plus: Release 12.1.0.2.0 Production on Wed Nov 23 13:27:22 2016

Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> select * from v$encryption_wallet;

WRL_TYPE
——————–
WRL_PARAMETER
——————————————————————————–
STATUS WALLET_TYPE WALLET_OR FULLY_BAC
—————————— ——————– ——— ———
CON_ID
———-
FILE
/ACFS/wallets/primedr/
OPEN AUTOLOGIN SINGLE NO
0
SQL> exit

 

5. Create Encrypted Tablespace
CREATE TABLESPACE encrypt_ts
DATAFILE ‘+DATAC1’ SIZE 250M
ENCRYPTION USING ‘AES256’
DEFAULT STORAGE (ENCRYPT);

Check ASM Diskgroup Space and Directory Size

The script below can be used to check the Disk Group Space Free and also Check Directory sizes for each Disk Group

The script is written by somebody at Pythian but i cannot re-collect the original link to the blog.

eg:
./asmcmd_du.sh

DiskGroup Total_MB Free_MB % Free
——— ——– ——- ——
DATAC1 15962112 11215880 70
DBFS_DG 415296 403740 97
RECOC1 3996000 3460272 86

./asmcmd_du.sh DATAC1

DiskGroup Total_MB Free_MB % Free
——— ——– ——- ——
DATAC1 15962112 10155732 63

DATAC1 subdirectories size

Subdir Used MB Mirror MB
—— ——- ———
CARATST/ 55646 111356
ECCTST/ 174912 349856
—— ——- ———
Total 2799978 5600788

Script Below :

[code language=”bash”]
#!/bin/bash
# Shadab Mohammad — 2016
#
# – If no parameter specified, show a du of each DiskGroup
# – If a parameter, print a du of each subdirectory
#

D=$1

#
# Colored thresholds (Red, Yellow, Green)
#
CRITICAL=90
WARNING=75

#
# Set the ASM env
#
ORACLE_SID=`ps -ef | grep pmon | grep asm | awk ‘{print $NF}’ | sed s’/asm_pmon_//’ | egrep "^[+]"`
export ORAENV_ASK=NO
. oraenv > /dev/null 2>&1

#
# A quick list of what is running on the server
#
ps -ef | grep pmon | grep -v grep | awk ‘{print $NF}’ | sed s’/.*_pmon_//’ | egrep "^([+]|[Aa-Zz])" | sort | awk -v H="`hostname -s`" ” | sed s’/, $//’

#
# Manage parameters
#
if [[ -z $D ]]
then # No directory provided, will check all the DG
DG=`asmcmd lsdg | grep -v State | awk ‘{print $NF}’ | sed s’/\///’`
SUBDIR="No" # Do not show the subdirectories details if no directory is specified
else
DG=`echo $D | sed s’/\/.*$//g’`
fi

#
# A header
#
printf "\n%25s%16s%16s%14s" "DiskGroup" "Total_MB" "Free_MB" "% Free"
printf "\n%25s%16s%16s%14s\n" "———" "——–" "——-" "——"

#
# Show DG info
#
for X in ${DG}
do
asmcmd lsdg ${X} | tail -1 |\
awk -v DG="$X" -v W="$WARNING" -v C="$CRITICAL" ‘\
BEGIN \
{COLOR_BEGIN = "\033[1;" ;
COLOR_END = "\033[m" ;
RED = COLOR_BEGIN"31m" ;
GREEN = COLOR_BEGIN"32m" ;
YELLOW = COLOR_BEGIN"33m" ;
COLOR = GREEN ;
}
{ FREE = sprintf("%12d", $8/$7*100) ;
if ((100-FREE) > W) {COLOR=YELLOW ;}
if ((100-FREE) > C) {COLOR=RED ;}
printf("%25s%16s%16s%s\n", DG, $7, $8, COLOR FREE COLOR_END) ; }’
done
printf "\n"

#
# Subdirs info
#
if [ -z ${SUBDIR} ]
then
(for DIR in `asmcmd ls ${D}`
do
echo ${DIR} `asmcmd du ${D}/${DIR} | tail -1`
done) | awk -v D="$D" ‘ BEGIN { printf("\n\t\t%40s\n\n", D " subdirectories size") ;
printf("%25s%16s%16s\n", "Subdir", "Used MB", "Mirror MB") ;
printf("%25s%16s%16s\n", "——", "——-", "———") ;}
{
printf("%25s%16s%16s\n", $1, $2, $3) ;
use += $2 ;
mir += $3 ;
}
END { printf("\n\n%25s%16s%16s\n", "——", "——-", "———") ;
printf("%25s%16s%16s\n\n", "Total", use, mir) ;} ‘
fi

#************************************************************************#
#* E N D O F S O U R C E *#
#************************************************************************#
[/code]

Oracle 12c Dataguard RAC Primary to RAC Standby with Dataguard Broker (on Exadata)

Setup for the Environment
——————-

Exadata X4-2 Quater Rack n Primary and Standby Site
Grid Version : 12.1.0.2
DB Home Version : 12.1.0.2

Primary, 2 Node RAC, db_unique_name = primeprd
Instance 2 : primeprd1
Instance 2 : primeprd2
TNS entry: PRIMEPRD, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEPRD =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = primaryhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd)
)
)

Standby, 2 Node RAC, db_unique_name= primedr
Instance 2 : primedr1
Instance 2 : primedr2
TNS ENTRY: PRIMEDR, but below entry in tnsnames.ora files on both node with DB Home user, most likely it will be Oracle user.

PRIMEDR =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = standbyhost.domain.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr)
)
)

1. Enable Force Logging on the Database

[code language=”sql”]
alter database force logging;
[/code]

Database altered.

2.1 change the below parameter in Primary database.

[code language=”sql”]

ALTER SYSTEM SET db_recovery_file_dest=’+RECOC1′ scope=both sid=’*’;

System altered.

alter system set LOG_ARCHIVE_CONFIG=’DG_CONFIG=(primeprd,primedr)’ scope=both sid=’*’;

alter system set LOG_ARCHIVE_DEST_1=’LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=primeprd’ scope=both sid=’*’;

alter system set LOG_ARCHIVE_DEST_2=’SERVICE=primedr LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=primedr’ scope=both sid=’*’;

alter system set log_archive_format=’%t_%s_%r.arc’ scope=spfile sid=’*’;

alter system set log_archive_max_processes=8 scope=both sid=’*’;

alter system set fal_server=primedr scope=both sid=’*’;

alter system set fal_client=primeprd scope=both sid=’*’;

alter system set standby_file_management=AUTO scope=both sid=’*’;

alter system set db_file_name_convert=(‘+DATAC1/primedr/’, ‘+DATAC1/primeprd/’, ‘+RECOC1/primedr/’, ‘+RECOC1/primeprd/’) scope=both sid=’*’;

alter system set log_file_name_convert=(‘+DATAC1/primedr/’, ‘+DATAC1/primeprd/’, ‘+RECOC1/primedr/’, ‘+RECOC1/primeprd/’) scope=both sid=’*’;

— Query to check if parameters are set correctly —

set linesize 500 pages 0
col value for a90
col name for a50
select name, value
from v$parameter
where name in (‘db_name’,’db_unique_name’,’log_archive_config’, ‘log_archive_dest_1′,’log_archive_dest_2’,
‘log_archive_dest_state_1′,’log_archive_dest_state_2’, ‘remote_login_passwordfile’,
‘log_archive_format’,’log_archive_max_processes’,’fal_server’,’db_file_name_convert’,
‘log_file_name_convert’, ‘standby_file_management’);

select name, value
from v$parameter
where name in (‘db_name’,’db_unique_name’,’log_archive_config’, ‘log_archive_dest_1′,’log_archive_dest_2’,
‘log_archive_dest_state_1′,’log_archive_dest_state_2’, ‘remote_login_passwordfile’,
‘log_archive_format’,’log_archive_max_processes’,’fal_server’,’db_file_name_convert’,
‘log_file_name_convert’, ‘standby_file_management’);

[/code]

3.1. verify the logfile using the below querys.

[code language=”sql”]
select group#,THREAD#,bytes/1024/1024,status,members from v$log;
select member from v$logfile;
[/code]

3.2. Add the below logfile and standby logfile in primary database, make sure you have large redolog files. In this day and age, 100MB does not really cut it.

[code language=”sql”]
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 1 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 2 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 3 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 1 GROUP 4 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 5 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 6 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 7 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 8 (‘+DATAC1′,’+RECOC1’) SIZE 1G;

select * from v$standby_log;

— Add Standby Redolog files, number of standby redo log files = Redo logfile Groups + 2 —

ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 9 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 10 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 11 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 12 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 1 GROUP 13 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 14 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 15 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 16 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 17 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
ALTER DATABASE ADD STANDBY LOGFILE THREAD 2 GROUP 18 (‘+DATAC1′,’+RECOC1’) SIZE 1G;
[/code]

4. Take the rman backup using below command.

[code language=”sql”]
rman target / nocatalog
run
{
sql ‘alter system archive log current’;
backup as compressed backupset database plus archivelog format ‘/u03/rmanbkp/online/Primary_bkp_for_stndby_%U’;
backup current controlfile for standby format ‘/u03/rmanbkp/online/stby.ctl’;
sql ‘alter system archive log current’;
}
[/code]

5.1. create the init file using the below command in primary database.

[code language=”sql”]
create pfile=’/u03/rmanbkp/prime/pfile_for_standby.txt’ from spfile;
[/code]

— Primary PFILE —

[code language=”sql”]
primeprd1.__data_transfer_cache_size=0
primeprd2.__data_transfer_cache_size=0
primeprd1.__db_cache_size=7046430720
primeprd2.__db_cache_size=7147094016
primeprd1.__java_pool_size=67108864
primeprd2.__java_pool_size=67108864
primeprd1.__large_pool_size=301989888
primeprd2.__large_pool_size=301989888
primeprd2.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primeprd1.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primeprd1.__pga_aggregate_target=3221225472
primeprd2.__pga_aggregate_target=3221225472
primeprd1.__sga_target=9663676416
primeprd2.__sga_target=9663676416
primeprd1.__shared_io_pool_size=469762048
primeprd2.__shared_io_pool_size=469762048
primeprd1.__shared_pool_size=1644167168
primeprd2.__shared_pool_size=1577058304
primeprd1.__streams_pool_size=100663296
primeprd2.__streams_pool_size=67108864
*.audit_file_dest=’/u03/app/oracle/admin/primeprd/adump’
*.audit_trail=’db’
*.cluster_database=TRUE
*.compatible=’12.1.0.2.0′
*.control_files=’+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763′,’+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763′
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest=’+DATAC1′
*.db_domain=”
*.db_file_name_convert=’+DATAC1/primedr/’,’+DATAC1/primeprd/’,’+RECOC1/primedr/’,’+RECOC1/primeprd/’
*.db_name=’primeprd’
*.db_recovery_file_dest=’+DATAC1′
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest=’/u03/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primeprdXDB)’
*.fal_client=’primeprd’
*.fal_server=’primedr’
primeprd2.instance_number=2
primeprd1.instance_number=1
*.log_archive_config=’dg_config=(primeprd,primedr)’
*.log_archive_dest_1=’LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primeprd’
*.log_archive_dest_2=’service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr’
*.log_archive_dest_state_1=’ENABLE’
*.log_archive_dest_state_2=’ENABLE’
*.log_archive_format=’log%t_%s_%r.arc’
*.log_file_name_convert=’+DATAC1/primedr/’,’+DATAC1/primeprd/’,’+RECOC1/primedr/’,’+RECOC1/primeprd/’
*.nls_length_semantics=’CHAR’
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile=’exclusive’
*.resource_manager_cpu_allocation=48
*.resource_manager_plan=’DEFAULT’
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management=’AUTO’
*.temp_undo_enabled=TRUE
primeprd2.thread=2
primeprd1.thread=1
primeprd2.undo_tablespace=’UNDOTBS2′
primeprd1.undo_tablespace=’UNDOTBS1′
[/code]

— DR PFILE —
[code language=”sql”]
primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primedr1.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest=’/u03/app/oracle/admin/primeprd/adump’ # Create on Both Nodes
*.audit_trail=’db’
*.cluster_database=TRUE
*.compatible=’12.1.0.2.0′
#*.control_files=’+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763′,’+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763′
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest=’+DATAC1′
*.db_domain=”
*.db_file_name_convert=’PRIMEPRD’,’PRIMEDR’ ##### Changed for dataguard
*.db_unique_name=’primedr’ ##### Changed for dataguard
*.db_name=’primeprd’
*.db_recovery_file_dest=’+RECOC1′
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest=’/u03/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)’ ##### Changed for dataguard
*.fal_client=’primedr’ ##### Changed for dataguard
*.fal_server=’primeprd’ ##### Changed for dataguard
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config=’dg_config=(primeprd,primedr)’
*.log_archive_dest_1=’LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr’ ##### Changed for dataguard
*.log_archive_dest_2=’service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd’ ##### Changed for dataguard
*.log_archive_dest_state_1=’ENABLE’
*.log_archive_dest_state_2=’ENABLE’
*.log_archive_format=’log%t_%s_%r.arc’
*.log_file_name_convert=’PRIMEPRD’,’PRIMEDR’ ##### Changed for dataguard
*.remote_listener=’exadr-scan:1521′ ##### Changed for dataguard
*.nls_length_semantics=’CHAR’
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile=’exclusive’
*.resource_manager_cpu_allocation=48
*.resource_manager_plan=’DEFAULT’
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management=’AUTO’
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace=’UNDOTBS2′
primedr1.undo_tablespace=’UNDOTBS1′
[/code]

cp -ip pfile_for_standby.txt pfile_to_apply.ora

vi pfile_to_apply.ora

— Actual PFILE after modifying all parameters —

[code language=”sql”]
primedr1.__data_transfer_cache_size=0
primedr2.__data_transfer_cache_size=0
primedr1.__db_cache_size=7046430720
primedr2.__db_cache_size=7147094016
primedr1.__java_pool_size=67108864
primedr2.__java_pool_size=67108864
primedr1.__large_pool_size=301989888
primedr2.__large_pool_size=301989888
primedr2.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primedr1.__oracle_base=’/u03/app/oracle’#ORACLE_BASE set from environment
primedr1.__pga_aggregate_target=3221225472
primedr2.__pga_aggregate_target=3221225472
primedr1.__sga_target=9663676416
primedr2.__sga_target=9663676416
primedr1.__shared_io_pool_size=469762048
primedr2.__shared_io_pool_size=469762048
primedr1.__shared_pool_size=1644167168
primedr2.__shared_pool_size=1577058304
primedr1.__streams_pool_size=100663296
primedr2.__streams_pool_size=67108864
*.audit_file_dest=’/u03/app/oracle/admin/primeprd/adump’
*.audit_trail=’db’
*.cluster_database=TRUE
*.compatible=’12.1.0.2.0′
#*.control_files=’+DATAC1/PRIMEPRD/CONTROLFILE/current.660.914322763′,’+DATAC1/PRIMEPRD/CONTROLFILE/current.665.914322763′
*.cpu_count=16
*.db_block_size=8192
*.db_create_file_dest=’+DATAC1′
*.db_domain=”
*.db_file_name_convert=’PRIMEPRD’,’PRIMEDR’
*.db_unique_name=’primedr’
*.db_name=’primeprd’
*.db_recovery_file_dest=’+RECOC1′
*.db_recovery_file_dest_size=536870912000
*.db_writer_processes=6
*.ddl_lock_timeout=30
*.diagnostic_dest=’/u03/app/oracle’
*.dispatchers='(PROTOCOL=TCP) (SERVICE=primedrXDB)’
*.fal_client=’primedr’
*.fal_server=’primeprd’
primedr2.instance_number=2
primedr1.instance_number=1
*.log_archive_config=’dg_config=(primeprd,primedr)’
*.log_archive_dest_1=’LOCATION=+RECOC1 valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=primedr’
*.log_archive_dest_2=’service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd’
*.log_archive_dest_state_1=’ENABLE’
*.log_archive_dest_state_2=’ENABLE’
*.log_archive_format=’log%t_%s_%r.arc’
*.log_file_name_convert=’PRIMEPRD’,’PRIMEDR’
*.remote_listener=’exadr-scan:1521′
*.nls_length_semantics=’CHAR’
*.open_cursors=2000
*.optimizer_adaptive_features=FALSE
*.pga_aggregate_target=3072m
*.processes=1500
*.remote_login_passwordfile=’exclusive’
*.resource_manager_cpu_allocation=48
*.resource_manager_plan=’DEFAULT’
*.session_cached_cursors=100
*.sessions=2272
*.sga_max_size=17179869184
*.sga_target=9216m
*.standby_file_management=’AUTO’
*.temp_undo_enabled=TRUE
primedr2.thread=2
primedr1.thread=1
primedr2.undo_tablespace=’UNDOTBS2′
primedr1.undo_tablespace=’UNDOTBS1′
[/code]

6. Create same RMAN backup directory in DR and copy the backup Piece and init file from Production to DR server.

Also do not forget to create directory for adump in path ‘/u03/app/oracle/admin/primeprd/adump’

[code language=”sql”]
startup nomount pfile=’/u03/rmanbkp/prime/pfile_to_apply.ora’;
[/code]

[code language=”sql”]
rman target sys/*****@primeprd auxiliary /

Recovery Manager: Release 12.1.0.2.0 – Production on Tue Jun 28 10:05:31 2016

Copyright (c) 1982, 2014, Oracle and/or its affiliates. All rights reserved.

connected to target database: PRIMEPRD (DBID=111111111111111)
connected to auxiliary database: PRIMEPRD (not mounted)

show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name PRIMEPRD are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF;
CONFIGURE DEFAULT DEVICE TYPE TO ‘SBT_TAPE’;
CONFIGURE CONTROLFILE AUTOBACKUP ON;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO ‘/u03/rmanbkp/prime/autobackup_control_file_%F’;
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE SBT_TAPE TO ‘%F’; # default
CONFIGURE DEVICE TYPE SBT_TAPE PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE SBT_TAPE TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT ‘/u03/rmanbkp/prime’;
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM ‘AES128’; # default
CONFIGURE COMPRESSION ALGORITHM ‘BASIC’ AS OF RELEASE ‘DEFAULT’ OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE RMAN OUTPUT TO KEEP FOR 7 DAYS; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO ‘/u03/app/oracle/product/12.1.0.2/payapps/dbs/snapcf_primeprd1.f’; # default

DUPLICATE TARGET DATABASE FOR STANDBY NOFILENAMECHECK;

[/code]

9. start the MRP process using the below command in SQL prompt.

[code language=”sql”]
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;
[/code]

10. In case if you get error while receiving logging onto standby use the below command and create password files on both nodes of production and dr

— For Oracle 12c, do not create password files with this method. Go to section below for Dataguard Broker Setup

[code language=”sql”]
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimeprd2 force=y ignorecase=y
[/code]

[code language=”sql”]
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr1 force=y ignorecase=y
orapwd file=/u03/app/oracle/product/12.1.0.2/payapps/dbs/orapwprimedr2 force=y ignorecase=y

select * from v$pwfile_users;
[/code]

Check by logging in from sqlplus from both production site and DR site from both nodes

From DR both nodes

[code language=”sql”]
sqlplus sys/*****@primeprd as sysdba
[/code]

From Prod Both nodes

[code language=”sql”]
sqlplus sys/******@primedr as sysdba
[/code]

Stop and start recovery again in DR

[code language=”sql”]
alter database recover managed standby database cancel;

alter database recover managed standby database using current logfile disconnect from session;

select process,status from v$managed_standby;

[/code]

11. Create the spfile and add the newly created control files

[code language=”sql”]
show parameter control_file

NAME TYPE VALUE
———————————— ———– ——————————
control_file_record_keep_time integer 7
control_files string +DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919, +RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923

[/code]

Add the controlfile string in the pfile you created earlier called pfile_to_apply.ora

cd /u03/rmanbkp/prime/
vi pfile_to_apply.ora

*.control_files=’+DATAC1/PRIMEDR/CONTROLFILE/current.636.915704919′,’+RECOC1/PRIMEDR/CONTROLFILE/current.15794.915704923′

[code language=”sql”]
shutdown immediate;
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.

startup nomount pfile=’/u03/rmanbkp/prime/pfile_to_apply.ora’;

create spfile=’+DATAC1/primedr/spfileprimedr.ora’ from pfile=’/u03/rmanbkp/prime/pfile_to_apply.ora’;

shutdown immediate;
[/code]

cd /u03/app/oracle/product/12.1.0.2/payapps/dbs/

vi initprimedr1.ora

–add only one line pointing to the SPFILE in ASM

spfile=’+DATAC1/primedr/spfileprimedr.ora’

[code language=”sql”]
startup mount pfile=’/u03/app/oracle/product/12.1.0.2/payapps/dbs/initprimedr1.ora’;

show parameter spfile

NAME TYPE VALUE
———————————— ———– ——————————
spfile string +DATAC1/primedr/spfileprimedr.
ora

alter database recover managed standby database using current logfile disconnect from session;

[/code]

12. Create cluster server using the below command.

[code language=”bash”]
srvctl add database -d primedr -o /u03/app/oracle/product/12.1.0.2/payapps -p +DATAC1/primedr/spfileprimedr.ora -r physical_standby -a DATAC1,RECOC1 -s MOUNT
srvctl add instance -d primedr -i primedr1 -n exadrdbadm01
srvctl add instance -d primedr -i primedr2 -n exadrdbadm02

srvctl add service -db onlndr -s onlnprod -preferred onlndr1,onlndr2
srvctl add service -db primedr -s primeprod -preferred primedr1,primedr2

srvctl config service -s onlnprod -db onlndr
srvctl config service -s primeprod -db primedr

srvctl stop database -d primedr

— if gives error stop from sqlplus

srvctl start database -d primedr -o mount

[/code]

— Start Recovery Again —

[code language=”sql”]
alter database recover managed standby database using current logfile disconnect from session;
[/code]

13.1. Add the below entries in both production nodes tnsnames.ora file as a oracle user. (Not required from ORacle 12c 12.1.0.2 onwards)

primeprd_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primeprd_DGMGRL)
)
)

13.2. Add the below entries in both DR nodes tnsnames.ora file as a oracle user.

primedr_DGMGRL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = ********)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = primedr_DGMGRL)
)
)

13.3. Add the below entries in production node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd1)
)

13.4. Add the below entries in production node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primeprd_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primeprd2)
)

13.5. Add the below entries in DR node1 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr1)
)

13.6. Add the below entries in DR node2 listener.ora file as a grid user.

(SID_DESC =
(GLOBAL_DBNAME = primedr_DGMGRL)
(ORACLE_HOME = /u03/app/oracle/product/12.1.0.2/payapps)
(SID_NAME = primedr2)
)

13.7. Using the below command to restart the listener.

[code language=”sql”]
$ srvctl stop listener
$ srvctl start listener
[/code]

14.1. Set the below parameter in primary database.

[code language=”sql”]
alter system set dg_broker_config_file1 = ‘+DATAC1/PRIMEPRD/dr1primeprd.dat’ scope=both sid=’*’;
alter system set dg_broker_config_file2 = ‘+DATAC1/PRIMEPRD/dr2primeprd.dat’ scope=both sid=’*’;
alter system set dg_broker_start = true scope=both sid=’*’;
[/code]

14.2. Set the below parameter in standby database.

[code language=”sql”]
alter system set dg_broker_config_file1 = ‘+DATAC1/PRIMEDR/dr1primedr.dat’ scope=both sid=’*’;
alter system set dg_broker_config_file2 = ‘+DATAC1/PRIMEDR/dr2primedr.dat’ scope=both sid=’*’;
alter system set dg_broker_start = true scope=both sid=’*’;
[/code]

15.1. Configure the dataguard brocker using the below command from production database server

[code language=”bash”]
/home/oracle::11g- dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

Configuration "dg_primeprd" created with primary database "primeprd"

add database primedr as connect identifier is primedr maintained as physical;
Error: ORA-16698: member has a LOG_ARCHIVE_DEST_n parameter with SERVICE attribute set

Failed.

remove configuration;
Removed configuration
[/code]

Set the log_archive_dest_2 settings from both the Primary and Standby databases to be nothing.

[code language=”sql”]
alter system set log_archive_dest_2=” scope=both sid=’*’;
[/code]

Disable then Enable the broker parameter on both the Primary and Standby databases.

— Primary
[code language=”sql”]
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;
[/code]

— Standby
[code language=”sql”]
alter system set dg_broker_start=false scope=both sid=’*’;
alter system set dg_broker_start=true scope=both sid=’*’;
[/code]

On the Primary database create the broker configuration for the Primary and Standby database and this time it should work fine with no issues since the log archive destination 2 setting is not set, this is the workaround/solution.

[code language=”bash”]
/home/oracle::11g– dgmgrl

connect sys/*****

create configuration dg_primeprd as primary database is primeprd connect identifier is primeprd;

add database primedr as connect identifier is primedr maintained as physical;

enable configuration;
[/code]

Set back the log_archive_dest_2 on both production and dr

— In Production Database —

[code language=”sql”]
alter system set log_archive_dest_2=’service=primedr LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primedr’ scope=both sid=’*’;
[/code]

— In DR Database —

[code language=”sql”]
alter system set log_archive_dest_2=’service=primeprd LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=primeprd’ scope=both sid=’*’;
[/code]

If in Broker you get below error

[code language=”bash”]
show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database
Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR (status updated 40 seconds ago)

Check in Broker Log File.

Mon Jul 04 14:41:46 2016
Error 1017 received logging on to the standby
————————————————————
Check that the primary and standby are using a password file
and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
and that the SYS password is same in the password files.
returning error ORA-16191

Failed to send message to site . Error code is ORA-01017.

[/code]

Then re-create password files in ASM in 12c like below, the correct format is pwd

Primary

[code language=”bash”]
srvctl config database -d primeprd
orapwd file=’+DATAC1/PRIMEPRD/pwdprimeprd’ entries=10 dbuniquename=primeprd password=****** force=y ignorecase=y
[/code]

Standby

[code language=”bash”]
srvctl config database -d primedr
orapwd file=’+DATAC1/PRIMEDR/pwdprimedr’ entries=10 dbuniquename=primedr password=****** force=y ignorecase=y
[/code]

Check configuration now and the error should have vanished and the status is success

[code language=”bash”]
show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 18 seconds ago)
[/code]

— Try doing Switchover using DGMGRL Now —

[code language=”bash”]
switchover to primedr
Performing switchover NOW, please wait…
Error: ORA-16644: apply instance not available

Failed.
Unable to switchover, primary database is still "primeprd"

edit database primedr set state=’APPLY-ON’ with apply instance=’primedr1′;

show database verbose primedr

Database – primedr

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 6.00 KByte/s
Active Apply Rate: 755.00 KByte/s
Maximum Apply Rate: 827.00 KByte/s
Real Time Query: OFF
Instance(s):
primedr1 (apply instance)
primedr2

switchover to primedr
Performing switchover NOW, please wait…
New primary database "primedr" is opening…
Oracle Clusterware is restarting database "primeprd" …
Switchover succeeded, new primary is "primedr"

show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS (status updated 40 seconds ago)

edit database primeprd set state=’APPLY-ON’ with apply instance=’primeprd1′;

show database verbose primeprd

Database – primeprd

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 0 seconds ago)
Apply Lag: 0 seconds (computed 0 seconds ago)
Average Apply Rate: 307.00 KByte/s
Active Apply Rate: 0 Byte/s
Maximum Apply Rate: 0 Byte/s
Real Time Query: OFF
Instance(s):
primeprd1 (apply instance)
primeprd2

DGMGRL

show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primedr – Primary database
primeprd – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 55 seconds ago)

Connect to DR site broker to intiate switchover

switchover to primeprd ;
Performing switchover NOW, please wait…
Operation requires a connection to instance "primeprd1" on database "primeprd"
Connecting to instance "primeprd1"…
Connected as SYSDBA.
New primary database "primeprd" is opening…
Oracle Clusterware is restarting database "primedr" …
Switchover succeeded, new primary is "primeprd

show configuration

Configuration – dg_primeprd

Protection Mode: MaxPerformance
Members:
primeprd – Primary database
primedr – Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS (status updated 11 seconds ago
[/code]

Query to check sessions writing to flashcache on Exadata

Query to check sessions writing to flashcache on Exadata
select se.sid, sn.name, s.value, se.program from v$sesstat s natural join v$statname sn left join v$session se on (s.sid = se.sid) where sn.name in (‘physical write requests optimized’, ‘cell writes to flash cache’, ‘cell overwrites in flash cache’) and s.value <> 0 order by s.sid,name;

ORA-12801: error signaled in parallel query server P001 ORA-00600: internal error code, arguments: [kxspoac : EXL 1], [32], [1], [], …

Oracle Version Affected : 12.1.0.2

OS : OEL 6 Update 7, x64

This error is related to Parallel Query and we encountered it many times, once on test and once on production. The solution is to apply below patch

Patch 19201867: SR12.2PX_HYBRID_LOAD – TRC – ORA-600 [KXSPOAC : EXL 1]

It is due to a bug, as mentioned in Oracle Support site

Bug 19201867 – Parallel slave process fails with ORA-600 [kxspoac : EXL 1] (Doc ID 19201867.8)

You can search the patch from Oracle support website and download it specific to your PSU. We had  to apply for Proactive BP 12.1.0.2.160419

The patch has to be applied in the Oracle DB home and all instances under that DB home need to be stopped before applying the patch. Since we were already on the BP 12.1.0.2.160419 there was no conflict in the application

1.Download the patch zip file

p19201867_12102160419DBEngSysandDBIM_Linux-x86-64.zip

2. Unzip the patch in directory

$ unzip -d <PATCH_TOP_DIR> p19201867_12102160419DBEngSysandDBIM_Linux-x86-64.zip

3. Run Pre-requsite check

$ cd <PATCH_TOP_DIR>/19201867
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

If no conflicts found proceed further, if there is some conflict open SR with Oracle to investigate further

4. Stop all databases on both nodes under the Oracle Home to where you are applying the patch

5. Apply the patch

$ opatch apply

6. Verify application

$ opatch lsinventory | grep 19201867

Create ACFS Filesystem Oracle 12c Linux 12.1.0.2

— Create ACFS filesystem on 12c Linux Exadata —
1. Create a volume in ASM

ASMCMD [+] > volcreate -G datac1 -s 500G ACFS_VOL

If you get like below error

ORA-15032: not all alterations performed
ORA-15479: ASM diskgroup does not support volumes
ORA-15221: ASM operation requires compatible.asm of 12.1.0.2.0 or higher (DBD ERROR: OCIStmtExecute

Check the Current Compatibility for the Diskgroup

select group_number, name,compatibility, database_compatibility from v$asm_diskgroup
GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY
———— —————————— ———————————————————— ————————————————————
1 DATAC1 12.1.0.1.0 11.2.0.2.0
2 DBFS_DG 12.1.0.0.0 11.2.0.2.0
3 RECOC1 12.1.0.1.0 11.2.0.2.0
SQL> alter diskgroup DATAC1 set attribute ‘compatible.asm’=’12.1.0.2.0’;

Diskgroup altered.

SQL> alter diskgroup RECOC1 set attribute ‘compatible.asm’=’12.1.0.2.0’;

Diskgroup altered.

SQL> alter diskgroup DBFS_DG set attribute ‘compatible.asm’=’12.1.0.2.0′;

Diskgroup altered.

SQL> select group_number, name,compatibility, database_compatibility from v$asm_diskgroup;

GROUP_NUMBER NAME COMPATIBILITY DATABASE_COMPATIBILITY
———— —————————— ———————————————————— ————————————————————
1 DATAC1 12.1.0.2.0 11.2.0.2.0
2 DBFS_DG 12.1.0.2.0 11.2.0.2.0
3 RECOC1 12.1.0.2.0 11.2.0.2.0

Run again the volcreate command
ASMCMD [+] > volcreate -G datac1 -s 500G ACFS_VOL
2. Check the volume information
ASMCMD [+] > volinfo -G datac1 ACFS_VOL

Diskgroup Name: DATAC1

Volume Name: ACFS_VOL
Volume Device: /dev/asm/acfs_vol-45
State: ENABLED
Size (MB): 512000
Resize Unit (MB): 64
Redundancy: MIRROR
Stripe Columns: 8
Stripe Width (K): 1024
Usage:
Mountpath:
sqlplus “/as sysasm”

SELECT volume_name, volume_device FROM V$ASM_VOLUME
WHERE volume_name =’ACFS_VOL’;

VOLUME_NAME
——————————
VOLUME_DEVICE
——————————————————————————–
ACFS_VOL
/dev/asm/acfs_vol-45

3. Create a file system with the Oracle ACFS mkfs command using output of above command

With root user run below command

/sbin/mkfs -t acfs /dev/asm/acfs_vol-45
mkfs.acfs: version = 12.1.0.2.0
mkfs.acfs: on-disk version = 39.0
mkfs.acfs: volume = /dev/asm/acfs_vol-45
mkfs.acfs: volume size = 536870912000 ( 500.00 GB )
mkfs.acfs: Format complete.
4. Register the file system with the acfsutil registry command.

Create a directory called ACFS

cd /
mkdir /ACFS

/sbin/acfsutil registry -a /dev/asm/acfs_vol-45 /ACFS

acfsutil registry: mount point /ACFS successfully added to Oracle Registry
Imp Note 1: Registering an Oracle ACFS file system also causes the file system to be mounted automatically whenever Oracle Clusterware or the system is restarted.
Imp Note 2: Oracle ACFS registration (acfsutil registry) is not supported in an Oracle Restart (standalone) configuration, which is a single-instance (non-clustered) environment.
5. Verify if ACFS filesystem mounted automatically

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VGExaDb-LVDbSys1
30G 17G 12G 59% /
tmpfs 252G 22G 231G 9% /dev/shm
/dev/sda1 496M 54M 418M 12% /boot
/dev/mapper/VGExaDb-LVDbOra1
99G 57G 37G 61% /u01
/dev/mapper/VGExaDb-LVDbOra2
197G 68G 119G 37% /u02
/dev/mapper/VGExaDb-LVBkp1
985G 288G 648G 31% /u03
/dev/asm/acfs_vol-45 500G 1.1G 499G 1% /ACFS
As you can see from above output the ACFS filesystem moutned automatically after registration
6. If you did not register the ACFS filesystem it will not mount automatically, you can mount the ACFS filesystem manually using below command

As root user

/bin/mount -t acfs /dev/asm/acfs_vol-45 /ACFS

7. Give appropriate permissions to the filesystem required by Oracle users
chown -R oracle:dba /ACFS

su – oracle

cd /ACFS

touch abc.txt

Enable WriteBack Cache for Exadata

Writeback cache is a feature of cache where the blocks are written to flash first before flushing to the disk. Writethrough is the default setting for Exadata cells, where the data is written first to disk and moved to flashcache for read purpose.

Download the setWBFC.sh script from Oracle support website

[root@exatestdbadm01 ~]# dcli -g cell_group -l root “cellcli -e ‘list cell attributes flashcachemode'”
exatestceladm01: WriteThrough
exatestceladm02: WriteThrough
exatestceladm03: WriteThrough

The Flashcache mode is in WriteBack mode in the Current Setup

[root@exatestdbadm01 ~]# dcli -g cell_group -l root cellcli -e list flashcache detail

[root@exatestdbadm01 u03]# mkdir writeback

[root@exatestdbadm01 u03]# cd writeback/

[root@exatestdbadm01 writeback]# unzip setWBFC_10020_20160418.zip
Archive: setWBFC_10020_20160418.zip
inflating: setWBFC_10020_20160418/setWBFC.sh
inflating: setWBFC_10020_20160418/wbfc_FLUSH.sh
inflating: setWBFC_10020_20160418/WBFC_README.txt

[root@exatestdbadm01 writeback]# cd setWBFC_10020_20160418

[root@exatestdbadm01 setWBFC_10020_20160418]# ls -ltr
total 168
-rw-r–r– 1 root root 683 Apr 18 16:24 wbfc_FLUSH.sh
-rw-r–r– 1 root root 144383 Apr 18 16:24 setWBFC.sh
-rw-r–r– 1 root root 14574 Apr 18 20:06 WBFC_README.txt

[root@exatestdbadm01 setWBFC_10020_20160418]# cd ..
[root@exatestdbadm01 writeback]# chmod -R 777 setWBFC_10020_20160418
[root@exatestdbadm01 writeback]# cd setWBFC_10020_20160418
— To run pre-req checks for enabling write back flash cach —

[root@exatestdbadm01 setWBFC_10020_20160418]# ./setWBFC.sh -g /root/cell_group -l /tmp -m WriteBack -o rolling -p
— To run pre-req checks for disabling write back flash cache —

[root@exatestdbadm01 setWBFC_10020_20160418]# ./setWBFC.sh -g /root/cell_group -l /tmp -m WriteThrough -o rolling -p
— Now run the Actual Script to Execute and change to WriteBack mode —

[root@exatestdbadm01 setWBFC_10020_20160418]# ./setWBFC.sh -g /root/cell_group -l /tmp -m WriteBack -o rolling

2016-05-19 14:37:56
Skipping STEP 9: Waiting for grid disks to sync not required.
2016-05-19 14:37:56
Validating inventory for griddisks
2016-05-19 14:38:01
Validation of griddisk: [PASSED]
2016-05-19 14:38:01
Validating inventory for flashdisks
2016-05-19 14:38:13
Validation of flashdisk: [PASSED]
2016-05-19 14:38:13
Validating inventory for flashsize
2016-05-19 14:38:14
Validation of flashsize: [PASSED]
2016-05-19 14:38:14
Setting flash cache to WriteBack completed successfully.
— To Check the Mode of FlashCache —

[root@exatestdbadm01 ~]# dcli -g ~/cell_group -l root cellcli -e “list cell attributes flashcachemode”
exatestceladm01: WriteBack
exatestceladm02: WriteBack
exatestceladm03: WriteBack

Install GUI X11 packages and VNC Server on Exadata Compute Nodes

Recently we patched our Exadata machines to the April 2016 QFSP. On running DBCA we got X11 forwarding error and upon further investigation we discovered that X11 related packages RPM are removed in the new compute node patches.

Image version: 12.1.2.3.1.160411

On Oracle support website there is a note about it:

Refer MOS for more information –> Unable to run graphical tools (gui) (runInstaller, dbca and dbua) on Exadata 12.1.2.1.0 – 12.1.2.2.0 (Doc ID 1969308.1)

In addition to adding pack the X11 packages we also installed VNC Server(tigervnc), which can be a useful thing to do some basic GUI stuff.

— Install GUI Packages back in Exadata —

1. Download OEL6 (OEL6.7) Linux from E-delivery website as image

2. FTP file to serve, preferably to Cluster file system to do on both nodes

3. Create a directory and mount the OEL6 media

mkdir /mnt

— IF exists just continue

mount -o loop /ACFS/V77197-01.iso /mnt
4. Change the REPO, Edit /etc/yum.repos.d/Exadata-computenode.repo

Remove old information and add below lines

[ol6_iso]
name=Oracle Exadata DB server
baseurl=file:///mnt
gpgcheck=0
enabled=1
5. Validate the repository

yum list –disablerepo=* –enablerepo=ol6_iso
6. Install the X11 packacges

yum –disablerepo=* –enablerepo=ol6_iso install xorg-x11-xauth

–Update 2 –: starting 12.1.2.2.0 this package is not mandatory for X applications, so you can run below if your imageinfo version is more than it–

–Update 1–: I had a conversation with Oracle Development team after posting this blog post and interestingly they could not re-produce the same scenario in their lab. We upgraded from OL5 to OL6 on the compute nodes with almost 2 years of difference between the image versions. If your compute node versions is already OL6 then most likely you should not encounter the above scenario (that’s what the Oracle development team told me atleast).  If you do encounter it then log a SR with Oracle and get their blessings before proceeding.

yum –disablerepo=* –enablerepo=ol6_iso install xorg-x11-utils
If packages already exist remove the pakages and reinstall since it will fix the library dependencies
yum –disablerepo=* –enablerepo=ol6_iso remove xorg-x11-utils
yum –disablerepo=* –enablerepo=ol6_iso install xorg-x11-utils
7. Edit /etc/ssh/sshd_config to enable X11Forwarding

vi /etc/ssh/sshd_config

Below values should not be commented

X11Forwarding yes
X11DisplayOffset 10
X11UseLocalhost yes

UseDNS yes
8. Restart the SSHD service to reflect the new changes
service sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
9. Run any GUI utility to check, export the display if required

export DISPLAY=10.10.10.1:0.0

dbca
10. Install VNC Server

yum –disablerepo=* –enablerepo=ol6_iso install vnc-server
11. Configure VNC Server for oracle user
vi /etc/sysconfig/vncservers

— Add / Edit Parameters like below in file, make sure the 2 lines are uncommented

VNCSERVERS=”2:oracle”
VNCSERVERARGS[2]=”-geometry 800×600 -nolisten tcp -localhost”

— Set VNC password for oracle user —

su – oracle

vncpasswd

Password:
Verify:

— Login back as root user and start the vnc server service —

service vncserver start
New ‘exatestdbadm01.domain.com:2 (oracle)’ desktop is exatestdbadm01.domain.com:2

Creating default startup script /home/oracle/.vnc/xstartup
Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/exatestdbadm01.domain.com:2.log

[ OK ]
Login with any VNC viewer to test

— To Stop Service Run —

service vncserver stop
Shutting down VNC server: 2:oracle [ OK ]