Switchover Operation On Physical Standby Database With Dataguard Broker

Posted: June 8, 2013 in StandBy
Tags: , ,

In this post i will be doing a switchover operation on physical standby database with dataguard broker. Following are assumptions for this post:

1) Fully functional dataguard broker setup
2) Primary Database name CORE
3) Standby Database name CORESTD

First, let’s check our configuration, to see if we have any error/issue in configuration:

DGMGRL> show configuration ;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE    - Primary database
    CORESTD - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL>
DGMGRL>
DGMGRL> show database verbose 'CORE';

Database - CORE

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE/CORE, /u04/oradata/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE/CORE, /u02/oradata/CORE, /u03/oradata/CORE/CORE, /u03/oradata/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db01-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'CORE_%s%t%r.ARC'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

If there is any error or warning message in above commands, we have to fix those. In my previous few posts, i have mentioned the problems which i faced with dataguard broker setup. Since, there is no error / warning with my configuration, i am ready to switchover to my standby database. With dataguard broker, all it requires a single command to do this job.Below is command:

DGMGRL> switchover to 'CORESTD';
Performing switchover NOW, please wait...
New primary database "CORESTD" is opening...
Operation requires shutdown of instance "CORE" on database "CORE"
Shutting down instance "CORE"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CORE" on database "CORE"
Starting instance "CORE"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "CORESTD"
DGMGRL>

Switchover operation will first convert the primary database to standby. At a particular point of time during switchover operation, i have only standby’s in my configuration. Once, Primary is converted to standby then the second phase start with converting the physical standby to primary database. Below are logs from primary database which is converted to standby.

Sat Jun 08 08:07:27 2013
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY  [Process Id: 22544482] (CORE)
Sat Jun 08 08:07:29 2013
Thread 1 advanced to log sequence 1318 (LGWR switch)
  Current log# 1 seq# 1318 mem# 0: /u02/oradata/CORE/redo01a.log
  Current log# 1 seq# 1318 mem# 1: /u03/oradata/CORE/redo01b.log
Waiting for all non-current ORLs to be archived...
Waiting for the ORL for thread 1 sequence 1317 to be archived...
Sat Jun 08 08:07:32 2013
Archived Log entry 682 added for thread 1 sequence 1317 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:40 2013
ORL for thread 1 sequence 1317 has been archived...
All non-current ORLs have been archived.
Waiting for all FAL entries to be archived...
All FAL entries have been archived.
Waiting for dest_id 2 to become synchronized...
Sat Jun 08 08:07:50 2013
Active, synchronized Physical Standby  switchover target has been identified
Sat Jun 08 08:07:50 2013
Thread 1 advanced to log sequence 1319 (LGWR switch)
  Current log# 2 seq# 1319 mem# 0: /u02/oradata/CORE/redo02a.log
  Current log# 2 seq# 1319 mem# 1: /u03/oradata/CORE/redo02b.log
ARCH: Standby redo logfile selected for thread 1 sequence 1318 for destination LOG_ARCHIVE_DEST_2
Archived Log entry 683 added for thread 1 sequence 1318 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:50 2013
Stopping background process CJQ0
Sat Jun 08 08:07:50 2013
SMON: disabling tx recovery
Stopping background process QMNC
CLOSE: killing server sessions.
CLOSE: all sessions shutdown successfully.
SMON: disabling cache recovery
Shutting down archive processes
Archiving is disabled
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC2: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC1: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC0: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC3: Archival stopped
Thread 1 closed at log sequence 1319
Successful close of redo thread 1
ARCH: Noswitch archival of thread 1, sequence 1319
ARCH: End-Of-Redo Branch archival of thread 1 sequence 1319
Archived Log entry 685 added for thread 1 sequence 1319 ID 0x36a9a16a dest 1:
ARCH: Archiving is disabled due to current logfile archival
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Primary will check for some target standby to have received all redo
Final check for a synchronized target standby. Check will be made once.
LOG_ARCHIVE_DEST_2 is a potential Physical Standby  switchover target
Active, synchronized target has been identified
Target has also applied all redo
Backup controlfile written to trace file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rsm0_22544482.trc
Clearing standby activation ID 917086570 (0x36a9a16a)
The primary database controlfile was created using the
'MAXLOGFILES 32' clause.
There is space for up to 29 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;
Archivelog for thread 1 sequence 1319 required for standby recovery
Switchover: Primary controlfile converted to standby controlfile succesfully.
Sat Jun 08 08:07:55 2013
MRP0 started with pid=18, OS id=12648840
MRP0: Background Managed Standby Recovery process started (CORE)
Serial Media Recovery started
Managed Standby Recovery not using Real Time Apply
Online logfile pre-clearing operation disabled by switchover
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1319_8v5lktwk_.arc
Identified End-Of-Redo for thread 1 sequence 1319
Resetting standby activation ID 0 (0x0)
Media Recovery End-Of-Redo indicator encountered
Media Recovery Applied until change 3702836
MRP0: Media Recovery Complete: End-Of-REDO (CORE)
MRP0: Background Media Recovery process shutdown (CORE)
Sat Jun 08 08:08:01 2013
Switchover: Complete - Database shutdown required (CORE)
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_2='' SCOPE=BOTH;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
Sat Jun 08 08:08:05 2013
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
License high water mark = 7
alter database CLOSE NORMAL
ORA-1109 signalled during: alter database CLOSE NORMAL...
alter database DISMOUNT
Completed: alter database DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Shutting down Data Guard Broker processes
Sat Jun 08 08:08:12 2013
Completed: Data Guard Broker shutdown
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Sat Jun 08 08:08:14 2013
Stopping background process VKTM:
Sat Jun 08 08:08:17 2013
Instance shutdown complete
Sat Jun 08 08:08:17 2013
Adjusting the default value of parameter parallel_max_servers
from 160 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_ora_12648846.trc:
ORA-27167: Attempt to determine if Oracle binary image is stored on remote server failed
ORA-27300: OS system dependent operation:parse_df failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: parse failed
ORA-27303: additional information: Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
Global           78643200   1704152   98%   383717    44% /u01/app/oracore/product
Image consistency checking encountered an error, checking disabled
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
Using parameter settings in server-side spfile /u01/app/oracore/product/11.2.0/dbhome_1/dbs/spfileCORE.ora
System parameters with non-default values:
  processes                = 150
  event                    = "10298 trace name context forever, level 32"
  nls_date_format          = "DD-MON-RRRR"
  nls_length_semantics     = "CHAR"
  filesystemio_options     = "SETALL"
  sga_target               = 4G
  control_files            = "/u01/oradata/CORE/control01.ctl"
  control_files            = "/u02/oradata/CORE/control02.ctl"
  control_files            = "/u03/oradata/CORE/control03.ctl"
  db_file_name_convert     = "/u04/oradata/CORE/CORE"
  db_file_name_convert     = "/u04/oradata/CORE"
  log_file_name_convert    = "/u02/oradata/CORE/CORE"
  log_file_name_convert    = "/u02/oradata/CORE"
  log_file_name_convert    = "/u03/oradata/CORE/CORE"
  log_file_name_convert    = "/u03/oradata/CORE"
  db_block_size            = 8192
  compatible               = "11.2.0.1"
  log_archive_dest_1       = "LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=CORE"
  log_archive_dest_2       = ""
  log_archive_dest_state_1 = "ENABLE"
  log_archive_dest_state_2 = "ENABLE"
  log_archive_min_succeed_dest= 1
  fal_server               = "CORESTD"
  log_archive_trace        = 0
  log_archive_config       = "dg_config=(CORE,CORESTD)"
  log_archive_format       = "CORE_%s%t%r.ARC"
  log_archive_max_processes= 4
  archive_lag_target       = 0
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = "/u05/oradata/CORE/FLASH"
  db_recovery_file_dest_size= 20G
  standby_file_management  = "AUTO"
  fast_start_mttr_target   = 300
  undo_management          = "AUTO"
  undo_tablespace          = "UNDOTBS_NEW"
  recyclebin               = "ON"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = "in.isol.net"
  global_names             = FALSE
  session_cached_cursors   = 100
  job_queue_processes      = 10
  cursor_sharing           = "SIMILAR"
  db_name                  = "CORE"
  db_unique_name           = "CORE"
  open_cursors             = 300
  pga_aggregate_target     = 500M
  dg_broker_start          = TRUE
  diagnostic_dest          = "/u01/app/oracore/diag/CORE"
Sat Jun 08 08:08:18 2013
PMON started with pid=2, OS id=25231484
Sat Jun 08 08:08:18 2013
VKTM started with pid=3, OS id=2490746 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Sat Jun 08 08:08:18 2013
GEN0 started with pid=4, OS id=49414280
Sat Jun 08 08:08:18 2013
DIAG started with pid=5, OS id=62980250
Sat Jun 08 08:08:18 2013
DBRM started with pid=6, OS id=20840590
Sat Jun 08 08:08:18 2013
PSP0 started with pid=7, OS id=64684232
Sat Jun 08 08:08:19 2013
DIA0 started with pid=8, OS id=63045672
Sat Jun 08 08:08:19 2013
MMAN started with pid=9, OS id=41680954
Sat Jun 08 08:08:19 2013
DBW0 started with pid=10, OS id=48038028
Sat Jun 08 08:08:19 2013
LGWR started with pid=11, OS id=3801352
Sat Jun 08 08:08:19 2013
CKPT started with pid=12, OS id=56099042
Sat Jun 08 08:08:19 2013
SMON started with pid=13, OS id=66912502
Sat Jun 08 08:08:19 2013
RECO started with pid=14, OS id=17563898
Sat Jun 08 08:08:19 2013
MMON started with pid=15, OS id=65994762
Sat Jun 08 08:08:19 2013
MMNL started with pid=16, OS id=57671796
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Sat Jun 08 08:08:20 2013
DMON started with pid=17, OS id=44761096
Sat Jun 08 08:08:20 2013
alter database  mount
ARCH: STARTING ARCH PROCESSES
Sat Jun 08 08:08:24 2013
ARC0 started with pid=19, OS id=28442740
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Successful mount of redo thread 1, with mount id 924279348
Allocated 15937344 bytes in shared pool for flashback generation buffer
Sat Jun 08 08:08:24 2013
ARC1 started with pid=20, OS id=49873126
Starting background process RVWR
Sat Jun 08 08:08:24 2013
RVWR started with pid=21, OS id=12648852
Physical Standby Database mounted.
Lost write protection disabled
Sat Jun 08 08:08:24 2013
ARC2 started with pid=22, OS id=22544484
Sat Jun 08 08:08:24 2013
ARC3 started with pid=23, OS id=50069672
ARC1: Archival started
ARC2: Archival started
ARC2: Becoming the 'no FAL' ARCH
ARC1: Becoming the heartbeat ARCH
Completed: alter database  mount
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Starting Data Guard Broker (DMON)
Sat Jun 08 08:08:28 2013
NSV1 started with pid=24, OS id=17629426
Sat Jun 08 08:08:31 2013
INSV started with pid=25, OS id=64487496
Sat Jun 08 08:08:36 2013
RSM0 started with pid=27, OS id=50528398
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='CORE_%s%t%r.ARC' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE/CORE','/u04/oradata/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE/CORE','/u02/oradata/CORE','/u03/oradata/CORE/CORE','/u03/oradata/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (CORE)
Sat Jun 08 08:08:40 2013
MRP0 started with pid=28, OS id=37748798
MRP0: Background Managed Standby Recovery process started (CORE)
 started logmerger process
Sat Jun 08 08:08:45 2013
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 8 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 /u02/oradata/CORE/redo01a.log
Clearing online log 1 of thread 1 sequence number 1318
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /u02/oradata/CORE/redo02a.log
Clearing online log 2 of thread 1 sequence number 1319
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /u02/oradata/CORE/redo03a.log
Sat Jun 08 08:08:47 2013
RFS[1]: Assigned to RFS process 6160894
RFS[1]: Identified database type as 'physical standby': Client is ARCH pid 33948024
Clearing online log 3 of thread 1 sequence number 1317
Sat Jun 08 08:08:47 2013
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Clearing online redo logfile 3 complete
Sat Jun 08 08:08:47 2013
RFS[2]: Assigned to RFS process 57868434
RFS[2]: Identified database type as 'physical standby': Client is ARCH pid 24904034
RFS[1]: Opened log for thread 1 sequence 1322 dbid 917065834 branch 810292394
Media Recovery Waiting for thread 1 sequence 1320
Fetching gap sequence in thread 1, gap sequence 1320-1321
Sat Jun 08 08:08:47 2013
RFS[3]: Assigned to RFS process 59703302
RFS[3]: Identified database type as 'physical standby': Client is ARCH pid 54788584
RFS[2]: Opened log for thread 1 sequence 1320 dbid 917065834 branch 810292394
Archived Log entry 687 added for thread 1 sequence 1322 rlc 810292394 ID 0x37171ce4 dest 2:
RFS[3]: Opened log for thread 1 sequence 1321 dbid 917065834 branch 810292394
Archived Log entry 688 added for thread 1 sequence 1320 rlc 810292394 ID 0x37171ce4 dest 2:
Archived Log entry 689 added for thread 1 sequence 1321 rlc 810292394 ID 0x37171ce4 dest 2:
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rfs_57868434.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rfs_59703302.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1320_8v5lmhxz_.arc
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1321_8v5lmhyo_.arc
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1322_8v5lmhv1_.arc
Sat Jun 08 08:08:49 2013
RFS[4]: Assigned to RFS process 38600762
RFS[4]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 19267688
Primary database is in MAXIMUM PERFORMANCE mode
Media Recovery Waiting for thread 1 sequence 1323
RFS[4]: Selected log 4 for thread 1 sequence 1323 dbid 917065834 branch 810292394
RFS[4]: Selected log 5 for thread 1 sequence 1324 dbid 917065834 branch 810292394
Sat Jun 08 08:08:50 2013
Archived Log entry 690 added for thread 1 sequence 1323 ID 0x37171ce4 dest 1:
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1323_8v5lml0x_.arc
Media Recovery Waiting for thread 1 sequence 1324 (in transit)
Recovery of Online Redo Log: Thread 1 Group 5 Seq 1324 Reading mem 0
  Mem# 0: /u02/oradata/CORE/std_redo05a.log
  Mem# 1: /u03/oradata/CORE/std_redo05b.log
Sat Jun 08 08:09:47 2013
RFS[5]: Assigned to RFS process 66453586
RFS[5]: Identified database type as 'physical standby': Client is ARCH pid 33948024

After primary is converted to standby, here are logs for standby conversion to primary by broker:

Sat Jun 08 08:07:29 2013
Media Recovery Waiting for thread 1 sequence 1318
Sat Jun 08 08:07:40 2013
RFS[4]: Assigned to RFS process 33358232
RFS[4]: Identified database type as 'physical standby': Client is ARCH pid 48955422
Sat Jun 08 08:07:50 2013
RFS[5]: Assigned to RFS process 20054084
RFS[5]: Identified database type as 'physical standby': Client is Foreground pid 22544482
RFS[5]: Selected log 4 for thread 1 sequence 1318 dbid 917065834 branch 810292394
Sat Jun 08 08:07:50 2013
Archived Log entry 473 added for thread 1 sequence 1318 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:51 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_08/o1_mf_1_1318_8v5lkpms_.arc
Media Recovery Waiting for thread 1 sequence 1319
Sat Jun 08 08:07:53 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Sat Jun 08 08:07:54 2013
RFS[6]: Assigned to RFS process 64159970
RFS[6]: Identified database type as 'physical standby': Client is Foreground pid 22544482
RFS[6]: Opened log for thread 1 sequence 1319 dbid 917065834 branch 810292394
Archived Log entry 474 added for thread 1 sequence 1319 rlc 810292394 ID 0x36a9a16a dest 2:
Sat Jun 08 08:07:55 2013
RFS[7]: Assigned to RFS process 64159974
RFS[7]: Identified database type as 'physical standby': Client is Foreground pid 22544482
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_08/o1_mf_1_1319_8v5lktvv_.arc
Identified End-Of-Redo for thread 1 sequence 1319
Resetting standby activation ID 917086570 (0x36a9a16a)
Media Recovery End-Of-Redo indicator encountered
Media Recovery Continuing
Resetting standby activation ID 917086570 (0x36a9a16a)
Media Recovery Waiting for thread 1 sequence 1320
Sat Jun 08 08:08:01 2013
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Sat Jun 08 08:08:02 2013
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_5112182.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Waiting for MRP0 pid 24248822 to terminate
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_5112182.trc:
ORA-16037: user requested cancel of managed recovery operation
Waiting for MRP0 pid 24248822 to terminate
Sat Jun 08 08:08:03 2013
MRP0: Background Media Recovery process shutdown (CORE)
Managed Standby Recovery Canceled (CORE)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SWITCHOVER TO PRIMARY (CORE)
Maximum wait for role transition is 15 minutes.
Backup controlfile written to trace file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_rsm0_52953440.trc
SwitchOver after complete recovery through change 3702836
Online log /u02/oradata/CORE/CORE/redo01a.log: Thread 1 Group 1 was previously cleared
Online log /u03/oradata/CORE/CORE/redo01b.log: Thread 1 Group 1 was previously cleared
Online log /u02/oradata/CORE/CORE/redo02a.log: Thread 1 Group 2 was previously cleared
Online log /u03/oradata/CORE/CORE/redo02b.log: Thread 1 Group 2 was previously cleared
Online log /u02/oradata/CORE/CORE/redo03a.log: Thread 1 Group 3 was previously cleared
Online log /u03/oradata/CORE/CORE/redo03b.log: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 3702834
Switchover: Complete - Database mounted as primary
Sat Jun 08 08:08:03 2013
ARC1: Becoming the 'no SRL' ARCH
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER SYSTEM SET log_archive_dest_2='service="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))"','LGWR ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name="CORE" net_timeout=30','valid_for=(all_logfiles,primary_role)' SCOPE=BOTH;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER DATABASE OPEN
Data Guard Broker initializing...
Sat Jun 08 08:08:04 2013
Assigning activation ID 924261604 (0x37171ce4)
Thread 1 advanced to log sequence 1321 (thread open)
Thread 1 opened at log sequence 1321
  Current log# 2 seq# 1321 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 1321 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Successful open of redo thread 1
Sat Jun 08 08:08:05 2013
SMON: enabling cache recovery
Sat Jun 08 08:08:05 2013
Archived Log entry 475 added for thread 1 sequence 1320 ID 0x37171ce4 dest 1:
Sat Jun 08 08:08:05 2013
NSA2 started with pid=20, OS id=19267688
Sat Jun 08 08:08:05 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Sat Jun 08 08:08:05 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc2_33948024.trc:
ORA-12537: TNS:connection closed
PING[ARC2]: Heartbeat failed to connect to standby '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'. Error is 12537.


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-12537: TNS:connection closed
Thread 1 advanced to log sequence 1322 (LGWR switch)
  Current log# 3 seq# 1322 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 1322 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
FAL[server, ARC3]: Error 12537 creating remote archivelog file '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
FAL[server, ARC3]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Archived Log entry 476 added for thread 1 sequence 1321 ID 0x37171ce4 dest 1:
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Sat Jun 08 08:08:12 2013
Starting background process QMNC
Sat Jun 08 08:08:13 2013
QMNC started with pid=29, OS id=26935404
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
ARC0: STARTING ARCH PROCESSES
Sat Jun 08 08:08:13 2013
ARC4 started with pid=30, OS id=24248614
Completed: ALTER DATABASE OPEN
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE','/u04/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE','/u02/oradata/CORE/CORE','/u03/oradata/CORE','/u03/oradata/CORE/CORE' SCOPE=SPFILE;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER SYSTEM ARCHIVE LOG
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
Thread 1 advanced to log sequence 1323 (LGWR switch)
  Current log# 1 seq# 1323 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 1323 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Archived Log entry 477 added for thread 1 sequence 1322 ID 0x37171ce4 dest 1:


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc2_33948024.trc:
ORA-12537: TNS:connection closed
PING[ARC2]: Heartbeat failed to connect to standby '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'. Error is 12537.


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Error 12537 for archive log file 3 to '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
ARC4: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Shutting down archive processes
ARCH shutting down
ARC4: Archival stopped
Sat Jun 08 08:08:17 2013
Starting background process CJQ0
Sat Jun 08 08:08:17 2013
CJQ0 started with pid=33, OS id=28508468
Setting Resource Manager plan SCHEDULER[0x2FF9]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sat Jun 08 08:08:20 2013
Starting background process VKRM
Sat Jun 08 08:08:20 2013
VKRM started with pid=30, OS id=55378200
Sat Jun 08 08:08:25 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
Sat Jun 08 08:08:47 2013
ALTER SYSTEM ARCHIVE LOG
Sat Jun 08 08:08:47 2013
Thread 1 cannot allocate new log, sequence 1324
Checkpoint not complete
  Current log# 1 seq# 1323 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 1323 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Sat Jun 08 08:08:48 2013
ARC3: Archive log rejected (thread 1 sequence 1320) at host '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
FAL[server, ARC3]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Sat Jun 08 08:08:48 2013
ARC0: Archive log rejected (thread 1 sequence 1321) at host '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc0_54788584.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
FAL[server, ARC0]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc0_54788584.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Sat Jun 08 08:08:49 2013
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
Thread 1 advanced to log sequence 1324 (LGWR switch)
  Current log# 2 seq# 1324 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 1324 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Archived Log entry 481 added for thread 1 sequence 1323 ID 0x37171ce4 dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 1323 for destination LOG_ARCHIVE_DEST_2
LNS: Standby redo logfile selected for thread 1 sequence 1324 for destination LOG_ARCHIVE_DEST_2
Sat Jun 08 08:13:18 2013
Starting background process SMCO
Sat Jun 08 08:13:18 2013
SMCO started with pid=36, OS id=58523730
/u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace =>

Now, check the configuration from dataguard broker. Roles of both databases should be reversed as:

DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORESTD - Primary database
    CORE    - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'cph-core-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose 'CORE';

Database - CORE

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE/CORE, /u04/oradata/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE/CORE, /u02/oradata/CORE, /u03/oradata/CORE/CORE, /u03/oradata/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'cph-core-db01-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'CORE_%s%t%r.ARC'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

Check if archives are shipped to new standby database or not. At new primary database:

SEQUENCE# FIRST_TIME
---------- -----------
      1319 08-JUN-2013
      1320 08-JUN-2013
      1320 08-JUN-2013
      1321 08-JUN-2013
      1321 08-JUN-2013
      1322 08-JUN-2013
      1322 08-JUN-2013
      1323 08-JUN-2013
      1323 08-JUN-2013

9 rows selected.

SQL> alter system switch logfile;

System altered.

SQL> alter system switch logfile;

System altered.

SQL> SELECT SEQUENCE#, FIRST_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

 SEQUENCE# FIRST_TIME
---------- -----------
      1319 08-JUN-2013
      1320 08-JUN-2013
      1320 08-JUN-2013
      1321 08-JUN-2013
      1321 08-JUN-2013
      1322 08-JUN-2013
      1322 08-JUN-2013
      1323 08-JUN-2013
      1323 08-JUN-2013
      1324 08-JUN-2013
      1324 08-JUN-2013

 SEQUENCE# FIRST_TIME
---------- -----------
      1325 08-JUN-2013
      1325 08-JUN-2013

At new standby database, check if archives are getting applied or not:

SQL> SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

  SEQUENCE# APPLIED
---------- ---------
      1318 YES
      1318 NO
      1319 YES
      1319 NO
      1320 YES
      1321 YES
      1322 YES
      1323 YES
      1324 YES
      1325 YES
      1326 IN-MEMORY
Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s