StandBy | ORASteps

Archive for the ‘StandBy’ Category

Switchover Operation On Physical Standby Database With Dataguard Broker

Posted: June 8, 2013 in StandBy
Tags: switchover, Switchover to physical standby, switchover with dataguard broker

In this post i will be doing a switchover operation on physical standby database with dataguard broker. Following are assumptions for this post:

1) Fully functional dataguard broker setup
2) Primary Database name CORE
3) Standby Database name CORESTD

First, let’s check our configuration, to see if we have any error/issue in configuration:

DGMGRL> show configuration ;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE    - Primary database
    CORESTD - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL>
DGMGRL>
DGMGRL> show database verbose 'CORE';

Database - CORE

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE/CORE, /u04/oradata/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE/CORE, /u02/oradata/CORE, /u03/oradata/CORE/CORE, /u03/oradata/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db01-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'CORE_%s%t%r.ARC'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

If there is any error or warning message in above commands, we have to fix those. In my previous few posts, i have mentioned the problems which i faced with dataguard broker setup. Since, there is no error / warning with my configuration, i am ready to switchover to my standby database. With dataguard broker, all it requires a single command to do this job.Below is command:

DGMGRL> switchover to 'CORESTD';
Performing switchover NOW, please wait...
New primary database "CORESTD" is opening...
Operation requires shutdown of instance "CORE" on database "CORE"
Shutting down instance "CORE"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CORE" on database "CORE"
Starting instance "CORE"...
ORACLE instance started.
Database mounted.
Switchover succeeded, new primary is "CORESTD"
DGMGRL>

Switchover operation will first convert the primary database to standby. At a particular point of time during switchover operation, i have only standby’s in my configuration. Once, Primary is converted to standby then the second phase start with converting the physical standby to primary database. Below are logs from primary database which is converted to standby.

Sat Jun 08 08:07:27 2013
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY  [Process Id: 22544482] (CORE)
Sat Jun 08 08:07:29 2013
Thread 1 advanced to log sequence 1318 (LGWR switch)
  Current log# 1 seq# 1318 mem# 0: /u02/oradata/CORE/redo01a.log
  Current log# 1 seq# 1318 mem# 1: /u03/oradata/CORE/redo01b.log
Waiting for all non-current ORLs to be archived...
Waiting for the ORL for thread 1 sequence 1317 to be archived...
Sat Jun 08 08:07:32 2013
Archived Log entry 682 added for thread 1 sequence 1317 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:40 2013
ORL for thread 1 sequence 1317 has been archived...
All non-current ORLs have been archived.
Waiting for all FAL entries to be archived...
All FAL entries have been archived.
Waiting for dest_id 2 to become synchronized...
Sat Jun 08 08:07:50 2013
Active, synchronized Physical Standby  switchover target has been identified
Sat Jun 08 08:07:50 2013
Thread 1 advanced to log sequence 1319 (LGWR switch)
  Current log# 2 seq# 1319 mem# 0: /u02/oradata/CORE/redo02a.log
  Current log# 2 seq# 1319 mem# 1: /u03/oradata/CORE/redo02b.log
ARCH: Standby redo logfile selected for thread 1 sequence 1318 for destination LOG_ARCHIVE_DEST_2
Archived Log entry 683 added for thread 1 sequence 1318 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:50 2013
Stopping background process CJQ0
Sat Jun 08 08:07:50 2013
SMON: disabling tx recovery
Stopping background process QMNC
CLOSE: killing server sessions.
CLOSE: all sessions shutdown successfully.
SMON: disabling cache recovery
Shutting down archive processes
Archiving is disabled
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC2: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC1: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC0: Archival stopped
Sat Jun 08 08:07:53 2013
ARCH shutting down
ARC3: Archival stopped
Thread 1 closed at log sequence 1319
Successful close of redo thread 1
ARCH: Noswitch archival of thread 1, sequence 1319
ARCH: End-Of-Redo Branch archival of thread 1 sequence 1319
Archived Log entry 685 added for thread 1 sequence 1319 ID 0x36a9a16a dest 1:
ARCH: Archiving is disabled due to current logfile archival
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
Primary will check for some target standby to have received all redo
Final check for a synchronized target standby. Check will be made once.
LOG_ARCHIVE_DEST_2 is a potential Physical Standby  switchover target
Active, synchronized target has been identified
Target has also applied all redo
Backup controlfile written to trace file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rsm0_22544482.trc
Clearing standby activation ID 917086570 (0x36a9a16a)
The primary database controlfile was created using the
'MAXLOGFILES 32' clause.
There is space for up to 29 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;
Archivelog for thread 1 sequence 1319 required for standby recovery
Switchover: Primary controlfile converted to standby controlfile succesfully.
Sat Jun 08 08:07:55 2013
MRP0 started with pid=18, OS id=12648840
MRP0: Background Managed Standby Recovery process started (CORE)
Serial Media Recovery started
Managed Standby Recovery not using Real Time Apply
Online logfile pre-clearing operation disabled by switchover
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1319_8v5lktwk_.arc
Identified End-Of-Redo for thread 1 sequence 1319
Resetting standby activation ID 0 (0x0)
Media Recovery End-Of-Redo indicator encountered
Media Recovery Applied until change 3702836
MRP0: Media Recovery Complete: End-Of-REDO (CORE)
MRP0: Background Media Recovery process shutdown (CORE)
Sat Jun 08 08:08:01 2013
Switchover: Complete - Database shutdown required (CORE)
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_2='' SCOPE=BOTH;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
Sat Jun 08 08:08:05 2013
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
License high water mark = 7
alter database CLOSE NORMAL
ORA-1109 signalled during: alter database CLOSE NORMAL...
alter database DISMOUNT
Completed: alter database DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Shutting down Data Guard Broker processes
Sat Jun 08 08:08:12 2013
Completed: Data Guard Broker shutdown
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Sat Jun 08 08:08:14 2013
Stopping background process VKTM:
Sat Jun 08 08:08:17 2013
Instance shutdown complete
Sat Jun 08 08:08:17 2013
Adjusting the default value of parameter parallel_max_servers
from 160 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_ora_12648846.trc:
ORA-27167: Attempt to determine if Oracle binary image is stored on remote server failed
ORA-27300: OS system dependent operation:parse_df failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: parse failed
ORA-27303: additional information: Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
Global           78643200   1704152   98%   383717    44% /u01/app/oracore/product
Image consistency checking encountered an error, checking disabled
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =27
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
Using parameter settings in server-side spfile /u01/app/oracore/product/11.2.0/dbhome_1/dbs/spfileCORE.ora
System parameters with non-default values:
  processes                = 150
  event                    = "10298 trace name context forever, level 32"
  nls_date_format          = "DD-MON-RRRR"
  nls_length_semantics     = "CHAR"
  filesystemio_options     = "SETALL"
  sga_target               = 4G
  control_files            = "/u01/oradata/CORE/control01.ctl"
  control_files            = "/u02/oradata/CORE/control02.ctl"
  control_files            = "/u03/oradata/CORE/control03.ctl"
  db_file_name_convert     = "/u04/oradata/CORE/CORE"
  db_file_name_convert     = "/u04/oradata/CORE"
  log_file_name_convert    = "/u02/oradata/CORE/CORE"
  log_file_name_convert    = "/u02/oradata/CORE"
  log_file_name_convert    = "/u03/oradata/CORE/CORE"
  log_file_name_convert    = "/u03/oradata/CORE"
  db_block_size            = 8192
  compatible               = "11.2.0.1"
  log_archive_dest_1       = "LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=CORE"
  log_archive_dest_2       = ""
  log_archive_dest_state_1 = "ENABLE"
  log_archive_dest_state_2 = "ENABLE"
  log_archive_min_succeed_dest= 1
  fal_server               = "CORESTD"
  log_archive_trace        = 0
  log_archive_config       = "dg_config=(CORE,CORESTD)"
  log_archive_format       = "CORE_%s%t%r.ARC"
  log_archive_max_processes= 4
  archive_lag_target       = 0
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = "/u05/oradata/CORE/FLASH"
  db_recovery_file_dest_size= 20G
  standby_file_management  = "AUTO"
  fast_start_mttr_target   = 300
  undo_management          = "AUTO"
  undo_tablespace          = "UNDOTBS_NEW"
  recyclebin               = "ON"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = "in.isol.net"
  global_names             = FALSE
  session_cached_cursors   = 100
  job_queue_processes      = 10
  cursor_sharing           = "SIMILAR"
  db_name                  = "CORE"
  db_unique_name           = "CORE"
  open_cursors             = 300
  pga_aggregate_target     = 500M
  dg_broker_start          = TRUE
  diagnostic_dest          = "/u01/app/oracore/diag/CORE"
Sat Jun 08 08:08:18 2013
PMON started with pid=2, OS id=25231484
Sat Jun 08 08:08:18 2013
VKTM started with pid=3, OS id=2490746 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Sat Jun 08 08:08:18 2013
GEN0 started with pid=4, OS id=49414280
Sat Jun 08 08:08:18 2013
DIAG started with pid=5, OS id=62980250
Sat Jun 08 08:08:18 2013
DBRM started with pid=6, OS id=20840590
Sat Jun 08 08:08:18 2013
PSP0 started with pid=7, OS id=64684232
Sat Jun 08 08:08:19 2013
DIA0 started with pid=8, OS id=63045672
Sat Jun 08 08:08:19 2013
MMAN started with pid=9, OS id=41680954
Sat Jun 08 08:08:19 2013
DBW0 started with pid=10, OS id=48038028
Sat Jun 08 08:08:19 2013
LGWR started with pid=11, OS id=3801352
Sat Jun 08 08:08:19 2013
CKPT started with pid=12, OS id=56099042
Sat Jun 08 08:08:19 2013
SMON started with pid=13, OS id=66912502
Sat Jun 08 08:08:19 2013
RECO started with pid=14, OS id=17563898
Sat Jun 08 08:08:19 2013
MMON started with pid=15, OS id=65994762
Sat Jun 08 08:08:19 2013
MMNL started with pid=16, OS id=57671796
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Sat Jun 08 08:08:20 2013
DMON started with pid=17, OS id=44761096
Sat Jun 08 08:08:20 2013
alter database  mount
ARCH: STARTING ARCH PROCESSES
Sat Jun 08 08:08:24 2013
ARC0 started with pid=19, OS id=28442740
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Successful mount of redo thread 1, with mount id 924279348
Allocated 15937344 bytes in shared pool for flashback generation buffer
Sat Jun 08 08:08:24 2013
ARC1 started with pid=20, OS id=49873126
Starting background process RVWR
Sat Jun 08 08:08:24 2013
RVWR started with pid=21, OS id=12648852
Physical Standby Database mounted.
Lost write protection disabled
Sat Jun 08 08:08:24 2013
ARC2 started with pid=22, OS id=22544484
Sat Jun 08 08:08:24 2013
ARC3 started with pid=23, OS id=50069672
ARC1: Archival started
ARC2: Archival started
ARC2: Becoming the 'no FAL' ARCH
ARC1: Becoming the heartbeat ARCH
Completed: alter database  mount
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Starting Data Guard Broker (DMON)
Sat Jun 08 08:08:28 2013
NSV1 started with pid=24, OS id=17629426
Sat Jun 08 08:08:31 2013
INSV started with pid=25, OS id=64487496
Sat Jun 08 08:08:36 2013
RSM0 started with pid=27, OS id=50528398
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='CORE_%s%t%r.ARC' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE/CORE','/u04/oradata/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE/CORE','/u02/oradata/CORE','/u03/oradata/CORE/CORE','/u03/oradata/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (CORE)
Sat Jun 08 08:08:40 2013
MRP0 started with pid=28, OS id=37748798
MRP0: Background Managed Standby Recovery process started (CORE)
 started logmerger process
Sat Jun 08 08:08:45 2013
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 8 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 /u02/oradata/CORE/redo01a.log
Clearing online log 1 of thread 1 sequence number 1318
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /u02/oradata/CORE/redo02a.log
Clearing online log 2 of thread 1 sequence number 1319
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /u02/oradata/CORE/redo03a.log
Sat Jun 08 08:08:47 2013
RFS[1]: Assigned to RFS process 6160894
RFS[1]: Identified database type as 'physical standby': Client is ARCH pid 33948024
Clearing online log 3 of thread 1 sequence number 1317
Sat Jun 08 08:08:47 2013
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Clearing online redo logfile 3 complete
Sat Jun 08 08:08:47 2013
RFS[2]: Assigned to RFS process 57868434
RFS[2]: Identified database type as 'physical standby': Client is ARCH pid 24904034
RFS[1]: Opened log for thread 1 sequence 1322 dbid 917065834 branch 810292394
Media Recovery Waiting for thread 1 sequence 1320
Fetching gap sequence in thread 1, gap sequence 1320-1321
Sat Jun 08 08:08:47 2013
RFS[3]: Assigned to RFS process 59703302
RFS[3]: Identified database type as 'physical standby': Client is ARCH pid 54788584
RFS[2]: Opened log for thread 1 sequence 1320 dbid 917065834 branch 810292394
Archived Log entry 687 added for thread 1 sequence 1322 rlc 810292394 ID 0x37171ce4 dest 2:
RFS[3]: Opened log for thread 1 sequence 1321 dbid 917065834 branch 810292394
Archived Log entry 688 added for thread 1 sequence 1320 rlc 810292394 ID 0x37171ce4 dest 2:
Archived Log entry 689 added for thread 1 sequence 1321 rlc 810292394 ID 0x37171ce4 dest 2:
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rfs_57868434.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
Errors in file /u01/app/oracore/diag/CORE/diag/rdbms/core/CORE/trace/CORE_rfs_59703302.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1320_8v5lmhxz_.arc
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1321_8v5lmhyo_.arc
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1322_8v5lmhv1_.arc
Sat Jun 08 08:08:49 2013
RFS[4]: Assigned to RFS process 38600762
RFS[4]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 19267688
Primary database is in MAXIMUM PERFORMANCE mode
Media Recovery Waiting for thread 1 sequence 1323
RFS[4]: Selected log 4 for thread 1 sequence 1323 dbid 917065834 branch 810292394
RFS[4]: Selected log 5 for thread 1 sequence 1324 dbid 917065834 branch 810292394
Sat Jun 08 08:08:50 2013
Archived Log entry 690 added for thread 1 sequence 1323 ID 0x37171ce4 dest 1:
Media Recovery Log /u05/oradata/CORE/FLASH/CORE/archivelog/2013_06_08/o1_mf_1_1323_8v5lml0x_.arc
Media Recovery Waiting for thread 1 sequence 1324 (in transit)
Recovery of Online Redo Log: Thread 1 Group 5 Seq 1324 Reading mem 0
  Mem# 0: /u02/oradata/CORE/std_redo05a.log
  Mem# 1: /u03/oradata/CORE/std_redo05b.log
Sat Jun 08 08:09:47 2013
RFS[5]: Assigned to RFS process 66453586
RFS[5]: Identified database type as 'physical standby': Client is ARCH pid 33948024

After primary is converted to standby, here are logs for standby conversion to primary by broker:

Sat Jun 08 08:07:29 2013
Media Recovery Waiting for thread 1 sequence 1318
Sat Jun 08 08:07:40 2013
RFS[4]: Assigned to RFS process 33358232
RFS[4]: Identified database type as 'physical standby': Client is ARCH pid 48955422
Sat Jun 08 08:07:50 2013
RFS[5]: Assigned to RFS process 20054084
RFS[5]: Identified database type as 'physical standby': Client is Foreground pid 22544482
RFS[5]: Selected log 4 for thread 1 sequence 1318 dbid 917065834 branch 810292394
Sat Jun 08 08:07:50 2013
Archived Log entry 473 added for thread 1 sequence 1318 ID 0x36a9a16a dest 1:
Sat Jun 08 08:07:51 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_08/o1_mf_1_1318_8v5lkpms_.arc
Media Recovery Waiting for thread 1 sequence 1319
Sat Jun 08 08:07:53 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:07:53
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Sat Jun 08 08:07:54 2013
RFS[6]: Assigned to RFS process 64159970
RFS[6]: Identified database type as 'physical standby': Client is Foreground pid 22544482
RFS[6]: Opened log for thread 1 sequence 1319 dbid 917065834 branch 810292394
Archived Log entry 474 added for thread 1 sequence 1319 rlc 810292394 ID 0x36a9a16a dest 2:
Sat Jun 08 08:07:55 2013
RFS[7]: Assigned to RFS process 64159974
RFS[7]: Identified database type as 'physical standby': Client is Foreground pid 22544482
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_08/o1_mf_1_1319_8v5lktvv_.arc
Identified End-Of-Redo for thread 1 sequence 1319
Resetting standby activation ID 917086570 (0x36a9a16a)
Media Recovery End-Of-Redo indicator encountered
Media Recovery Continuing
Resetting standby activation ID 917086570 (0x36a9a16a)
Media Recovery Waiting for thread 1 sequence 1320
Sat Jun 08 08:08:01 2013
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Sat Jun 08 08:08:02 2013
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_5112182.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Waiting for MRP0 pid 24248822 to terminate
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_5112182.trc:
ORA-16037: user requested cancel of managed recovery operation
Waiting for MRP0 pid 24248822 to terminate
Sat Jun 08 08:08:03 2013
MRP0: Background Media Recovery process shutdown (CORE)
Managed Standby Recovery Canceled (CORE)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER DATABASE SWITCHOVER TO PRIMARY (CORE)
Maximum wait for role transition is 15 minutes.
Backup controlfile written to trace file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_rsm0_52953440.trc
SwitchOver after complete recovery through change 3702836
Online log /u02/oradata/CORE/CORE/redo01a.log: Thread 1 Group 1 was previously cleared
Online log /u03/oradata/CORE/CORE/redo01b.log: Thread 1 Group 1 was previously cleared
Online log /u02/oradata/CORE/CORE/redo02a.log: Thread 1 Group 2 was previously cleared
Online log /u03/oradata/CORE/CORE/redo02b.log: Thread 1 Group 2 was previously cleared
Online log /u02/oradata/CORE/CORE/redo03a.log: Thread 1 Group 3 was previously cleared
Online log /u03/oradata/CORE/CORE/redo03b.log: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 3702834
Switchover: Complete - Database mounted as primary
Sat Jun 08 08:08:03 2013
ARC1: Becoming the 'no SRL' ARCH
Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PRIMARY WAIT WITH SESSION SHUTDOWN
ALTER SYSTEM SET log_archive_dest_2='service="(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))"','LGWR ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name="CORE" net_timeout=30','valid_for=(all_logfiles,primary_role)' SCOPE=BOTH;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER DATABASE OPEN
Data Guard Broker initializing...
Sat Jun 08 08:08:04 2013
Assigning activation ID 924261604 (0x37171ce4)
Thread 1 advanced to log sequence 1321 (thread open)
Thread 1 opened at log sequence 1321
  Current log# 2 seq# 1321 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 1321 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Successful open of redo thread 1
Sat Jun 08 08:08:05 2013
SMON: enabling cache recovery
Sat Jun 08 08:08:05 2013
Archived Log entry 475 added for thread 1 sequence 1320 ID 0x37171ce4 dest 1:
Sat Jun 08 08:08:05 2013
NSA2 started with pid=20, OS id=19267688
Sat Jun 08 08:08:05 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Sat Jun 08 08:08:05 2013


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc2_33948024.trc:
ORA-12537: TNS:connection closed
PING[ARC2]: Heartbeat failed to connect to standby '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'. Error is 12537.


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:05
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-12537: TNS:connection closed
Thread 1 advanced to log sequence 1322 (LGWR switch)
  Current log# 3 seq# 1322 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 1322 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
FAL[server, ARC3]: Error 12537 creating remote archivelog file '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
FAL[server, ARC3]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Archived Log entry 476 added for thread 1 sequence 1321 ID 0x37171ce4 dest 1:
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Sat Jun 08 08:08:12 2013
Starting background process QMNC
Sat Jun 08 08:08:13 2013
QMNC started with pid=29, OS id=26935404
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
ARC0: STARTING ARCH PROCESSES
Sat Jun 08 08:08:13 2013
ARC4 started with pid=30, OS id=24248614
Completed: ALTER DATABASE OPEN
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE','/u04/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE','/u02/oradata/CORE/CORE','/u03/oradata/CORE','/u03/oradata/CORE/CORE' SCOPE=SPFILE;
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;
ALTER SYSTEM ARCHIVE LOG
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
Thread 1 advanced to log sequence 1323 (LGWR switch)
  Current log# 1 seq# 1323 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 1323 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Archived Log entry 477 added for thread 1 sequence 1322 ID 0x37171ce4 dest 1:


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc2_33948024.trc:
ORA-12537: TNS:connection closed
PING[ARC2]: Heartbeat failed to connect to standby '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'. Error is 12537.


***********************************************************************

Fatal NI connect error 12537, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=oracore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 08-JUN-2013 08:08:14
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12537

TNS-12537: TNS:connection closed
    ns secondary err code: 12560
    nt main err code: 507

TNS-00507: Connection closed
    nt secondary err code: 0
    nt OS err code: 0
Error 12537 received logging on to the standby
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Error 12537 for archive log file 3 to '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
ARC4: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_nsa2_19267688.trc:
ORA-12537: TNS:connection closed
Shutting down archive processes
ARCH shutting down
ARC4: Archival stopped
Sat Jun 08 08:08:17 2013
Starting background process CJQ0
Sat Jun 08 08:08:17 2013
CJQ0 started with pid=33, OS id=28508468
Setting Resource Manager plan SCHEDULER[0x2FF9]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Sat Jun 08 08:08:20 2013
Starting background process VKRM
Sat Jun 08 08:08:20 2013
VKRM started with pid=30, OS id=55378200
Sat Jun 08 08:08:25 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
Sat Jun 08 08:08:47 2013
ALTER SYSTEM ARCHIVE LOG
Sat Jun 08 08:08:47 2013
Thread 1 cannot allocate new log, sequence 1324
Checkpoint not complete
  Current log# 1 seq# 1323 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 1323 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Sat Jun 08 08:08:48 2013
ARC3: Archive log rejected (thread 1 sequence 1320) at host '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
FAL[server, ARC3]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc3_24904034.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Sat Jun 08 08:08:48 2013
ARC0: Archive log rejected (thread 1 sequence 1321) at host '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc0_54788584.trc:
ORA-16401: archive log rejected by Remote File Server (RFS)
FAL[server, ARC0]: FAL archive failed, see trace file.
Errors in file /u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace/CORE_arc0_54788584.trc:
ORA-16055: FAL request rejected
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance CORE - Archival Error. Archiver continuing.
Sat Jun 08 08:08:49 2013
******************************************************************
LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
******************************************************************
Thread 1 advanced to log sequence 1324 (LGWR switch)
  Current log# 2 seq# 1324 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 1324 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Archived Log entry 481 added for thread 1 sequence 1323 ID 0x37171ce4 dest 1:
LNS: Standby redo logfile selected for thread 1 sequence 1323 for destination LOG_ARCHIVE_DEST_2
LNS: Standby redo logfile selected for thread 1 sequence 1324 for destination LOG_ARCHIVE_DEST_2
Sat Jun 08 08:13:18 2013
Starting background process SMCO
Sat Jun 08 08:13:18 2013
SMCO started with pid=36, OS id=58523730
/u01/app/oracore/diag/diag/rdbms/corestd/CORE/trace =>

Now, check the configuration from dataguard broker. Roles of both databases should be reversed as:

DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORESTD - Primary database
    CORE    - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'cph-core-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> show database verbose 'CORE';

Database - CORE

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE/CORE, /u04/oradata/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE/CORE, /u02/oradata/CORE, /u03/oradata/CORE/CORE, /u03/oradata/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'cph-core-db01-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=cph-core-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.dk.flsmidth.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'CORE_%s%t%r.ARC'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

Check if archives are shipped to new standby database or not. At new primary database:

SEQUENCE# FIRST_TIME
---------- -----------
      1319 08-JUN-2013
      1320 08-JUN-2013
      1320 08-JUN-2013
      1321 08-JUN-2013
      1321 08-JUN-2013
      1322 08-JUN-2013
      1322 08-JUN-2013
      1323 08-JUN-2013
      1323 08-JUN-2013

9 rows selected.

SQL> alter system switch logfile;

System altered.

SQL> alter system switch logfile;

System altered.

SQL> SELECT SEQUENCE#, FIRST_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

 SEQUENCE# FIRST_TIME
---------- -----------
      1319 08-JUN-2013
      1320 08-JUN-2013
      1320 08-JUN-2013
      1321 08-JUN-2013
      1321 08-JUN-2013
      1322 08-JUN-2013
      1322 08-JUN-2013
      1323 08-JUN-2013
      1323 08-JUN-2013
      1324 08-JUN-2013
      1324 08-JUN-2013

 SEQUENCE# FIRST_TIME
---------- -----------
      1325 08-JUN-2013
      1325 08-JUN-2013

At new standby database, check if archives are getting applied or not:

SQL> SELECT SEQUENCE#,APPLIED FROM V$ARCHIVED_LOG ORDER BY SEQUENCE#;

  SEQUENCE# APPLIED
---------- ---------
      1318 YES
      1318 NO
      1319 YES
      1319 NO
      1320 YES
      1321 YES
      1322 YES
      1323 YES
      1324 YES
      1325 YES
      1326 IN-MEMORY

Convert Physical Standby To Snapshot Standby Using Dataguard Broker

Posted: June 6, 2013 in StandBy
Tags: convert physical standby to snapshot standby, dataguard broker snapshot standby, ORA-12514, snapshot standby

In this post i will be converting a physical standby database to snapshot standby using dataguard broker which i have configured in my previous post. Steps for dataguard broker set can be found here. Database version is 11.2.0.1 on AIX 6.1. Following are assumptions for this post:

1) Fully functional dataguard broker setup
2) Primary Database CORE on server ind-del-db01-s
3) Standby Database CORESTD on server ind-del-db02-s

First, let’s check the role of standby database and then any restore point information from standby database.

SQL> conn sys as sysdba
Enter password:
Connected.
SQL> select database_role from v$database;

DATABASE_ROLE
----------------
PHYSICAL STANDBY

SQL> select * from v$restore_point;

no rows selected

Following is my dataguard broker configuration.

DGMGRL> connect sys@corestd
Password:
Connected.
DGMGRL>
DGMGRL>
DGMGRL> show database 'CORESTD';

Database - CORESTD

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

Database Status:
SUCCESS

Now, let’s convert this to a snapshot standby database.

DGMGRL> convert database 'CORESTD' to snapshot standby;
Converting database "CORESTD" to a Snapshot Standby database, please wait...
Database "CORESTD" converted successfully
DGMGRL>

You can check that standby in now a snaphot standby database.

SQL> select database_role from v$database;

DATABASE_ROLE
----------------
SNAPSHOT STANDBY

SQL> select SCN,GUARANTEE_FLASHBACK_DATABASE,STORAGE_SIZE,time from v$restore_point;

       SCN GUA STORAGE_SIZE TIME
---------- --- ------------ ---------------------------------
   2587302 YES     15941632 29-MAY-13 07.12.45.000000000 AM

Now, database is converted to snapshot standby which is open in read-write mode. Notice that a new restore point is created during conversion process which is requirment for converting it back to physical standby. Below are logs of conversion from the standby database.

Wed May 29 07:12:44 2013
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
Wed May 29 07:12:44 2013
MRP0: Background Media Recovery cancelled with status 16037
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_54526418.trc:
ORA-16037: user requested cancel of managed recovery operation
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
Recovered data files to a consistent state at change 2587303
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_pr00_54526418.trc:
ORA-16037: user requested cancel of managed recovery operation
Waiting for MRP0 pid 38470088 to terminate
Waiting for MRP0 pid 38470088 to terminate
Wed May 29 07:12:45 2013
MRP0: Background Media Recovery process shutdown (CORE)
Managed Standby Recovery Canceled (CORE)
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL
alter database convert to snapshot standby
Created guaranteed restore point SNAPSHOT_STANDBY_REQUIRED_05/29/2013 07:12:45
krsv_proc_kill: Killing 3 processes (all RFS)
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
RESETLOGS after incomplete recovery UNTIL CHANGE 2587303
Resetting resetlogs activation ID 917086570 (0x36a9a16a)
Online log /u02/oradata/CORE/CORE/redo01a.log: Thread 1 Group 1 was previously cleared
Online log /u03/oradata/CORE/CORE/redo01b.log: Thread 1 Group 1 was previously cleared
Online log /u02/oradata/CORE/CORE/redo02a.log: Thread 1 Group 2 was previously cleared
Online log /u03/oradata/CORE/CORE/redo02b.log: Thread 1 Group 2 was previously cleared
Online log /u02/oradata/CORE/CORE/redo03a.log: Thread 1 Group 3 was previously cleared
Online log /u03/oradata/CORE/CORE/redo03b.log: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 2587301
Wed May 29 07:12:47 2013
Setting recovery target incarnation to 2
CONVERT TO SNAPSHOT STANDBY: Complete - Database mounted as snapshot standby
Completed: alter database convert to snapshot standby
ALTER DATABASE OPEN
Data Guard Broker initializing...
Data Guard Broker initialization complete
Wed May 29 07:12:49 2013
Assigning activation ID 923378597 (0x3709a3a5)
Thread 1 opened at log sequence 1
  Current log# 1 seq# 1 mem# 0: /u02/oradata/CORE/CORE/redo01a.logWed May 29 07:12:49 2013
ARC3: Becoming the 'no SRL' ARCH
  Current log# 1 seq# 1 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Successful open of redo thread 1
Wed May 29 07:12:49 2013
ARC0: Becoming the 'no SRL' ARCH
Wed May 29 07:12:49 2013
SMON: enabling cache recovery
Successfully onlined Undo Tablespace 2.
Dictionary check beginning
Wed May 29 07:12:50 2013
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_dbw0_42926254.trc:
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '/u04/oradata/CORE/CORE/temp01.dbf'
ORA-27037: unable to obtain file status
IBM AIX RISC System/6000 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_dbw0_42926254.trc:
ORA-01186: file 201 failed verification tests
ORA-01157: cannot identify/lock data file 201 - see DBWR trace file
ORA-01110: data file 201: '/u04/oradata/CORE/CORE/temp01.dbf'
File 201 not verified due to error ORA-01157
Dictionary check complete
Verifying file header compatibility for 11g tablespace encryption..
Verifying 11g file header compatibility for tablespace encryption completed
SMON: enabling tx recovery
Re-creating tempfile /u04/oradata/CORE/CORE/temp01.dbf
Database Characterset is AL32UTF8
No Resource Manager plan active
replication_dependency_tracking turned off (no async multimaster replication found)
Starting background process QMNC
Wed May 29 07:12:53 2013
QMNC started with pid=30, OS id=8585708
LOGSTDBY: Validating controlfile with logical metadata
LOGSTDBY: Validation complete
Wed May 29 07:12:54 2013
Completed: ALTER DATABASE OPEN
Wed May 29 07:12:54 2013
RFS[5]: Assigned to RFS process 49414370
RFS[5]: Identified database type as 'snapshot standby': Client is LGWR ASYNC pid 29753408
Wed May 29 07:12:54 2013
RFS[6]: Assigned to RFS process 47841488
RFS[6]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
RFS[5]: Selected log 5 for thread 1 sequence 1016 dbid 917065834 branch 810292394
ARC3: Becoming the 'no SRL' ARCH
RFS[5]: Selected log 4 for thread 1 sequence 1017 dbid 917065834 branch 810292394
Archived Log entry 40 added for thread 1 sequence 1016 ID 0x36a9a16a dest 1:
Wed May 29 07:12:59 2013
RFS[7]: Assigned to RFS process 61997390
RFS[7]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed May 29 07:16:40 2013
ARC3: Becoming the 'no SRL' ARCH
Wed May 29 07:16:40 2013
ARC0: Becoming the 'no SRL' ARCH
Wed May 29 07:16:40 2013
ARC1: Becoming the 'no SRL' ARCH
Wed May 29 07:17:58 2013
Starting background process SMCO
Wed May 29 07:17:58 2013
SMCO started with pid=36, OS id=27001342

Now, let’s create a table in this snapshot standby, just for testing and then convert it back to physical standby. Steps are as:

SQL> conn snaptest/snaptest@corestd
Connected.
SQL>
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
BIN$2CK5+mwyAELgQwotDFQAQg==$0 TABLE
TEST_VIEW                      VIEW
YYA_TEST_NEW_12                TABLE

SQL>
SQL> create table YYA_STANDBY_TEST as select * from dba_objects;

Table created.

SQL> select count(*) from YYA_STANDBY_TEST;

  COUNT(*)
----------
     13059

Now, let’s convert it back to physical standby and then check the existence of this table.

DGMGRL> convert database 'CORESTD' to physical standby;
Converting database "CORESTD" to a Physical Standby database, please wait...
Operation requires shutdown of instance "CORE" on database "CORESTD"
Shutting down instance "CORE"...
Unable to connect to database
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

Failed.
Warning: You are no longer connected to ORACLE.

Please complete the following steps and reissue the CONVERT command:
        shut down instance "CORE" of database "CORESTD"
        start up and mount instance "CORE" of database "CORESTD"

oooop’s, got a error while converting back 😦 . Now, let’s check my configuration again, if anything is wrong with it. On checking the configuration, i see that port 1521 is getting used in StaticConnectIdentifier which is incorrect. My port is 1522 as per my listener configuration. So, i changed this parameter on both primary and standby database as given in below steps.

DGMGRL> show database verbose 'CORE';

Database - CORE

  Role:            PRIMARY
  Intended State:  TRANSPORT-ON
  Instance(s):
    CORE
      
  Properties:
    DGConnectIdentifier             = 'CORE'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'optional'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE/CORE, /u04/oradata/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE/CORE, /u02/oradata/CORE, /u03/oradata/CORE/CORE, /u03/oradata/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db01-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = 'CORE_%s%t%r.ARC'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> edit database 'CORE' set property StaticConnectIdentifier='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db01-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORE_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))';
Property "staticconnectidentifier" updated
DGMGRL>
DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            SNAPSHOT STANDBY
  Intended State:  APPLY-OFF
  Transport Lag:   0 seconds
  Apply Lag:       3 hours 24 minutes 36 seconds
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = 'CORESTD'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

DGMGRL> edit database 'CORESTD' set property StaticConnectIdentifier='(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1522))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
> ;
Property "staticconnectidentifier" updated
DGMGRL>

With changed ports configuration, i was hoping that problem is resolved. But, i got same error when trying to convert back to physical standby. Now, time to check more logs. Error is showing that some problem with listener service, so i checked the listener service status.

/u01/app/incore/product/11.2.0.1/dbhome_1/network/admin => lsnrctl services LISTENER_CORE

LSNRCTL for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production on 05-JUN-2013 13:06:03

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db02-s)(PORT=1522)))
Services Summary...
Service "CORE" has 1 instance(s).
 Instance "CORE", status UNKNOWN, has 1 handler(s) for this service...
 Handler(s):
 "DEDICATED" established:24 refused:9
 LOCAL SERVER
Service "CORESTD_DGMGRL.in.isol.net" has 1 instance(s).
 Instance "CORE", status UNKNOWN, has 1 handler(s) for this service...
 Handler(s):
 "DEDICATED" established:1 refused:0
 LOCAL SERVER
Service "PLSExtProc" has 1 instance(s).
 Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
 Handler(s):
 "DEDICATED" established:0 refused:0
 LOCAL SERVER
The command completed successfully

Below are lines from my listener.log file.

29-MAY-2013 12:10:47 * (CONNECT_DATA=(SERVICE_NAME=CORESTD_DGB.in.isol.net)(INSTANCE_NAME=CORE)(CID=(PROGRAM=dgmgrl)(HOST=ind-del-db02-s)(USER=oracore))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.45.11.139)(PORT=36211)) * establish * CORESTD_DGB.in.isol.net * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Wed May 29 12:12:57 2013
29-MAY-2013 12:12:57 * ping * 0
Wed May 29 12:13:42 2013
29-MAY-2013 12:13:42 * (CONNECT_DATA=(SERVICE_NAME=CORE)(CID=(PROGRAM=dgmgrl)(HOST=ind-del-db02-s)(USER=oracore))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.45.11.139)(PORT=36857)) * establish * CORE * 0
Wed May 29 12:14:10 2013
29-MAY-2013 12:14:10 * (CONNECT_DATA=(SERVICE_NAME=CORESTD_DGB.in.isol.net)(INSTANCE_NAME=CORE)(CID=(PROGRAM=dgmgrl)(HOST=ind-del-db02-s)(USER=oracore))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.45.11.139)(PORT=36948)) * establish * CORESTD_DGB.in.isol.net * 12514
TNS-12514: TNS:listener does not currently know of service requested in connect descriptor
Wed May 29 12:14:30 2013

Above error shows that service CORESTD_DGB.in.isol.net is not recognized by my listener. So, i need to add this service to my listener.ora file. Below is my modified listener.ora file at standby database.

SID_LIST_LISTENER_CORE =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracore/product/11.2.0.1/dbhome_1)
      (PROGRAM = extproc)
    )
   (SID_DESC =
      (GLOBAL_DBNAME = CORE)
      (ORACLE_HOME = /u01/app/oracore/product/11.2.0.1/dbhome_1)
      (SID_NAME = CORE)
    )
   (SID_DESC =
      (GLOBAL_DBNAME = CORESTD_DGB.in.isol.net)
      (ORACLE_HOME = /u01/app/oracore/product/11.2.0.1/dbhome_1)
      (SID_NAME = CORE)
   )
   (SID_DESC =
      (GLOBAL_DBNAME = CORESTD_DGMGRL.in.isol.net)
      (ORACLE_HOME = /u01/app/oracore/product/11.2.0.1/dbhome_1)
      (SID_NAME = CORE)
    )
 )
LISTENER_CORE =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ind-del-db02-s)(PORT = 1522))
      )

restart the listener on standby and then try again to convert snapshot back to physical standby. Below are steps.

DGMGRL> connect sys@corestd
Password:
Connected.
DGMGRL>
DGMGRL>
DGMGRL> convert database 'CORESTD' to physical standby;
Converting database "CORESTD" to a Physical Standby database, please wait...
Operation requires shutdown of instance "CORE" on database "CORESTD"
Shutting down instance "CORE"...
Database closed.
Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CORE" on database "CORESTD"
Starting instance "CORE"...
ORACLE instance started.
Database mounted.
Continuing to convert database "CORESTD" ...
Operation requires shutdown of instance "CORE" on database "CORESTD"
Shutting down instance "CORE"...
ORA-01109: database not open

Database dismounted.
ORACLE instance shut down.
Operation requires startup of instance "CORE" on database "CORESTD"
Starting instance "CORE"...
ORACLE instance started.
Database mounted.
Database "CORESTD" converted successfully

Finally, it got converted back to physical standby 🙂 . Below are complete logs for this activity.

Wed Jun 05 00:17:44 2013
RFS[11]: Selected log 5 for thread 1 sequence 1204 dbid 917065834 branch 810292394
Wed Jun 05 00:17:44 2013
Archived Log entry 349 added for thread 1 sequence 1203 ID 0x36a9a16a dest 1:
Wed Jun 05 01:03:08 2013
Thread 1 advanced to log sequence 122 (LGWR switch)
  Current log# 2 seq# 122 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 122 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Wed Jun 05 01:03:08 2013
Archived Log entry 350 added for thread 1 sequence 121 ID 0x370a7740 dest 1:
Thread 1 advanced to log sequence 123 (LGWR switch)
  Current log# 3 seq# 123 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 123 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
Wed Jun 05 01:03:13 2013
Archived Log entry 351 added for thread 1 sequence 122 ID 0x370a7740 dest 1:
Thread 1 cannot allocate new log, sequence 124
Checkpoint not complete
  Current log# 3 seq# 123 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 123 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
Thread 1 advanced to log sequence 124 (LGWR switch)
  Current log# 1 seq# 124 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 124 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Wed Jun 05 01:03:14 2013
Archived Log entry 352 added for thread 1 sequence 123 ID 0x370a7740 dest 1:
Thread 1 cannot allocate new log, sequence 125
Checkpoint not complete
  Current log# 1 seq# 124 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 124 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Thread 1 advanced to log sequence 125 (LGWR switch)
  Current log# 2 seq# 125 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 125 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Archived Log entry 353 added for thread 1 sequence 124 ID 0x370a7740 dest 1:
Thread 1 cannot allocate new log, sequence 126
Checkpoint not complete
  Current log# 2 seq# 125 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 125 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Thread 1 advanced to log sequence 126 (LGWR switch)
  Current log# 3 seq# 126 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 126 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
Archived Log entry 354 added for thread 1 sequence 125 ID 0x370a7740 dest 1:
Wed Jun 05 01:03:18 2013
Thread 1 cannot allocate new log, sequence 127
Checkpoint not complete
  Current log# 3 seq# 126 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 126 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
Thread 1 advanced to log sequence 127 (LGWR switch)
  Current log# 1 seq# 127 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 127 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Archived Log entry 355 added for thread 1 sequence 126 ID 0x370a7740 dest 1:
Wed Jun 05 01:18:45 2013
RFS[11]: Selected log 4 for thread 1 sequence 1205 dbid 917065834 branch 810292394
Wed Jun 05 01:18:45 2013
Archived Log entry 356 added for thread 1 sequence 1204 ID 0x36a9a16a dest 1:
RFS[11]: Selected log 5 for thread 1 sequence 1206 dbid 917065834 branch 810292394
Wed Jun 05 01:18:46 2013
Archived Log entry 357 added for thread 1 sequence 1205 ID 0x36a9a16a dest 1:
RFS[11]: Selected log 4 for thread 1 sequence 1207 dbid 917065834 branch 810292394
Archived Log entry 358 added for thread 1 sequence 1206 ID 0x36a9a16a dest 1:
RFS[11]: Selected log 5 for thread 1 sequence 1208 dbid 917065834 branch 810292394
Archived Log entry 359 added for thread 1 sequence 1207 ID 0x36a9a16a dest 1:
RFS[11]: Selected log 4 for thread 1 sequence 1209 dbid 917065834 branch 810292394
Archived Log entry 360 added for thread 1 sequence 1208 ID 0x36a9a16a dest 1:
Wed Jun 05 01:18:57 2013
RFS[11]: Selected log 5 for thread 1 sequence 1210 dbid 917065834 branch 810292394
Wed Jun 05 01:18:57 2013
Archived Log entry 361 added for thread 1 sequence 1209 ID 0x36a9a16a dest 1:
Wed Jun 05 02:00:19 2013
Thread 1 advanced to log sequence 128 (LGWR switch)
  Current log# 2 seq# 128 mem# 0: /u02/oradata/CORE/CORE/redo02a.log
  Current log# 2 seq# 128 mem# 1: /u03/oradata/CORE/CORE/redo02b.log
Wed Jun 05 02:00:19 2013
Archived Log entry 362 added for thread 1 sequence 127 ID 0x370a7740 dest 1:
Wed Jun 05 02:02:52 2013
RFS[11]: Selected log 4 for thread 1 sequence 1211 dbid 917065834 branch 810292394
Wed Jun 05 02:02:52 2013
Archived Log entry 363 added for thread 1 sequence 1210 ID 0x36a9a16a dest 1:
Wed Jun 05 03:00:00 2013
Clearing Resource Manager plan via parameter
Wed Jun 05 03:59:57 2013
RFS[11]: Selected log 5 for thread 1 sequence 1212 dbid 917065834 branch 810292394
Wed Jun 05 03:59:57 2013
Archived Log entry 364 added for thread 1 sequence 1211 ID 0x36a9a16a dest 1:
Wed Jun 05 05:33:02 2013
RFS[11]: Selected log 4 for thread 1 sequence 1213 dbid 917065834 branch 810292394
Wed Jun 05 05:33:02 2013
Archived Log entry 365 added for thread 1 sequence 1212 ID 0x36a9a16a dest 1:
Wed Jun 05 06:00:49 2013
Thread 1 advanced to log sequence 129 (LGWR switch)
  Current log# 3 seq# 129 mem# 0: /u02/oradata/CORE/CORE/redo03a.log
  Current log# 3 seq# 129 mem# 1: /u03/oradata/CORE/CORE/redo03b.log
Wed Jun 05 06:00:50 2013
Archived Log entry 366 added for thread 1 sequence 128 ID 0x370a7740 dest 1:
Wed Jun 05 07:05:49 2013
RFS[11]: Selected log 5 for thread 1 sequence 1214 dbid 917065834 branch 810292394
Wed Jun 05 07:05:49 2013
Archived Log entry 367 added for thread 1 sequence 1213 ID 0x36a9a16a dest 1:
Wed Jun 05 07:52:47 2013
ALTER SYSTEM SET fal_server='CORE' SCOPE=BOTH;
Wed Jun 05 07:52:47 2013
Archived Log entry 368 added for thread 1 sequence 1214 ID 0x36a9a16a dest 1:
Wed Jun 05 07:52:47 2013
RFS[13]: Assigned to RFS process 53280880
RFS[13]: Identified database type as 'snapshot standby': Client is LGWR ASYNC pid 66912454
RFS[13]: Selected log 4 for thread 1 sequence 1215 dbid 917065834 branch 810292394
Wed Jun 05 07:52:59 2013
RFS[14]: Assigned to RFS process 17105318
RFS[14]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed Jun 05 07:56:30 2013
ALTER SYSTEM SET fal_server='CORE' SCOPE=BOTH;
Wed Jun 05 07:56:30 2013
Archived Log entry 369 added for thread 1 sequence 1215 ID 0x36a9a16a dest 1:
Wed Jun 05 07:59:02 2013
RFS[15]: Assigned to RFS process 52429000
RFS[15]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed Jun 05 07:59:02 2013
ALTER SYSTEM SET fal_server='CORE' SCOPE=BOTH;
Wed Jun 05 07:59:02 2013
RFS[16]: Assigned to RFS process 51314852
RFS[16]: Identified database type as 'snapshot standby': Client is LGWR ASYNC pid 66912454
RFS[16]: Selected log 4 for thread 1 sequence 1216 dbid 917065834 branch 810292394
RFS[16]: Selected log 5 for thread 1 sequence 1217 dbid 917065834 branch 810292394
Wed Jun 05 07:59:02 2013
Archived Log entry 370 added for thread 1 sequence 1216 ID 0x36a9a16a dest 1:
Wed Jun 05 09:31:03 2013
Archived Log entry 371 added for thread 1 sequence 1217 ID 0x36a9a16a dest 1:
Wed Jun 05 11:00:21 2013
Thread 1 advanced to log sequence 130 (LGWR switch)
  Current log# 1 seq# 130 mem# 0: /u02/oradata/CORE/CORE/redo01a.log
  Current log# 1 seq# 130 mem# 1: /u03/oradata/CORE/CORE/redo01b.log
Wed Jun 05 11:00:22 2013
Archived Log entry 372 added for thread 1 sequence 129 ID 0x370a7740 dest 1:
Wed Jun 05 13:06:24 2013
RFS[17]: Assigned to RFS process 58523890
RFS[17]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed Jun 05 13:06:24 2013
RFS[18]: Assigned to RFS process 32571526
RFS[18]: Identified database type as 'snapshot standby': Client is LGWR ASYNC pid 66912454
RFS[18]: Selected log 4 for thread 1 sequence 1220 dbid 917065834 branch 810292394
Wed Jun 05 13:06:25 2013
RFS[19]: Assigned to RFS process 33161356
RFS[19]: Identified database type as 'snapshot standby': Client is ARCH pid 61276214
Wed Jun 05 13:06:25 2013
RFS[20]: Assigned to RFS process 32309400
RFS[20]: Identified database type as 'snapshot standby': Client is ARCH pid 64487628
RFS[19]: Opened log for thread 1 sequence 1218 dbid 917065834 branch 810292394
RFS[18]: Selected log 5 for thread 1 sequence 1221 dbid 917065834 branch 810292394
Wed Jun 05 13:06:25 2013
Archived Log entry 373 added for thread 1 sequence 1220 ID 0x36a9a16a dest 1:
RFS[20]: Opened log for thread 1 sequence 1219 dbid 917065834 branch 810292394
Archived Log entry 374 added for thread 1 sequence 1218 rlc 810292394 ID 0x36a9a16a dest 2:
Archived Log entry 375 added for thread 1 sequence 1219 rlc 810292394 ID 0x36a9a16a dest 2:
Wed Jun 05 13:06:26 2013
RFS[21]: Assigned to RFS process 17957134
RFS[21]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed Jun 05 13:07:08 2013
Shutting down instance (immediate)
Stopping background process SMCO
Shutting down instance: further logons disabled
Stopping background process QMNC
Wed Jun 05 13:07:09 2013
Stopping background process CJQ0
Stopping background process MMNL
Stopping background process MMON
License high water mark = 28
alter database CLOSE NORMAL
Wed Jun 05 13:07:15 2013
SMON: disabling tx recovery
SMON: disabling cache recovery
Wed Jun 05 13:07:15 2013
Shutting down archive processes
Archiving is disabled
Wed Jun 05 13:07:15 2013
ARCH shutting downWed Jun 05 13:07:15 2013

ARCH shutting down
Wed Jun 05 13:07:15 2013
ARCH shutting down
ARC0: Archival stoppedARC1: Archival stopped

ARC2: Archival stopped
Wed Jun 05 13:07:15 2013
ARCH shutting down
ARC3: Archival stopped
Thread 1 closed at log sequence 130
Successful close of redo thread 1
Completed: alter database CLOSE NORMAL
alter database DISMOUNT
Completed: alter database DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Shutting down Data Guard Broker processes
Wed Jun 05 13:07:20 2013
Completed: Data Guard Broker shutdown
Wed Jun 05 13:07:22 2013
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Wed Jun 05 13:07:22 2013
Stopping background process VKTM:
Archive process shutdown avoided: 0 active
Wed Jun 05 13:07:27 2013
Instance shutdown complete
Wed Jun 05 13:07:28 2013
Adjusting the default value of parameter parallel_max_servers
from 400 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_ora_27066652.trc:
ORA-27167: Attempt to determine if Oracle binary image is stored on remote server failed
ORA-27300: OS system dependent operation:parse_df failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: parse failed
ORA-27303: additional information: Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
Global           78643200  51726304   35%   274082     3% /u01/app/incore/product
Image consistency checking encountered an error, checking disabled
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =28
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
Using parameter settings in server-side spfile /u01/app/incore/product/11.2.0.1/dbhome_1/dbs/spfileCORE.ora
System parameters with non-default values:
  processes                = 150
  event                    = "10298 trace name context forever, level 32"
  nls_date_format          = "DD-MON-RRRR"
  nls_length_semantics     = "CHAR"
  filesystemio_options     = "SETALL"
  sga_target               = 4G
  control_files            = "/u01/oradata/CORE/CORE/control01.ctl"
  control_files            = "/u02/oradata/CORE/CORE/control02.ctl"
  control_files            = "/u03/oradata/CORE/CORE/control03.ctl"
  db_file_name_convert     = "/u04/oradata/CORE"
  db_file_name_convert     = "/u04/oradata/CORE/CORE"
  log_file_name_convert    = "/u02/oradata/CORE"
  log_file_name_convert    = "/u02/oradata/CORE/CORE"
  log_file_name_convert    = "/u03/oradata/CORE"
  log_file_name_convert    = "/u03/oradata/CORE/CORE"
  db_block_size            = 8192
  compatible               = "11.2.0.1"
  log_archive_dest_1       = "LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=CORESTD"
  log_archive_dest_2       = "SERVICE=CORE ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=CORE"
  log_archive_dest_state_1 = "ENABLE"
  log_archive_dest_state_2 = "DEFER"
  log_archive_min_succeed_dest= 1
  fal_server               = "CORE"
  log_archive_trace        = 0
  log_archive_config       = "dg_config=(CORESTD,CORE)"
  log_archive_format       = "%t_%s_%r.dbf"
  log_archive_max_processes= 4
  archive_lag_target       = 0
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = "/u05/oradata/CORE/CORE/FLASH"
  db_recovery_file_dest_size= 20G
  standby_file_management  = "AUTO"
  fast_start_mttr_target   = 300
  undo_management          = "AUTO"
  undo_tablespace          = "UNDOTBS_NEW"
  recyclebin               = "ON"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = "in.isol.net"
  global_names             = FALSE
  session_cached_cursors   = 100
  job_queue_processes      = 10
  cursor_sharing           = "SIMILAR"
  db_name                  = "CORE"
  db_unique_name           = "CORESTD"
  open_cursors             = 300
  pga_aggregate_target     = 500M
  dg_broker_start          = TRUE
  diagnostic_dest          = "/u01/app/incore/diag"
Wed Jun 05 13:07:31 2013
PMON started with pid=2, OS id=12845378
Wed Jun 05 13:07:31 2013
VKTM started with pid=3, OS id=30146638 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Wed Jun 05 13:07:31 2013
GEN0 started with pid=4, OS id=50594268
Wed Jun 05 13:07:32 2013
DIAG started with pid=5, OS id=29294606
Wed Jun 05 13:07:32 2013
DBRM started with pid=6, OS id=51511624
Wed Jun 05 13:07:32 2013
PSP0 started with pid=7, OS id=59047986
Wed Jun 05 13:07:32 2013
DIA0 started with pid=8, OS id=22479288
Wed Jun 05 13:07:32 2013
MMAN started with pid=9, OS id=20250820
Wed Jun 05 13:07:33 2013
DBW0 started with pid=10, OS id=24904096
Wed Jun 05 13:07:33 2013
DBW1 started with pid=11, OS id=45809962
Wed Jun 05 13:07:33 2013
DBW2 started with pid=12, OS id=34013688
Wed Jun 05 13:07:34 2013
LGWR started with pid=13, OS id=44040692
Wed Jun 05 13:07:34 2013
CKPT started with pid=14, OS id=35586234
Wed Jun 05 13:07:34 2013
SMON started with pid=15, OS id=46793164
Wed Jun 05 13:07:34 2013
RECO started with pid=16, OS id=32899376
Wed Jun 05 13:07:35 2013
MMON started with pid=17, OS id=41025606
Wed Jun 05 13:07:35 2013
MMNL started with pid=18, OS id=39059622
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Wed Jun 05 13:07:35 2013
DMON started with pid=19, OS id=32309446
Wed Jun 05 13:07:35 2013
alter database  mount
ARCH: STARTING ARCH PROCESSES
Wed Jun 05 13:07:40 2013
ARC0 started with pid=21, OS id=19857764
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Successful mount of redo thread 1, with mount id 924126935
Allocated 15937344 bytes in shared pool for flashback generation buffer
Wed Jun 05 13:07:41 2013
ARC1 started with pid=22, OS id=24969566
Starting background process RVWR
Wed Jun 05 13:07:42 2013
RVWR started with pid=23, OS id=33161234
Database mounted in Exclusive Mode
Lost write protection disabled
Wed Jun 05 13:07:42 2013
ARC2 started with pid=24, OS id=60031080
Wed Jun 05 13:07:42 2013
ARC3 started with pid=25, OS id=27853160
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
ARC2: Becoming the heartbeat ARCH
Completed: alter database  mount
Starting Data Guard Broker (DMON)
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Wed Jun 05 13:07:43 2013
NSV0 started with pid=26, OS id=16646190


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:07:47
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:07:47
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:07:47
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
Wed Jun 05 13:07:47 2013
INSV started with pid=27, OS id=41550292
Wed Jun 05 13:07:48 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
RFS[1]: Assigned to RFS process 24445180
RFS[1]: Identified database type as 'snapshot standby': Client is ARCH pid 48955422
Wed Jun 05 13:07:51 2013
RFS[2]: Assigned to RFS process 15400978
RFS[2]: Identified database type as 'snapshot standby': Client is LGWR ASYNC pid 66912454
RFS[2]: Selected log 5 for thread 1 sequence 1221 dbid 917065834 branch 810292394
RFS[2]: Selected log 4 for thread 1 sequence 1222 dbid 917065834 branch 810292394
Archived Log entry 376 added for thread 1 sequence 1221 ID 0x36a9a16a dest 1:
Wed Jun 05 13:07:52 2013
RSM0 started with pid=31, OS id=7798920
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE','/u04/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE','/u02/oradata/CORE/CORE','/u03/oradata/CORE','/u03/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='CORE' SCOPE=BOTH;
RFS[2]: Selected log 5 for thread 1 sequence 1223 dbid 917065834 branch 810292394
Wed Jun 05 13:07:56 2013
Archived Log entry 377 added for thread 1 sequence 1222 ID 0x36a9a16a dest 1:
alter database convert to physical standby
ALTER DATABASE CONVERT TO PHYSICAL STANDBY (CORE)
krsv_proc_kill: Killing 2 processes (all RFS)
Wed Jun 05 13:08:07 2013
Flashback Restore Start
Wed Jun 05 13:08:47 2013
Flashback Restore Complete
Guaranteed restore point  dropped
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tcjmxg3_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tcvhz71_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8t8x6lrl_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tdv37qd_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8t9o4zsq_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tdvdtfd_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tb70twr_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tb7b0dl_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tbfro8r_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tbj34yj_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tbkbb19_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tc6cox2_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8th1fv09_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8thhhfpc_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8thhrlrj_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8thpjowf_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8thstlbm_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tht14pw_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tjclphq_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tjtqyb0_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tkc75ns_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tksxfb5_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tl40o0q_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tl461g2_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tlbvo91_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tlg914h_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tlgd919_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tlwjl8l_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tm0486x_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tm06frd_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tmgl13t_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tmw43gw_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tn6nj6o_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tnnq2y0_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tnz7m32_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8to0ysdd_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8to2rf98_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tojvfn2_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tonfqp6_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8too749y_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tp2fn8l_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tpjh43w_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tpvy4x3_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tq921j7_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tqmnj28_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tqmnwok_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tqq58wv_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tr56dn8_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8trpss2j_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8ts8c66s_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tspg0vb_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tt13gs7_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tt188wd_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tt835r0_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tt83k7c_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8tth2d8v_.flb
Deleted Oracle managed file /u05/oradata/CORE/CORE/FLASH/CORESTD/flashback/o1_mf_8ttybch8_.flb
Clearing standby activation ID 923432768 (0x370a7740)
The primary database controlfile was created using the
'MAXLOGFILES 32' clause.
There is space for up to 29 standby redo logfiles
Use the following SQL commands on the standby database to create
standby redo logfiles that match the primary database:
ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 10485760;
ALTER DATABASE ADD STANDBY LOGFILE 'srl4.f' SIZE 10485760;
Completed: alter database convert to physical standby
Wed Jun 05 13:08:51 2013
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Stopping background process MMNL
Stopping background process MMON
License high water mark = 5
alter database CLOSE NORMAL
ORA-1109 signalled during: alter database CLOSE NORMAL...
alter database DISMOUNT
Completed: alter database DISMOUNT
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Wed Jun 05 13:08:55 2013
ARCH shutting down
Wed Jun 05 13:08:55 2013
Wed Jun 05 13:08:55 2013
ARCH shutting downARCH shutting down

Wed Jun 05 13:08:55 2013
ARCH shutting down
ARC1: Archival stopped
ARC3: Archival stopped
ARC0: Archival stopped
ARC2: Archival stopped
Shutting down Data Guard Broker processes
Wed Jun 05 13:08:59 2013
Completed: Data Guard Broker shutdown
Wed Jun 05 13:09:01 2013
Stopping background process VKTM:
Wed Jun 05 13:09:01 2013
ARCH: Archival disabled due to shutdown: 1089
Shutting down archive processes
Archiving is disabled
Archive process shutdown avoided: 0 active
Wed Jun 05 13:09:04 2013
Instance shutdown complete
Wed Jun 05 13:09:04 2013
Adjusting the default value of parameter parallel_max_servers
from 400 to 135 due to the value of parameter processes (150)
Starting ORACLE instance (normal)
sskgpgetexecname failed to get name
Errors in file /u01/app/incore/diag/diag/rdbms/corestd/CORE/trace/CORE_ora_51642480.trc:
ORA-27167: Attempt to determine if Oracle binary image is stored on remote server failed
ORA-27300: OS system dependent operation:parse_df failed with status: 2
ORA-27301: OS failure message: No such file or directory
ORA-27302: failure occurred at: parse failed
ORA-27303: additional information: Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on
Global           78643200  51726244   35%   274092     3% /u01/app/incore/product
Image consistency checking encountered an error, checking disabled
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on.
IMODE=BR
ILAT =28
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options.
Using parameter settings in server-side spfile /u01/app/incore/product/11.2.0.1/dbhome_1/dbs/spfileCORE.ora
System parameters with non-default values:
  processes                = 150
  event                    = "10298 trace name context forever, level 32"
  nls_date_format          = "DD-MON-RRRR"
  nls_length_semantics     = "CHAR"
  filesystemio_options     = "SETALL"
  sga_target               = 4G
  control_files            = "/u01/oradata/CORE/CORE/control01.ctl"
  control_files            = "/u02/oradata/CORE/CORE/control02.ctl"
  control_files            = "/u03/oradata/CORE/CORE/control03.ctl"
  db_file_name_convert     = "/u04/oradata/CORE"
  db_file_name_convert     = "/u04/oradata/CORE/CORE"
  log_file_name_convert    = "/u02/oradata/CORE"
  log_file_name_convert    = "/u02/oradata/CORE/CORE"
  log_file_name_convert    = "/u03/oradata/CORE"
  log_file_name_convert    = "/u03/oradata/CORE/CORE"
  db_block_size            = 8192
  compatible               = "11.2.0.1"
  log_archive_dest_1       = "LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=CORESTD"
  log_archive_dest_2       = "SERVICE=CORE ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=CORE"
  log_archive_dest_state_1 = "ENABLE"
  log_archive_dest_state_2 = "DEFER"
  log_archive_min_succeed_dest= 1
  fal_server               = "CORE"
  log_archive_trace        = 0
  log_archive_config       = "dg_config=(CORESTD,CORE)"
  log_archive_format       = "%t_%s_%r.dbf"
  log_archive_max_processes= 4
  archive_lag_target       = 0
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = "/u05/oradata/CORE/CORE/FLASH"
  db_recovery_file_dest_size= 20G
  standby_file_management  = "AUTO"
  fast_start_mttr_target   = 300
  undo_management          = "AUTO"
  undo_tablespace          = "UNDOTBS_NEW"
  recyclebin               = "ON"
  remote_login_passwordfile= "EXCLUSIVE"
  db_domain                = "in.isol.net"
  global_names             = FALSE
  session_cached_cursors   = 100
  job_queue_processes      = 10
  cursor_sharing           = "SIMILAR"
  db_name                  = "CORE"
  db_unique_name           = "CORESTD"
  open_cursors             = 300
  pga_aggregate_target     = 500M
  dg_broker_start          = TRUE
  diagnostic_dest          = "/u01/app/incore/diag"
Wed Jun 05 13:09:06 2013
PMON started with pid=2, OS id=35586236
Wed Jun 05 13:09:06 2013
VKTM started with pid=3, OS id=46596124 at elevated priority
VKTM running at (10)millisec precision with DBRM quantum (100)ms
Wed Jun 05 13:09:07 2013
GEN0 started with pid=4, OS id=43712930
Wed Jun 05 13:09:07 2013
DIAG started with pid=5, OS id=20250830
Wed Jun 05 13:09:08 2013
DBRM started with pid=6, OS id=50594280
Wed Jun 05 13:09:08 2013
PSP0 started with pid=7, OS id=27721986
Wed Jun 05 13:09:08 2013
DIA0 started with pid=8, OS id=17629500
Wed Jun 05 13:09:09 2013
MMAN started with pid=9, OS id=13172778
Wed Jun 05 13:09:09 2013
DBW0 started with pid=10, OS id=33948118
Wed Jun 05 13:09:09 2013
DBW1 started with pid=11, OS id=27853176
Wed Jun 05 13:09:10 2013
DBW2 started with pid=12, OS id=8651246
Wed Jun 05 13:09:10 2013
LGWR started with pid=13, OS id=24904138
Wed Jun 05 13:09:10 2013
CKPT started with pid=14, OS id=49217814
Wed Jun 05 13:09:10 2013
SMON started with pid=15, OS id=7798968
Wed Jun 05 13:09:11 2013
RECO started with pid=16, OS id=25231746
Wed Jun 05 13:09:11 2013
MMON started with pid=17, OS id=41877722
Wed Jun 05 13:09:11 2013
MMNL started with pid=18, OS id=8585440
ORACLE_BASE not set in environment. It is recommended
that ORACLE_BASE be set in the environment
Wed Jun 05 13:09:12 2013
DMON started with pid=19, OS id=17498618
Wed Jun 05 13:09:12 2013
alter database  mount
ARCH: STARTING ARCH PROCESSES
Wed Jun 05 13:09:16 2013
ARC0 started with pid=21, OS id=45809972
ARC0: Archival started
ARCH: STARTING ARCH PROCESSES COMPLETE
ARC0: STARTING ARCH PROCESSES
Successful mount of redo thread 1, with mount id 924113208
Allocated 15937344 bytes in shared pool for flashback generation buffer
Wed Jun 05 13:09:17 2013
ARC1 started with pid=22, OS id=24969366
Starting background process RVWR
Wed Jun 05 13:09:18 2013
RVWR started with pid=23, OS id=18612506
Wed Jun 05 13:09:18 2013
ARC2 started with pid=24, OS id=47120860
Physical Standby Database mounted.
Lost write protection disabled
Wed Jun 05 13:09:18 2013
ARC3 started with pid=25, OS id=58458504
ARC1: Archival started
ARC2: Archival started
ARC1: Becoming the 'no FAL' ARCH
ARC2: Becoming the heartbeat ARCH
Completed: alter database  mount
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Starting Data Guard Broker (DMON)
Wed Jun 05 13:09:20 2013
NSV0 started with pid=26, OS id=19857836


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:09:24
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:09:24
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0


***********************************************************************

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=ind-del-db01-s)(PORT=1522)))(CONNECT_DATA=(SERVICE_NAME=CORE_DGB.in.isol.net)(CID=(PROGRAM=oracle)(HOST=ind-del-db02-s)(USER=incore))))

  VERSION INFORMATION:
        TNS for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
        TCP/IP NT Protocol Adapter for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - Production
  Time: 05-JUN-2013 13:09:24
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564

TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
Wed Jun 05 13:09:24 2013
INSV started with pid=27, OS id=29294728
Wed Jun 05 13:09:25 2013
Using STANDBY_ARCHIVE_DEST parameter default value as USE_DB_RECOVERY_FILE_DEST
RFS[1]: Assigned to RFS process 24445078
RFS[1]: Identified database type as 'physical standby': Client is ARCH pid 48955422
Wed Jun 05 13:09:27 2013
RFS[2]: Assigned to RFS process 60031170
RFS[2]: Identified database type as 'physical standby': Client is LGWR ASYNC pid 66912454
Primary database is in MAXIMUM PERFORMANCE mode
RFS[2]: Selected log 5 for thread 1 sequence 1223 dbid 917065834 branch 810292394
RFS[2]: Selected log 4 for thread 1 sequence 1224 dbid 917065834 branch 810292394
Archived Log entry 378 added for thread 1 sequence 1223 ID 0x36a9a16a dest 1:
Wed Jun 05 13:09:30 2013
RSM0 started with pid=31, OS id=24313888
ALTER SYSTEM SET log_archive_trace=0 SCOPE=BOTH SID='CORE';
ALTER SYSTEM SET log_archive_format='%t_%s_%r.dbf' SCOPE=SPFILE SID='CORE';
ALTER SYSTEM SET standby_file_management='AUTO' SCOPE=BOTH SID='*';
ALTER SYSTEM SET archive_lag_target=0 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_max_processes=4 SCOPE=BOTH SID='*';
ALTER SYSTEM SET log_archive_min_succeed_dest=1 SCOPE=BOTH SID='*';
ALTER SYSTEM SET db_file_name_convert='/u04/oradata/CORE','/u04/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET log_file_name_convert='/u02/oradata/CORE','/u02/oradata/CORE/CORE','/u03/oradata/CORE','/u03/oradata/CORE/CORE' SCOPE=SPFILE;
ALTER SYSTEM SET fal_server='CORE' SCOPE=BOTH;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (CORE)
Wed Jun 05 13:09:35 2013
MRP0 started with pid=32, OS id=54788452
MRP0: Background Managed Standby Recovery process started (CORE)
 started logmerger process
Wed Jun 05 13:09:41 2013
Managed Standby Recovery starting Real Time Apply
Wed Jun 05 13:09:48 2013
Parallel Media Recovery started with 20 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Clearing online redo logfile 1 /u02/oradata/CORE/CORE/redo01a.log
Clearing online log 1 of thread 1 sequence number 130
Clearing online redo logfile 1 complete
Clearing online redo logfile 2 /u02/oradata/CORE/CORE/redo02a.log
Clearing online log 2 of thread 1 sequence number 1223
Clearing online redo logfile 2 complete
Clearing online redo logfile 3 /u02/oradata/CORE/CORE/redo03a.log
Clearing online log 3 of thread 1 sequence number 1224
Wed Jun 05 13:09:51 2013
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE  THROUGH ALL SWITCHOVER DISCONNECT  USING CURRENT LOGFILE
Clearing online redo logfile 3 complete
Wed Jun 05 13:09:51 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1028_8tco0lxt_.arc
Wed Jun 05 13:09:52 2013
Archived Log entry 379 added for thread 1 sequence 1224 ID 0x36a9a16a dest 1:
Wed Jun 05 13:09:52 2013
RFS[2]: Selected log 4 for thread 1 sequence 1225 dbid 917065834 branch 810292394
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1029_8tcozrff_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1030_8tcvrb28_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1031_8td2hxvn_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1032_8td8y9tl_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1033_8tdgktng_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1034_8tdn49fl_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1035_8tdv3rco_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1036_8tdvbgb3_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1037_8tdvd5dv_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1038_8tdvfhl1_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1039_8tdvh5v2_.arc
Wed Jun 05 13:10:03 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1040_8tdvhtwc_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_29/o1_mf_1_1041_8tdvjlrl_.arc
Wed Jun 05 13:10:05 2013
RFS[3]: Assigned to RFS process 20054058
RFS[3]: Identified database type as 'physical standby': Client is ARCH pid 48955422
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1042_8tdyrk2y_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1043_8tf4k2xq_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1044_8tf4k5gg_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1045_8tf4k90j_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1046_8tf4kd2d_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1047_8tf4kh14_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1048_8tf4kl12_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1049_8tf5oz84_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1050_8tfd9zcd_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1051_8tfky48f_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1052_8tfqr23k_.arc
Wed Jun 05 13:10:13 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1053_8tfy7n6j_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1054_8tg4ygk5_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1055_8tgblnhz_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1056_8tgj168c_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1057_8tgpw98t_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1058_8tgx1rbz_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1059_8th2npr9_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1060_8th8fwfs_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1061_8thhhbtq_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1062_8thhqs91_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1063_8thhsof3_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1064_8thhvlmd_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1065_8thhwsg8_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1066_8thhydpt_.arc
Wed Jun 05 13:10:23 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_30/o1_mf_1_1067_8thhzzr6_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1068_8thop32s_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1069_8thrzmlz_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1070_8thrznkx_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1071_8thrzolb_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1072_8thrzrjq_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1073_8thrzvj6_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1074_8thrzyny_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1075_8thxlbf9_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1076_8tj2w07x_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1077_8tj8of4r_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1078_8tjh41rk_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1079_8tjopybx_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1080_8tjv46pg_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1081_8tk0sqw7_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1082_8tk7qbs5_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1083_8tkflvqd_.arc
Wed Jun 05 13:10:33 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1084_8tkm1jn9_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1085_8tks92jl_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1086_8tl09omx_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1087_8tl41qlo_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1088_8tl45b4t_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1089_8tl4790x_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1090_8tl48pb9_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1091_8tl49c97_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1092_8tl4blpy_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1093_8tl4cbll_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_05_31/o1_mf_1_1094_8tl6gwsc_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1095_8tlc51nl_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1096_8tlfg0hc_.arc
Wed Jun 05 13:10:44 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1097_8tlfg1fx_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1098_8tlfg322_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1099_8tlfg5z3_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1100_8tlfg90b_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1101_8tlfgh11_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1102_8tlkkks6_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1103_8tlpthc6_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1104_8tlwnd18_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1105_8tm04wn9_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1106_8tm065fg_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1107_8tm06o9o_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1108_8tm06y8l_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1109_8tm07b82_.arc
Wed Jun 05 13:10:54 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1110_8tm07zb4_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1111_8tm3hqdj_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1112_8tm8hkd5_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1113_8tmg280n_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1114_8tmgcq9k_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1115_8tmo1x20_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1116_8tmtws40_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1117_8tmyr11b_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1118_8tn3p7jo_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1119_8tnb4k83_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1120_8tnfof5r_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1121_8tnnqf6m_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_01/o1_mf_1_1122_8tnrknz6_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1123_8tny2myk_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1124_8to03yn4_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1125_8to0408d_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1126_8to042oh_.arc
Wed Jun 05 13:11:05 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1127_8to045o0_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1128_8to048ns_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1129_8to0cvod_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1130_8to69lvc_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1131_8toclwkn_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1132_8tok8goo_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1133_8tongsh0_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1134_8tonhq6z_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1135_8tonj5dw_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1136_8tonjn8s_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1137_8tonk98z_.arc
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1138_8tonkod3_.arc
Wed Jun 05 13:11:15 2013
Media Recovery Log /u05/oradata/CORE/CORE/FLASH/CORESTD/archivelog/2013_06_02/o1_mf_1_1139_8topcr0p_.arc
/u01/app/incore/diag/diag/rdbms/corestd/CORE/trace =&gt;

Data Guard Broker Setup 11g

Posted: May 28, 2013 in StandBy
Tags: broker, Dataguard Broker, DGMGRL

In this post i will setup dataguard broker services for managing dataguard setup. Follwing are prerequisites for this setup:

1) Fully functional Primary and Standby setup
2) Spfile must be in use at primary and standby database
3) DG_BROKER_START must be set TRUE at both primary and standby
4) DG_BROKER_CONFIG_FILEn parameter is optional
5) GLOBAL_DBNAME must be set in listener.ora with predefined format

My assumption in this post are Primary database name is CORE and standby database name is CORESTD. Standby setup is running fine. Among the above requirements, we have already fulfilled the requirements 1 – 4 during our init.ora parameters settings. Following is the entry for GLOBAL_DBNAME in listener.ora file of both the primary and standby databases. This is a basic requirement for broker setup and following is example from the standby database. Make necessary change at primary DB listener too, only host name will be changed in most of the cases.

SID_LIST_LISTENER_CORE =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/incore/product/11.2.0.1/dbhome_1)
      (PROGRAM = extproc)
    )
   (SID_DESC =
      (GLOBAL_DBNAME = CORE)
      (ORACLE_HOME = /u01/app/incore/product/11.2.0.1/dbhome_1)
      (SID_NAME = CORE)
    )
   (SID_DESC =
      (GLOBAL_DBNAME = CORE_DGMGRL.in.isol.net)
      (ORACLE_HOME = /u01/app/incore/product/11.2.0.1/dbhome_1)
      (SID_NAME = CORE)
    )
 )
LISTENER_CORE =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS_LIST =
        (ADDRESS = (PROTOCOL = TCP)(HOST = ind-del-db02-s)(PORT = 1522))
      )
    )
  )

Add the standby redo logfiles at both primary and standby side, if they are not already added. Once these are added, we are ready for our broker setup. use “DGMGRL” prompt for broker setup.

incore@ind-del-db01-s $ dgmgrl
DGMGRL for IBM/AIX RISC System/6000: Version 11.2.0.1.0 - 64bit Production

Copyright (c) 2000, 2009, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.
DGMGRL> connect sys@core
Password:
Connected.
DGMGRL>
DGMGRL>
DGMGRL> create configuration 'DRConfig' as primary database is 'CORE' connect identifier is 'CORE';
Configuration "DRConfig" created with primary database "CORE"
DGMGRL>
DGMGRL>
DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE - Primary database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED

DGMGRL> add database 'CORESTD' as connect identifier is 'CORESTD';
Database "CORESTD" added
DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE    - Primary database
    CORESTD - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
DISABLED

DGMGRL> enable configuration;
Enabled.
DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE    - Primary database
    CORESTD - Physical standby database
      Error: ORA-16664: unable to receive the result from a database

Fast-Start Failover: DISABLED

Configuration Status:
ERROR

As the configuration status is error, we need to resolve this issue before we can use broker to manage our standby. Use the following command to see the error in Standby configuration.

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       3 minutes 45 seconds
  Real Time Query: OFF
  Instance(s):
    CORE
      Warning: ORA-16714: the value of property ArchiveLagTarget is inconsistent with the database setting
      Warning: ORA-16714: the value of property LogArchiveMaxProcesses is inconsistent with the database setting
      Warning: ORA-16714: the value of property LogArchiveMinSucceedDest is inconsistent with the database setting
      Warning: ORA-16714: the value of property LogArchiveTrace is inconsistent with the database setting
      Warning: ORA-16714: the value of property LogArchiveFormat is inconsistent with the database setting

  Database Warning(s):
    ORA-16826: apply service state is inconsistent with the DelayMins property

  Properties:
    DGConnectIdentifier             = 'CORESTD'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
WARNING

DGMGRL> edit database 'CORESTD' set property 'ArchiveLagTarget'=0;
Property "ArchiveLagTarget" updated
DGMGRL>
DGMGRL> edit database 'CORESTD' set property 'LogArchiveMaxProcesses'=4;
Property "LogArchiveMaxProcesses" updated
DGMGRL> edit database 'CORESTD' set property 'LogArchiveMinSucceedDest'=1;
Property "LogArchiveMinSucceedDest" updated
DGMGRL> edit database 'CORESTD' set property 'LogArchiveTrace'=0;
Property "LogArchiveTrace" updated
DGMGRL>
DGMGRL> edit database 'CORESTD' set property 'LogArchiveFormat'='%t_%s_%r.dbf';
Property "LogArchiveFormat" updated
DGMGRL>
DGMGRL> edit database 'CORESTD' set property 'DelayMins'=0;
Property "DelayMins" updated

Once, you corrected all the warning messages, now command should return the with success message.

DGMGRL> show database verbose 'CORESTD';

Database - CORESTD

  Role:            PHYSICAL STANDBY
  Intended State:  APPLY-ON
  Transport Lag:   0 seconds
  Apply Lag:       0 seconds
  Real Time Query: OFF
  Instance(s):
    CORE

  Properties:
    DGConnectIdentifier             = 'CORESTD'
    ObserverConnectIdentifier       = ''
    LogXptMode                      = 'ASYNC'
    DelayMins                       = '0'
    Binding                         = 'OPTIONAL'
    MaxFailure                      = '0'
    MaxConnections                  = '1'
    ReopenSecs                      = '300'
    NetTimeout                      = '30'
    RedoCompression                 = 'DISABLE'
    LogShipping                     = 'ON'
    PreferredApplyInstance          = ''
    ApplyInstanceTimeout            = '0'
    ApplyParallel                   = 'AUTO'
    StandbyFileManagement           = 'AUTO'
    ArchiveLagTarget                = '0'
    LogArchiveMaxProcesses          = '4'
    LogArchiveMinSucceedDest        = '1'
    DbFileNameConvert               = '/u04/oradata/CORE, /u04/oradata/CORE/CORE'
    LogFileNameConvert              = '/u02/oradata/CORE, /u02/oradata/CORE/CORE, /u03/oradata/CORE, /u03/oradata/CORE/CORE'
    FastStartFailoverTarget         = ''
    StatusReport                    = '(monitor)'
    InconsistentProperties          = '(monitor)'
    InconsistentLogXptProps         = '(monitor)'
    SendQEntries                    = '(monitor)'
    LogXptStatus                    = '(monitor)'
    RecvQEntries                    = '(monitor)'
    HostName                        = 'ind-del-db02-s'
    SidName                         = 'CORE'
    StaticConnectIdentifier         = '(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ind-del-db02-s)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=CORESTD_DGMGRL.in.isol.net)(INSTANCE_NAME=CORE)(SERVER=DEDICATED)))'
    StandbyArchiveLocation          = 'USE_DB_RECOVERY_FILE_DEST'
    AlternateLocation               = ''
    LogArchiveTrace                 = '0'
    LogArchiveFormat                = '%t_%s_%r.dbf'
    TopWaitEvents                   = '(monitor)'

Database Status:
SUCCESS

Now, check the configuration, it should be success.

DGMGRL> show configuration;

Configuration - DRConfig

  Protection Mode: MaxPerformance
  Databases:
    CORE    - Primary database
    CORESTD - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:
SUCCESS

Now, we are ready to use the broker service for managing our standby by setup. To disable the archive shipping to standby we use the following command in SQL:

SQL> alter system set log_archive_dest_state_2='DEFER' scope=both;

System altered.

Now, we will be using below command for disabling the archive shipping.

DGMGRL> edit database 'CORE' set state=transport-off;
Succeeded.
DGMGRL>

Similarly, to cancel the managed mode of standby database use the below command.

DGMGRL> connect sys@corestd
Password:
Connected.
DGMGRL>
DGMGRL>
DGMGRL> edit database 'CORESTD' set state='APPLY-OFF';
Succeeded.

Standby Recovery To A Particular SCN/Time

Posted: March 24, 2013 in StandBy
Tags: Physical Standby, standby by recovery, standby recovery to a particular SCN

In this post, i will be recovering a standby to a particular SCN/Time to recover from a truncate table situation when we have a Physical standby implemented with some time delay between archive shipping and getting applied. I have a time delay of 120 Minutes.
Below are the steps i followed on PRIMARY database. Well, original blog from which i got these steps is of laurentschneider

SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
A_OB                           TABLE

SQL>
SQL> create table A_AB as select * from dba_objects;

Table created.

SQL> select count(*) from A_AB;

  COUNT(*)
----------
     13798

SQL> commit;

Commit complete.

SQL>
SQL>
SQL> alter system switch logfile;

System altered.

SQL>
SQL> select current_scn from v$database;

CURRENT_SCN
-----------
   21661731

SQL> select systimestamp from dual;

SYSTIMESTAMP
---------------------------------------------------------------------------
23-MAR-13 10.12.56.196491 AM +01:00

SQL> truncate table A_AB;

Table truncated.

SQL> select current_scn from v$database;

CURRENT_SCN
-----------
   21661836

SQL> select systimestamp from dual;

SYSTIMESTAMP
---------------------------------------------------------------------------
23-MAR-13 10.13.28.984307 AM +01:00

SQL> alter system switch logfile;

System altered.

SQL> select count(*) from A_AB;

  COUNT(*)
----------
         0

SQL> alter system switch logfile;

System altered.

SQL> alter system switch logfile;

System altered.

I have created table A_AB at 23-MAR-13 10:12:56 with SCN 21661731 and truncated by 10:13:38 with SCN 21661836. SO, i will be applying archived logs at standby till 10:12:56. At standby database, i have followed the following steps:

SQL> alter database recover managed standby database cancel;

Database altered.

SQL> ALTER DATABASE RECOVER automatic standby database until time '2013-03-23 10:12:56';

Database altered.

Following is the output from standby alert.log file

Managed Standby Recovery Canceled (PLMQDBS)
Completed: alter database recover managed standby database cancel
Sat Mar 23 10:19:31 2013
ALTER DATABASE RECOVER automatic standby database until time '2013-03-23 10:12:56'
Media Recovery Start
 started logmerger process
Sat Mar 23 10:19:31 2013
Managed Standby Recovery not using Real Time Apply
Parallel Media Recovery started with 8 slaves
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3272_8ntv43gt_.arc
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3273_8ntv43ht_.arc
Sat Mar 23 10:19:44 2013
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3274_8ntv43kb_.arc
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3275_8ntv6d5h_.arc
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3276_8ntv6dyj_.arc
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3277_8ntvmv0g_.arc
Media Recovery Log /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3278_8ntwg1fl_.arc
Sat Mar 23 10:19:48 2013
Incomplete Recovery applied until change 21661756 time 03/23/2013 10:12:56
Media Recovery Complete (PLMQDBS)
Completed: ALTER DATABASE RECOVER automatic standby database until time '2013-03-23 10:12:56'
Sat Mar 23 10:23:45 2013

Now, i just need to check the contents of table A_AB. I have converted standby into Snapshot standby and then back to physical standby.

SQL> select * from v$restore_point;

no rows selected

SQL> alter database convert to snapshot standby;

Database altered.

SQL> select * from v$restore_point;

       SCN DATABASE_INCARNATION# GUA STORAGE_SIZE TIME                             RESTORE_POINT_TIME  PRE NAME
---------- --------------------- --- ------------ -------------------------------- ------------------- --- ------------------------------------
  21661755                     1 YES     15941632 23-MAR-13 10.23.45.000000000 AM  10:23:45            YES SNAPSHOT_STANDBY_REQUIRED_03/23/2013 

SQL> shutdown immediate;
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area 4175568896 bytes
Fixed Size                  2213384 bytes
Variable Size             805308920 bytes
Database Buffers         3355443200 bytes
Redo Buffers               12603392 bytes
Database mounted.
Database opened.
SQL> conn backup/backup
Connected.
SQL> select * from tab;

TNAME                          TABTYPE  CLUSTERID
------------------------------ ------- ----------
A_AB                           TABLE
A_OB                           TABLE

SQL> select count(*) from A_AB;

  COUNT(*)
----------
     13798

SQL> select count(*) from A_OB;

  COUNT(*)
----------
     13795

SQL> conn sys as sysdba
Enter password:
Connected.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 4175568896 bytes
Fixed Size                  2213384 bytes
Variable Size             805308920 bytes
Database Buffers         3355443200 bytes
Redo Buffers               12603392 bytes
Database mounted.
SQL> alter database convert to physical standby;

Database altered.

SQL> shutdown immediate;
ORA-01507: database not mounted


ORACLE instance shut down.
SQL> startup mount;
ORACLE instance started.

Total System Global Area 4175568896 bytes
Fixed Size                  2213384 bytes
Variable Size             805308920 bytes
Database Buffers         3355443200 bytes
Redo Buffers               12603392 bytes
Database mounted.
SQL> alter database recover managed standby database disconnect from session;

Database altered.

SQL>

Below are lines from alert.log file of standby database.

alter database convert to snapshot standby
Starting background process RVWR
Sat Mar 23 10:23:45 2013
RVWR started with pid=24, OS id=29360282
Allocated 15937344 bytes in shared pool for flashback generation buffer
Created guaranteed restore point SNAPSHOT_STANDBY_REQUIRED_03/23/2013 10:23:45
krsv_proc_kill: Killing 2 processes (all RFS)
Sat Mar 23 10:23:48 2013
Deleted Oracle managed file /u05/oradata/CORE/flash/PLMQDBST/archivelog/2013_03_23/o1_mf_1_3279_8ntwl8sd_.arc
Begin: Standby Redo Logfile archival
End: Standby Redo Logfile archival
RESETLOGS after incomplete recovery UNTIL CHANGE 21661756
Resetting resetlogs activation ID 165702244 (0x9e06a64)
Online log /u02/oradata/CORE/PLMQDBS/redo01a.log: Thread 1 Group 1 was previously cleared
Online log /u03/oradata/CORE/PLMQDBS/redo01b.log: Thread 1 Group 1 was previously cleared
Online log /u02/oradata/CORE/PLMQDBS/redo02a.log: Thread 1 Group 2 was previously cleared
Online log /u03/oradata/CORE/PLMQDBS/redo02b.log: Thread 1 Group 2 was previously cleared
Online log /u02/oradata/CORE/PLMQDBS/redo03a.log: Thread 1 Group 3 was previously cleared
Online log /u03/oradata/CORE/PLMQDBS/redo03b.log: Thread 1 Group 3 was previously cleared
Standby became primary SCN: 21661754
Sat Mar 23 10:23:49 2013
Setting recovery target incarnation to 2
WARNING: STANDBY_FILE_MANAGEMENT initialization parameter is
not set to the value "AUTO".
This may cause recovery of the standby database to terminate
prior to applying all available redo data.
It may be necessary to use the ALTER DATABASE CREATE DATAFILE
command to add datafiles created on the primary database.
CONVERT TO SNAPSHOT STANDBY: Complete - Database mounted as snapshot standby
Completed: alter database convert to snapshot standby

ORA-19573: cannot obtain exclusive enqueue for datafile 1

Posted: March 17, 2013 in StandBy
Tags: exclusive enqueue, Incremental backup apply, ORA-19573

I got the following error while applying the incremental backup to the standby database “ORA-19573: cannot obtain exclusive enqueue for datafile 1”. This was very annoying as i have cross checked everything many times. Here is few lines from RMAN sessions from which applying incremental backup:

channel ORA_DISK_4: reading from backup piece /oraclebackup/PLMQDBS/inc/incr_for_stdby_f7o3u54k_1_1 RMAN-00571: =========================================================== RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS =============== RMAN-00571: =========================================================== RMAN-03002: failure of recover command at 03/07/2013 12:39:20 ORA-19870: error while restoring backup piece /oraclebackup/PLMQDBS/inc/incr_for_stdby_f4o3u54k_1_1 ORA-19573: cannot obtain exclusive enqueue for datafile 1

Well, error was because of very silly mistake, i forgot to cancel the media recovery process before starting the RMAN incremental apply session. Once i have cancelled the recovery process as below, incremental backup ran fine.

oracore@myhost $ sqlplus /nolog

SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 7 12:41:16 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

SQL> conn sys as sysdba
Enter password:
Connected.
SQL> select name from v$database;

NAME
---------
PLMQDBS

SQL> select open_mode from v$database;

OPEN_MODE
--------------------
MOUNTED

SQL> alter database recover managed standby database cancel;

Database altered.

what a small mistake !!!!!!!!

ORASteps

Recent Posts

Archives

Categories

Blogs I Follow

Follow Blog via Email