GoldenGate Integrated Capture Mode
The Integrated Capture GoldenGate Mode also known as the Integrated Extract Goldengate process in 12c (also backported to 11gr2) is one of the more interesting and useful feature released with this version. This capture process is the component responsible for extracting the DML transactional data and DDL’s from the source database redo log files. This data is then written to local trail files which eventually is move to the destination database to be applied there.
• What is the GoldenGate Integrated Capture Mode?
• On-Source Capture
• Downstream Capture
• Prerequisites
• Configuration
• Monitoring/Views
What is the GoldenGate Integrated Capture Mode?
Integrated Capture Mode (IC) is a new form of the Extract process, were in this process is moved closer, inside the source database. In the traditional Classic extract process, the extract works on the redo logs outside the domain of the actual database. In this new integrated capture mode, a server Log Miner process is started which extracts all the DML data and DDLS statements creating Logical Change Records (LCR’s). These are then handed to the Goldengate memory processes which writes these LCR’s to the local trail files. This Log Miner server process is not the Log Miner utility we are used to in the database but is a similar mechanism which has been tuned and enhanced for specific use by the Goldengate processes.
The purpose of moving this inside the database is to be able to make use of the already existing internal procedures in the database, making it easier to provide support for the newer features of Oracle faster than was previously possible. Due to this change, Oracle is now able to provide the following.
• Full Support of Basic, OLTP and EHCC compressed data.
• No need to fetch LOB’s from tables.
• Full Secure File support for Secure file lobs.
• Full XML support.
• Automatically handles addition of nodes and threads in RAC environment.
• Senses node up down in RAC and handles it in its processes transparently.
Integrated Capture Modes
Integration capture supports two types of deployment configurations. They are:
• On-Source Capture
• Downstream Capture
On-Source Capture
When the integrated capture process is configured using the on-source capture mode, the capture process is started on the actual source database server itself. Changes as they happen on source database will be captured locally, routed, transformed and applied on target database in near real-time
This may seem convenient but consideration needs to be given to the additional workload that will be placed by this process on the database server. However if real-time replication is required this is the best option.
Note: All features are supported in both On-Source or Downstream Deployment
Downstream Capture
In the downstream mode, the capture process is configured to run on a remote database host. All the database redo logs from the source database are shipped to this remote server using Dataguard technology and then mined there by the capture process.
In this mode there is an inherent latency introduced due to the fact that the redo log on the source needs to switch first before the log can be shipped downstream. So there will be some delay in the replication of data to a target database as the extraction will be delayed due to the log switch. The main benefit of this setup however is the offset of the resource usage on the source server.
In this mode, to overcome the log switch latency, Oracle has provide a near Real time capture using Standby redo logs for extraction. In this configuration the redo log from the source continuously writes into the standby redo logs of the downstream database. The capture process directly capture the data from here.
It is important to keep in mind when deciding whether to use the Integrated capture or the classic capture mechanism that both configuration will remain available in future releases. However Oracle recommends to use the new Integrated capture mechanism as Oracle will not be adding new features to classic capture in the future and it will only be there for legacy support purposes.
Prerequisites
The database where integrated capture runs:
• Must be at least 11.2.0..3
• Database patch 1411356.1 must be installed.
• Works with Oracle 12c.
In Downstream Configuration:
• DBID’s of ALL source databases must be unique.
• Downstream capture must have same OS/platform as source.
• Standby redo logs must be as large as the largest source redo logs.
• Only one database can have real time mining
Configuration:
Add Extract
On the source database the capture is created by first adding the capture parameter file EXT.prm in the dirprm directory.
gg>cat EXT.prm
EXTRACT EXT
USERID gguser, PASSWORD gguser
EXTTRAIL ./dirdat/xi
TRANLOGOPTIONS EXCLUDEUSER gguser
TABLE EQ2.*;
Next add the capture as an integrated capture specifying the “INTEGRATED TRANLOG” option.
gg> ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.3 14400833 OGGCORE_11.2.1.0.3_PLATFORMS_120823.1258
...
GGSCI (xgoldengate01) 2> ADD EXTRACT EXT, INTEGRATED TRANLOG, BEGIN NOW
EXTRACT added.
GGSCI (xgoldengate01) 3> ADD EXTTRAIL ./dirdat/xn EXTRACT EXT, megabytes 100
EXTTRAIL added.
GGSCI (xgoldengate01) 4> start extract EXT
Sending START request to MANAGER ...
Here is the entries in the ggserror.log when the extract was created.
2013-04-25 17:18:54 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): ADD EXTRACT EXT INTEGRATED TRANLOG, BEGIN NOW.
2013-04-25 17:19:22 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): ADD EXTTRAIL ./dirdat/xn EXTRACT EXT megabytes 100.
2013-04-25 17:19:39 INFO OGG-00987 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (oracle): start extract EXT.
2013-04-25 17:19:39 INFO OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from GGSCI on host gg.vst.com:35534 (START EXTRACT EXT ).
2013-04-25 17:19:40 INFO OGG-00992 Oracle GoldenGate Capture for Oracle, EXT.prm: EXTRACT EXT starting.
2013-04-25 17:19:40 INFO OGG-03035 Oracle GoldenGate Capture for Oracle, EXT.prm: Operating system character set identified as US-ASCII. Locale: en_US_POSIX, LC_ALL: C.
2013-04-25 17:19:40 INFO OGG-03500 Oracle GoldenGate Capture for Oracle, EXT.prm: WARNING: NLS_LANG environment variable does not match database character set, or not set. Using database character set value of AL32UTF8.
2013-04-25 17:19:40 INFO OGG-01635 Oracle GoldenGate Capture for Oracle, EXT.prm: BOUNDED RECOVERY: reset to initial or altered checkpoint.
2013-04-25 17:19:40 INFO OGG-01815 Oracle GoldenGate Capture for Oracle, EXT.prm: Virtual Memory Facilities for: BR
anon alloc: mmap(MAP_ANON) anon free: munmap
file alloc: mmap(MAP_SHARED) file free: munmap
target directories:
/u01/app/ggs/BR/EXT.
2013-04-25 17:19:41 INFO OGG-01815 Oracle GoldenGate Capture for Oracle, EXT.prm: Virtual Memory Facilities for: COM
anon alloc: mmap(MAP_ANON) anon free: munmap
file alloc: mmap(MAP_SHARED) file free: munmap
target directories:
/u01/app/ggs/dirtmp.
2013-04-25 17:19:43 WARNING OGG-01842 Oracle GoldenGate Capture for Oracle, EXT.prm: CACHESIZE PER DYNAMIC DETERMINATION (2G) LESS THAN RECOMMENDED: 64G (64bit system)
vm found: 3.82G
Check swap space. Recommended swap/extract: 128G (64bit system).
Monitoring
There are several components of the Integrated Capture Mode all of which need to be monitored for the effective tuning and troubleshooting of the Replication. Since the extract process mostly resides in the database, the Goldengate, Capture and the Logminer statistics integrated capture monitoring views can be used to view the progress of the extract process. The main components to keep an eye on are below.
• Capture Processes configured in the database.
• Dynamic stats of the Goldengate capture process
• Logminer performance
• Outbound progress table
DBA_CAPTURE
col CAPTURE_NAME for a20;
col QUEUE_NAME for a15;
col START_SCN for 9999999999;
col STATUS for a10;
col CAPTURED_SCN for 9999999999;
col APPLIED_SCN for 9999999999;
col SOURCE_DATABASE for a10;
col LOGMINER_ID for 9999999;
col REQUIRED_CHECKPOINTSCN for a30;
col STATUS_CHANGE_TIME for a15;
col ERROR_NUMBER for a15;
col ERROR_MESSAGE for a10;
col START_TIME for a30
col CAPTURE_TYPE for a10;
SELECT CAPTURE_NAME, QUEUE_NAME, START_SCN, STATUS,
CAPTURED_SCN, APPLIED_SCN, SOURCE_DATABASE,
LOGMINER_ID, REQUIRED_CHECKPOINT_SCN,
STATUS_CHANGE_TIME, ERROR_NUMBER, ERROR_MESSAGE,
CAPTURE_TYPE, START_TIME
FROM DBA_CAPTURE;
GOLDENGATE CAPTURE/Trans
col state for a30;
SELECT sid, serial#, capture#, CAPTURE_NAME, STARTUP_TIME, CAPTURE_TIME,
state, SGA_USED, BYTES_OF_REDO_MINED,
to_char(STATE_CHANGED_TIME, 'mm-dd-yy hh24:mi') STATE_CHANGED_TIME
FROM V$GOLDENGATE_CAPTURE;
col capture_message_create_time for a30;
col enqueue_message_create_time for a27;
col available_message_create_time for a30;
SELECT capture_name,
to_char(capture_time, 'mm-dd-yy hh24:mi') capture_time,
capture_message_number,
to_char(capture_message_create_time ,'mm-dd-yy hh24:mi') capture_message_create_time,
to_char(enqueue_time,'mm-dd-yy hh24:mi') enqueue_time,
enqueue_message_number,
to_char(enqueue_message_create_time, 'mm-dd-yy hh24:mi') enqueue_message_create_time,
available_message_number,
to_char(available_message_create_time,'mm-dd-yy hh24:mi') available_message_create_time
FROM GV$GOLDENGATE_CAPTURE; SELECT component_name capture_name, count(*) open_transactions,
sum(cumulative_message_count) LCRs
FROM GV$GOLDENGATE_TRANSACTION
WHERE component_type='CAPTURE'
group by component_name;
LOGMINER SESSIONS/STATS
col db_name for a15;
select INST_ID, SESSION_ID,SESSION_NAME,SESSION_STATE, DB_NAME,
NUM_PROCESS,START_SCN,END_SCN,SPILL_SCN, PROCESSED_SCN, PREPARED_SCN,
READ_SCN MAX_MEMORY_SIZE,USED_MEMORY_SIZE PINNED_TXN, PINNED_COMMITTED_TXN
from GV$LOGMNR_SESSION;
SELECT SESSION_ID, NAME, VALUE
FROM V$LOGMNR_STATS;
OUTBOUND PROGRESS TABLE
SELECT inst_id, sid, serial#, spid,server_name, startup_time, state,
tztal_messages_sent, committed_data_only, last_sent_message_number,
send_time, elapsed_send_time,bytes_sent,
to_char(last_sent_message_create_time,'mm-dd-yy hh24:mi')
last_sent_message_create_time,
FROM GV$XSTREAM_OUTBOUND_SERVER;