ORA-00600 [kghstack_alloc], [krsr_check_itgap] on Downstream Mining Database hosted on Exadata Machine
While working on an issue on Oracle Goldengate downstream database, configured on a RAC to RAC Dataguard environment on Exadata Hardware, I’ve noticed ORA-00600 error and it appeared that this error code is not documented yet. This info was confirmed also by Oracle Support.
The issue started with Oracle Goldengate EXTRACT parameter. After a weekend we noticed that the Extract process is lagging by 48 hours, even though operating in RUNNING status. Various scenarios and solutions were implemented with no luck. While checking the archive destination, downstream destination was showing and error and getting invalid:
DEST_ID DEST_NAME STATUS TYPE ERROR
-------- -------------------- ------- ---------- -----------------------------------------------------------------
1 LOG_ARCHIVE_DEST_1 VALID LOCAL
2 LOG_ARCHIVE_DEST_2 VALID PHYSICAL
5 LOG_ARCHIVE_DEST_5 ERROR DOWNSTREAM ORA-16099: internal error ORA-00600 occurred at standby databaseThe LOG_ARCHIVE_DEST_2 is VALID and log shipment to Physical Standby Database operates with no issues. But LOG_ARCHIVE_DEST_5’s status get ERROR due to ORA-00600 on Downstream Mining Database.
Related Alert Log File Content:
--------------------------------------------------------------------
RFS LogMiner Error Report
--------------------------------------------------------------------
Component: RFS LogMiner
RFS ID: 146278
Thread: 7
Incident ID: 80181
Trace File: /u01/app/oracle/diag/rdbms/*******/*******/trace/******_rfs_136378.trc
Incident File: /u01/app/oracle/diag/rdbms/*******/*******/incident/******_rfs_136378.trc_i80181.trc
Error Message:
ORA-00600: internal error code, arguments: [kghstack_alloc], [krsr_check_itgap],
[], [], [], [], [], [], [], [], [], []
Action Required:
Use ADRCI or Support Workbench to package the incident.
--------------------------------------------------------------------Version Details:
================================================================================
SYSTEM CONFIGURATION REPORT
================================================================================
DATABASE INFORMATION
--------------------------------------------------------------------------------
Component | Version/Details
------------------ | -----------------------------------------------------------
DB Version | 12.1.0.2.0
DB Architecture | Oracle Goldengate Downstream Configuration
Extract Type | Integrated
OPERATING SYSTEM
--------------------------------------------------------------------------------
OS Distribution | Linux
Kernel Version | 4.1.12-94.8.4.el6uek.x86_64
OS Platform | Oracle Linux with Unbreakable Enterprise Kernel (UEK)
HARDWARE PLATFORM
--------------------------------------------------------------------------------
Hardware | Exadata X6-2 Elastic Rack HC 8TB
Machine Model | Exadata Database Machine
Rack Type | High Capacity (HC) 8TB storage
GOLDENGATE CONFIGURATION
--------------------------------------------------------------------------------
Component | Version
------------------ | -----------------------------------------------------------
Oracle Goldengate | 12.2.0.1.160419
Extract Mode | Integrated Extract (Downstream)
-------------------------------------------------------------------------------Workaround: Convert integrated extract to Classic extract and start the extract with a point-in-time SCN where you lost the synchronization.
To keep the replication running while investigating the root cause, we decided working with Classic Extract which does not support downstream database, as we have error on downstream database. Temporarily we stopped Integrated Extract and running with Classic Extract.
Actuall root cause wasn’t detected until this moment, however we noticed that the extract process was registered on the source DB instead of the downstream database. As soon as we corrected it, the replication resumed and ORA-00600 error disappeared. We can hardly call it a root cause, but at least this approach resolved the issue.
Get Omer Narman’s stories in your inbox
Join Medium for free to get updates from this writer.
Source Database:
SQL> SELECT capture_name FROM dba_capture;
CAPTURE_NAME
------------------------------
OGG$CAP_E_Test1
1 row selected.And two old capture processes on downstream database:
SQL> SELECT capture_name FROM dba_capture;
CAPTURE_NAME
------------------------------
OGG$CAP_E_TESTA
OGG$CAP_E_TESTB
2 rows selected.Solution: Unregister E_TEST1 extract from source database, Unregister E_TESTA and E_TESTB from downstream database.
For Source Database:
ggsci> dblogin userid ogguser@testdb, password ********
Successfully logged into database testdb.
ggsci> unregister extract E_TEST1 database
Extract E_TEST1 successfully unregistered from database.For Downstream Database: For downstream capture processes, make sure that you are connected to both source database and downstream database.
ggsci> dblogin userid ogguser@testdb, password ********
Successfully logged into database testdb.
ggsci> MININGDBLOGIN userid ogguser@downstream1, password ***********
Successfully logged into mining database downstream1.
ggsci> unregister extract E_TESTA database
Extract E_TESTA successfully unregistered from database.
ggsci> unregister extract E_TESTB database
Extract E_TESTB successfully unregistered from database.Make sure that the capture processes are not running on source and downstream database:
Source
SQL> SELECT capture_name FROM dba_capture;
no rows selected
SQL>Downstream
SQL> SELECT capture_name FROM dba_capture;
no rows selected
SQL>If they retrieve any rows, you can drop them with DBMS_CAPTURE_ADM package:
SQL> exec DBMS_CAPTURE_ADM.DROP_CAPTURE('OGG$_E_TESTB');
PL/SQL procedure successfully completed.
SQL>Stop the application, create new integrated extract and new replicat processes, and start the replication up.
