Salesforce

UNDO of transaction writing LOB data to large database corrupts database BI file

« Go Back

Information

 
TitleUNDO of transaction writing LOB data to large database corrupts database BI file
URL Name000047594
Article Number000168272
EnvironmentProduct: OpenEdge
Version: 10.2B, 11.0x, 11.1x, 11.2x, 11.3.1, 11.3.2
OS: All supported platforms
Question/Problem Description
UNDO of transaction writing LOB data to large database corrupts database BI file
Database keeps running after client session terminates and fails transaction backout with error 815
When the database is stopped, it cannot be restarted as crash recovery fails error 815.
Crash recovery during the Logical Undo Phase triggers error "SYSTEM ERROR: Error in undo of record delete (815)"

Client BUFFER-COPYs record with BLOB to database.
Client undoes the transaction or terminates their session resulting in a transaction undo

Remote Server process servicing the client dies with error 815
 
... SRV     1: (815)   SYSTEM ERROR: Error in undo of record delete
... SRV     1: (439)   ** Save file named core for analysis by Progress Software Corporation.
... SRV     1: (2252)  Begin transaction backout.
... SRV     1: (815)   SYSTEM ERROR: Error in undo of record delete
... SRV     1: (2520)  Stopped. 

 
Steps to Reproduce
Clarifying Information
Table used to store lobs contains large amount of data (200GB +)
dbkey pointers for the Storage Area where the BLOB/CLOB resides are in the 64-bit range
Issue happens both with and without -nosavepoints on the client
The data changes triggering this issue are propagated via Replication
Roll forward fails

Stack from remote server _mprosrv on Linux reads:
dsmFatalMsgnCallBack
rmUndoLogicalDelete
rlundo
rlrej
tmrej
lkresync
dsmUserReset

Stack from Crash recovery _mprosrv on Linux reads:
dsmFatalMsgnCallBack
rmUndoLogicalDelete
rlundo
rllbk
warmstrt
rlseto
dbSetOpen
Error MessageSYSTEM ERROR: Error in undo of record delete (815)
Defect NumberPSC00293031
Enhancement Number
Cause
This defect is triggered by a very specific scenario.

The root of the issue is in the logical undo of a record delete operation involving a LOB. This can happen online (roll back a transaction -- undo, savepoint etc.) or during crash recovery (logical undo phase). ​This problem does not have to be inside the undo of a sub-transaction, but can be.

When we update a small record, there is a special case to delete the record and re-create it. The delete operation should have the same flag in the note, as the one from the create operation. This flag is used as a sanity check during undo process to make sure all the fragments of the record will be undone in the right order. If the sanity check fails, error (815):   SYSTEM ERROR: Error in undo of record delete.  As a consequence, bi Logical Undo fails.
Resolution
This transaction undo note was fixed in OpenEdge 11.3.3, 11.4 or later

This particular issue is superseded by PSC00312230, where further internal investigations in this area (wrt blobs, undo, and dancing around the 32-bit boundary) revealed that while this issue fixes a problem in one scenario additional issues existed.  PSC00312230 has fixes for the other scenarios in addition to this one. Refer to Article:
Workaround
Restore the last known un-corrupted backup and roll-forward using endtime | endtrans

Or force into the database using TRUNCATE -F. Before proceeding first understand the consequences skipping crash recovery will have until both physical integrity is restored and logical integrity is assured by application developers.
Notes
Keyword Phrase
Last Modified Date11/25/2021 2:41 PM

Powered by