Admin server on Repl Target crashed earlier for a client. 10.2B08. Windows. We advised a restart of the server.
Restart completed at 12:36, and Replication started up ok.
[2017/07/17@12:37:53.346+0100] P-3764 T-1536 I RPLA 81: (10670) The Source and Target databases are synchronized. Normal processing is resuming.
It all ran ok until
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (1028) SYSTEM ERROR: Rollforward Apply AI note, BKUPDCTR=3411, note updctr=3410.
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (8999) area 6, dbkey 1101944192, record type 33
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (10428) Function dsaAIApplyNote failed in rpNOT_ProcessNoteBlock with error -30000.
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) Diagnostic Dump of RLAIRollFState_t - After Apply Failure
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0000: 0000 0000 0000 0000 4451 0000 0000 0000 f92a 0000 0100 0100 0000 0000 0200 0100
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0020: 6531 cc75 0100 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0040: 0000 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) Diagnostic Dump of RPAIBlock_t - After Apply Failure
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0000: 0040 0000 0000 0000 0900 0000 2f38 0000 0080 0703 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0020: 1e0c 0000 0000 0000 0200 0000 dab7 1800 0040 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) 0040: 0000 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.961+0100] P-3764 T-1536 I RPLA 81: (-----) Diagnostic Dump of RPNote_t - After Apply Failure
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0000: 2817 0000 6d00 0000 0200 0000 6d00 0000 0000 0000 0000 0000 0900 0000 2f38 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0020: 0080 0703 0000 0000 0000 0000 0000 0000 1e0c 0000 0000 0000 0000 0000 007d 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0040: 4000 0000 0200 006f 2100 0000 0000 0000 e81d ad01 0000 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (10495) Block processing ended in error at block 3102 in area 9. Fathom Replication cannot continue.
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) Diagnostic Dump of RPAIBlock_t - Block processing error
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0000: 0040 0000 0000 0000 0900 0000 2f38 0000 0080 0703 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0020: 1e0c 0000 0000 0000 0200 0000 dab7 1800 0040 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (-----) 0040: 0000 0000 0000 0000 0000 0000 0000 0000
[2017/07/17@13:40:53.962+0100] P-3764 T-1536 I RPLA 81: (10504) Unexpected error -30000 returned to function rpAGT_AgentLoop.
[2017/07/17@13:40:57.972+0100] P-3764 T-1536 I RPLA 81: (10506) The Fathom Replication Agent l_idx_pws is ending.
On source it's quite benign:
[2017/07/17@13:40:54.079+0100] P-4412 T-4408 I RPLS 81: (10699) The Fathom Replication Agent l_idx_pws is requesting to be terminated.
[2017/07/17@13:40:54.079+0100] P-4412 T-4408 I RPLS 81: (10700) The Fathom Replication Agent l_idx_pws is being terminated.
[2017/07/17@13:40:54.081+0100] P-4412 T-4408 I RPLS 81: (10700) The Fathom Replication Agent l_idx_pws is being terminated.
[2017/07/17@13:40:54.081+0100] P-4412 T-4408 I RPLS 81: (10702) All Fathom Replication Agents have been terminated or have ended.
[2017/07/17@13:40:54.081+0100] P-4412 T-4408 I RPLS 81: (10698) The Fathom Replication Server will shutdown but the source database will remain active.
[2017/07/17@13:40:56.082+0100] P-4412 T-4408 I RPLS 81: (10505) The Fathom Replication Server is ending.
Restarting Replication has the same issue. It's an issue with 2 out of 10 databases.
Any ideas how to get this running again?
I've ended up reseeding Replication and we're back up and running. No idea what caused the issue though. Strange it was just 2 DBs out of 10.
I've ended up reseeding Replication and we're back up and running. No idea what caused the issue though. Strange it was just 2 DBs out of 10.