Salesforce

Application Performance impact of large LOB creation with COPY-LOB

« Go Back

Information

 
TitleApplication Performance impact of large LOB creation with COPY-LOB
URL NameApplication-Performance-impact-large-LOB-creation-with-COPY-LOB
Article Number000178536
EnvironmentProduct: OpenEdge
Version: 10.x, 11.x, 12.x
OS: All supported platforms
Other: COPY-LOB
Question/Problem Description
Application Performance impact of large LOB creation with COPY-LOB

All Agents are in BUSY state while LOB is being created.
Until the LOB is created: COPY-LOB FROM <file> TO <table.lob-field>.
  • Clients connected to the AppServer hang
  • All clients running transactions on the Database hang
Steps to Reproduce
Clarifying Information
LOBs being created from MB sized files.

The database is checkpointing at under 10 second intervals: PROMON > R&D > 3 > Checkpoints 

Creation of BLOBS is faster than CLOBS
Performance is slightly worse in OpenEdge 11.5 and later
Performance is slightly worse when the BIW is running
Performance is slightly worse with -LRUSKIPS
Performance is worse by at least a third when After-imaging (AI) is enabled.
Performance is vastly improved running with no integrity (-i)

Copying the LOB to a file takes seconds by comparison and does not hang clients.
COPY-LOB FROM <table.lob-field> TO <file> 
Error Message
Defect Number
Enhancement Number
Cause
Creating a LOB affects other clients performance:

The reason that transaction processing against the database slows down, is because the LOB create is I/O bound. 

When clients are connected through a tiered architecture, the agent / session can only handle one lob insert at a time which blocks other requests is coming from the throttle of the AppServer managing connections.

In addition, LOBS are more likely to pollute the MRU/LRU chain which affects other user's performance in having to re-read their blocks into bufers.

Creating a LOB uses the record subsystem:

A LOB is saved in (almost) the same format as a regular record. It's not block based. The LOB data are cut (by the AVM runtime) into fragments that are stored separately in the same manner as normal data records are, 
where each slice is stored as a record fragment. When the LOB value is larger than the database blocksize, 
it will be fragmented across multiple blocks which will all need to be read to retrieve the data with a LONGCHAR or MEMPTR.

The amount of work the record manager is performing is exactly the same operation by operation to load the lob (with and without integrity). The reduction in performance is due to writing to the bi subsystem, raising the high-water mark and extending storage areas and record consolidation.

Creating a LOB is slower than Dumping a LOB:
  • COPY-LOB to a file, dumps contents of a LOB to a file as a read operation and shared latched when assembled for reading. There are no bi notes on a read operation.
  • COPY-LOB from a file, inserts the contents as a record write operation and as such bi notes are needed.  
Creating a CLOB is slower than a BLOB:

A CLOB takes more work to retrieve than a simple VARCHAR or LONGCHAR value due to code page conversion. 
Equally sized CLOBs are slower than BLOBs, as:
  • CLOBs are character based instead of bytes based;
  • Characters can be in different encodings.
  • CLOBS need to support more complicated operations like copy, truncate, and overlay (akin to a text editor that can support multiple languages, ccToUTF8)
The contention is not on TXE latching:
  • TXE acquire/release are only in the rm layer for create and update as there are many record segments being created to populate the lob record.
  • TXE acquire/release latches are held for the duration of the segmented lob creation and then released. They are not held until transaction commit
Creating BI notes needed for the LOB create with integrity (no -i) :

Running reliable means completely logging all activity that goes into creating a lob object. A significant number of notes need to be written for:
  • RM CHAIN manipulation 
  • The creation/update of record fragments (how lob segments are stored), 
  • Increasing the High Water Mark. 
  • All Physical and Physiological notes are logged. 
  • All purely Logical notes are written. 
With a BIW running performance is slightly worse because the BIW is competing with the user for the same latch.

Creating BI notes needed for the LOB create with no integrity (-i) :

There is a huge difference between note activity when LOBS are created with no-integrity (specifying -i) as there is a much lower degree of recovery:
  • Only physiological notes are logged, which are significantly fewer. 
  • Most purely logical notes are not written (savepoints and logical record operation markers). 
Creating a LOB in 11.5 or later

The bi notes pre and post 11.5 are the same, there were minor code changes to support partitionId in OpenEdge 11.5 and 11.6 which make creating notes more expensive.
 
OpenEdge 11.7.6, 12.1, bi cluster optimization was reworked to improve adding new clusters to the bi chain, which can improve performance at checkpoint time: Performance improvement when new biclusters are added  
Resolution
A. Review the Area structure of LOB objects
  1. Ensure LOB objects are in their own Type II LOB Area in order to negate contention on the Chains with other objects. 
  2. Consider using a unique Type II Area for each group of Lobs (ie, if Table A has 3 lobs, these 3 lobs go to their own Type II Area). A storage area for each LOB is preferable particularly when these are not 'small lobs' ( much larger than the database blocksize) and there are many of them (10,000's +). When the application is heavily LOB based a separate "LOB database" may be best advised instead and/or separate Storage Areas for each Tables LOB objects.
  3. Set the size of the Area recbits to avoid record fragmentation which is frequent with LOBS because the database manager tries to consolidate them into max record size (32 KB) chunks. A cluster size of 512 or 64 (Fixed extents) and recbits between 1 and 4
  4. Avoid extending the files in the LOB Area by ensuring there are sufficient large fixed length extents
  5. Increase the TossLimit on the LOB area to at least half the database blocksize to evict these off the RM chain
Example:
$ proutil <dbname> -C setAreaTossLimit "LargeLOBTableA" 2048
d "LargeLOBTableA":100,2;512 . f 2040000
d "LargeLOBTableA":100,2;512 . f 2040000
d "LargeLOBTableA":100,2;512 .
$ proutil <dbname> -C setlobtosslimit tableB.FLOB1 512
$ proutil <dbname> -C setlobtosslimit tableB.FLOB2 2048
d "SmallLOBTableB":110,4;64 . f 2040000
d "SmallLOBTableA":110,4;64 . f 2040000
d "SmallLOBTableB":110,4;64 . v 2040000

Refer to Articles
B. Tune the BI subsystem 
  1. Use a larger bi cluster and biblock sizes. This will also reduce the contention between the BIW and the user for the same latches. 
    If AI is enabled, the aiblocksize needs to be set to the same size as the biblocksize.
    $ proutil dbname -C truncate bi -bi 65536 -biblocksize 16 
  2. Before starting the database, pre-grow the bi file chains
    $ proutil dbname -C bigrow 12 -r 
  3. Ensure sufficient bi buffers are used, if "BI Buf Waits" are consistent, increase the number of bibufs online
    $ proutil dbname -C increaseto -bibufs 150
C.  LOB activity can be viewed in PROMON or with VST's

PROMON > R&D, > Other Displays > I/O Operations by User by Lob
_LobStat and _UserLobStat

For example: <lob> Buffer Hit %  =  _LobStat._lobstat-osread * 100 / _LobStat._lobstat-read
Workaround
Notes
Keyword Phrase
Last Modified Date6/23/2021 12:34 PM

Powered by