Salesforce

How to scope and define a multi-volume srt file for idxbuild

Information

 
TitleHow to scope and define a multi-volume srt file for idxbuild
URL NameP55740
Article Number000169310
EnvironmentProduct: Progress
Version: 8.x, 9.x
Product: OpenEdge
Version: 10.x, 11.x
OS: All Supported platforms
Other: Database
Question/Problem Description
How to create multi-volume sort file?
How to create a multi-volume srt file?
Overcoming the SRT Files Size limitation
How to define a multi-volume sort file for "idxbuild"
How to size and define a multi-volume srt file for idxbuild
How to create a multi-volume srt file for idxbuild
How to calculate the size of a multi-volume srt file?
What would a multi volume .srt file look like?
When do I need to define a multi volume sort file ?
Steps to Reproduce
Clarifying Information
Error Message
Defect Number
Enhancement Number
Cause
Resolution
When an Index Rebuild is run with sorting, space limitations during the sort phase will cause the utility to terminate. The temporary sort file allocated to do the sort could reach disk space, operating system limit for file size, user limits or the Progress 2GB limit on sort files.

The disk space needed for sorting can be larger than the database size in cases where for example, many indexes on char field(s) are being rebuilt because strings are not compacted in Phase 1 of IDXBUILD.  Even when disk space itself is not the problem, the .srt files themselves may not exceed 2GB, hence the need for defining a multi-volume srt file for the operation. 

Using a text file to define "multi sort file locations" was first introduced in Progress 8.0B to overcome both the 2 GB limit of the sort file itself and space limits within directories. 
Since Progress 9.1D08, OpenEdge 10.0A, IDXBUILD can create sort files larger than 2 GB:
  • irrespective of the database being enabled for large files,
  • if the filesystem supports large files and
  • if the user running the idxbuild operation can create the filesizes in the directories the sort files will be created in.
  • if there are sufficient disk space at the time of the sort and merge operations
Since the limit on srt file sizes was lifted, there is no longer an implicit reason for having to define a multi volume sort file.  It is never-the-less useful when the disk space needed by index rebuild needs to be spread across several file systems or more importantly, to minimize disk contention during the sort and merge phases of idxbuild by spreading out the I/O across filesystems or at least to filesystems distinct from the database files.

To define a Multi-Volume Sort file for IDXBUILD

Create a text file that specifies the directories and the amount of space per directory the Sort and Merge Phases of IDXBUILD will have access to during the Index Rebuild. 

The contents of the file must follow these conventions: 
  1. The naming format is [databasename].srt, which is created in the same directory that the database control area (.db), unless the -SS parameter is used. Refer to Article -SS parameter for IDXBUILD explained   
  2. List the directory and the amount of space to allocate for the index rebuild sort on separate lines. Separate the directories from the size by at least one blank space. Do not use TABs or indents.
  3. The size specified in the directory specification is the maximum (in 1 KB units) that the file can grow. Specifying 0 for any directory, indicates unlimited growth. This space must available on disk for the duration of the index build and the directories must exist before IDXBUILD is run.
  4. Prior to Progress 9.1D08, OpenEdge 10.0A, each directory defining the sort space that can be allocated is less than 2GB in each instance, otherwise optionally used to restrict directory space.
  5. Directory names for sorting are listed, not file names. Ensure the trailing slash "/" for UNIX, or backslash "\" for Windows is in place, followed by end of line character.
  6. Ensure there are no empty lines, not even at the end of the dbname.srt file.
  7. To estimate the sort space needed refer to Article  How to estimate disk space needed for sorting by IDXBUILD   
Example of a Multi Volume Sort File 

There are 3 GB space in /usr1 and then enough space remaining in /usr2,
a 5 GB sort file definition which uses 1 GB delimiters for each .srt file leaving the last one as variable would look as follows where the directories listed already exist:

Prior to Progress 9.1D08, OpenEdge 10.0A
1000000 /usr1/sort1/
1000000 /usr1/sort2/
1000000 /usr1/sort3/
0 /usr2/sorta/


Since Progress 9.1D08, OpenEdge 10.0A
3000000 /usr1/sort1/
0 /usr2/sorta/


On Windows, the only difference is the directory reference:
3000000 C:\tmps\sort1\
0 F:\tmps\


The .srt files:  

Once the multi-volume sort file has been configured, a complete index build can be run offline with the proutil command.
$ proutil dbname -C idxbuild ALL -TB 24 -TM 32 -B 200

Please refer to the documentation for additional IDXBUILD parameters specific to the OpenEdge version in use. For example -SS explicitly references the location of the multi-volume srt file since OpenEdge 10.0A.

At the prompt: Do you have enough disk space for index sorting? (y/n)
You must answer "y" for the multi-volume sort file to then be used by idxbuild.
Answering "n" would make idxbuild use an entirely different algorithm, which does not rely on sort files at all.

When Index Rebuild does not find a multi-volume sort file, it uses the directory supplied by the -T parameter otherwise the current working directory to create .srt files.

The Index Rebuild utility opens the files for each of the directories before it starts the sort process. As a result, one of the following messages is displayed for each file:

(11447) Temporary sort file at: <directory path> will use up to  <n> K of disk space.
(11443) Temporary sort file at: <directory path> will use the available disk space 
The 11443 message is displayed when a zero (0) file size is specified or when a multi-volume srt file is not found.

When the sort completes, the following message is displayed for each file. This information is useful to reference for future IDXBUILD operations:
    
(11480) Temporary sort file at <directory path> used up <n>K of disk space
(11483) A total of <n>K of temporary sort disk space was used for area <area number>.
When these messages display 0 K, this simply means that the sort took place completely in memory.
 
Should the IDXBUILD abort unexpectedly or intentionally.
After addressing the reason for failure, the multi-volume sort file can be redefined if needed before re-running the full idxbuild. In this case, you see the following:

Previous index rebuild was interrupted. This is a new try. (2748)
Can't find selected index list file .xb; assuming ALL. (2786)
Is this correct? (y/n)

 
Choosing "y" will start the Index Rebuild, with the current multi-volume srt file definitions.
For re-running selective IDXBUILD operations, refer to Article  How to perform an idxbuild on original tables / indexes when an idxbuild operation fails.  
Workaround
Notes
Keyword Phrase
Last Modified Date11/20/2020 7:09 AM

Powered by