I18N. New Word-break Functionality For International Collations

How to use the word-break table feature for those countries that need a collation other than Basic (for example; Swedish, Norwegian, Danish)?
How to use the word-break table with collation other than Basic?

When the Progress Database has to maintain a word-index; each time a record is created/updated, the word-indexed field or array is split up into individual words and one entry is added/updated for each word in the field.  To tell which characters in the field are part of words and which are not, Progress uses word-break tables.  For collations other than Basic, a further step is involved: converting each character in the field to its sort weight.

Prior to Progress version 9 the characters in the word-indexed field were first converted into their sort weight then broken down into words. While this poses no problems for the Basic collation (no conversion is performed), for non-Basic collations the default word-break table causes the CONTAINS operator not to work as expected, therefore building a customized word-break table can be tricky.

Progress Knowledge Base Solution 18429 explains how to work around this problem for Version 8.x.  In a nutshell, word-break tables were built by referring to characters' sort weights for the target collation, rather than to characters' ASCII values.

In Progress version 9, a new type of word-break table was introduced which resolves the problem altogether.  Using this new type of word-break table allows the ability of always referring to characters' ASCII values, no matter what the target collation is.

What distinguishes a new table from an old one is the presence of a header at the beginning of the word-break table.

/*
* Sample word-break table in version 9
*/

/* This header was introduced in version 9 */
version = 9
codepage = <target code page, eg: iso8859-1>
wordrules-name = <generic name for the word-break table, eg: swedish>
type = 3

/* The word_attr field has the same format as before */
word_attr = { ..... }

/*
* End of sample word-break table
*/

Currently, "version" must be "9" and "type" must be "3".  The "codepage" field must match the code page in the target database.

Except for this header, new word-break tables must be compiled and used just like the old ones.

NOTES:
- To migrate from an old work-break table to a new one, word indexes must be rebuilt once the new word-break table is set up.
- The new word-break table does work in conjunction with the -ttwrdrul startup parameter (Word-break rule for temp-tables).

References to Other Documentation:

Progress Article(s):
Solution 18429, "Creating a word-break File for International Collation"
Solution 20441, "-ttwrdrul and Applying Word Rules to Temp Tables"

OpenEdge Development, Internationalizing Applications, Character Processing Table Formats, "Work-break table"