Salesforce

Running proutil dbname -C convchar convert UTF-8 without indexes falsely reports success

« Go Back

Information

 
TitleRunning proutil dbname -C convchar convert UTF-8 without indexes falsely reports success
URL NameP167614
Article Number000132192
EnvironmentProduct: Progress
Version: 9.x
Product: OpenEdge
Version: 10.x, 11.x
OS: All supported platforms
Other: i18N
Question/Problem Description
Running proutil dbname -C convchar convert UTF-8 without indexes falsely reports success
Database converted to UTF-8 with binary dump and load.
PROUTIL -C convchar convert UTF-8 logs a complete message 3939 and a successful message 3945

Indexes were not rebuilt during or after binary load prior to running PROUTIL convchar convert UTF-8
Indexes were rebuilt after applying word rules and loading the tran.df

Some character are missing and fields that had data that needed translation were not translated.
Steps to Reproduce
Clarifying Information
PROUTIL convchar convert UTF8 completes without error when indexes are INACTIVE

Database is encoded using iso8859-1 for character data types. (3939) 
Processing all metadata tables and fields. (3982) 
    Total of <n> fields. 
Processing all user tables and fields. (3985) 
    Total of <n> fields. 
You must have your database backed up before running convchar. (3940) 
Have you done this (y/n) ? 

Conversion beginning, please await success confirmation... (1100) 

Starting scan to convert database. (3941) 

Results of scan to convert database. (3944) 

Old Encoding:   iso8859-1 
New Encoding:   UTF-8 
Scan Mode:      CONVERT/ANALYZE 
Scan Complete:  YES 
Scan Coverage:  FULL 
Database State: CONVERTED 

The following fields contain data requiring translation: (3963) 
    Total of 0 fields. 
Database is encoded using UTF-8 for character data types. (3939) 
Conversion complete, character encoding conversion successful. (3945)
Error MessageDatabase is encoded using UTF-8 for character data types. (3939)
Conversion complete, character encoding conversion successful. (3945)
Defect NumberEnhancement PSC00225375 / OE00198157
Enhancement Number
Cause
Binary load without the build option does not enforce data integrity because it does not create keys for the records it loads, after the load user is warned with the following message:
YOU SHOULD REBUILD THE FOLLOWING INDEXES IN TABLE <tablename>.

CONVCHAR uses an index search to find which record fields need to be converted, therefore the index rebuild process prior to the conversion cannot be eliminated from the conversion procedure.

Since, indexes are required to correctly run proutil dbname -C convchar convert UTF-8, we should not be reporting the conversion process as a successful completion when there are no indexes to use. Or, we should give a message that convchar convert cannot be done with out first rebuilding the indexes.
Resolution
To ensure that proutil dbname -C convchar convert UTF-8 runs successfully after a dump and load, indexes must be rebuilt twice:
  • During or after the binary load
  • After the conversion, word-rules and df file
Outline of UTF-8 conversion after binary dumping data: 

1. Create the new database structure
$   prostrct create dbname dbname.st -blocksize 8192
$   
procopy dlc/empty8 dbname

2. Load in the database definitions (dictionary/load/df)
$   pro dbname -p _admin.p 

3. Load all the binary dump files
$   proutil dbname -C load (dumped files) -i

4. Index rebuild the database
$   proutil dbname -C idxbuild all -TM 31 -TB 32 -SG 65 -T /cidx

5. Convert the database to UTF-8
$   proutil dbname -C convchar convert UTF-8

6. Load the word rules for UTF-8 and place the resulting
proword.<n> in DLC
$   proutil dbname -C word rules 254

7. Load the prolang/utf/_tran.df file  
(dictionary/load/df)
$   pro dbname -p _admin.p 

8. Index rebuild the database 
$   proutil dbname -cpinternal UTF-8 -C idxbuild all -TM 31 -TB 32 -SG 65 -T /cidx

Workaround
Notes
Keyword Phrase
Last Modified Date12/17/2015 9:04 AM

Powered by