Why does the Progress database allow invalid UTF-8 characters to be inserted? 

Why is there no mechanism to prevent invalid UTF-8 characters being inserted into the database by validating input?

Encountering a problem similar to the one described in Article  "Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data". 

In that case, the JDBC driver reports the error "Transliteration failed, reason: invalid UTF8 data" if it encounters a character already in the UTF-8 database that is invalid in the UTF-8 code page.
If the JDBC driver can detect the problem then why can the database not detect this when the data is being inserted / updated, preventing the problem?

The primary reason invalid database codepage characters can be inserted is performance.

With a single-byte code page, every byte represents a valid character. When -cpinternal = database code page = -cpstream, there is no conversion applied when taking data from disk through the AVM into the database. 

The AVM does the same for UTF-8 but, in this case, not every byte sequence is a valid byte sequence. There is some trust on the part of the user that they have correctly configured their code page settings and have valid UTF-8 data. Furthermore if the source data is not valid UTF-8 and the ABL program does not check/prevent it, then the invalid data will get into the database. 

If the AVM had to validate every character in a UTF-8 string for every operation, even though the AVM has been told that everything is running in UTF-8, then this will add a significant performance hit.

The AVM only performs validation of characters at certain times, codepage conversions are one. 

• When -cpinternal == -cpstream, then no conversion will take place: you have told the AVM that the data coming from a stream is in the -cpstream codepage. If this is not the case, it is a site configuration issue.
• Data will usually go through some codepage conversion to -cpinternal on its way into the AVM, which ensures the data in CHAR variables is in -cpinternal. Since all chars are in –cpinternal, there is typically no other validation done to ensure the characters in the CHAR are correctly formed. To determine why the codepage conversion is not catching the problem in the data when it enters the AVM, investigate where the source data are originating ( DLL call, a socket read operation, IMPORT etc.)
• When a malformed UTF-8 character is created in a MEMPTR. This avoids any validation during codepage conversion when the data enters the AVM. The GET-STRING function does not perform any validation of the data, it is assumed to be -cpinternal. Reading directly from raw memory, there is no guarantee how the data is encoded. It could have come from anywhere: a DLL call, a socket read operation, IMPORT… It is the responsibility of the application code design to check/prevent it.

Once the malformed data that does not conform to the code page of the database are in the database, this results in corruption. Corrupt data can be detected using a LONGCHAR variable, LONGCHAR processing performs validation to ensure characters are encoded correctly for the LONGCHAR’s codepage. Refer to the code example provided in Article: 

•  i18N: ABL code to find field data that does not match the database codepage  

It was determined that the Progress Product is functioning as designed. An enhancement to the product can be requested through the Progress Community via an Ideas submission.  Customer feedback is valuable and Idea submissions are monitored by our Product Management team. Enhancement requests are reviewed during the planning phase of each new product release and a list of the enhancements chosen for implementation can be found in the Release Notes documents that accompany each release.  Once an Idea is submitted the Progress Software Community will have the opportunity to comment on and vote for the Idea.
 
For detailed information on how to submit an Idea, please refer to Article "How to submit an idea for a Progress product enhancement (embedded video)".

Progress Articles:

"Attempt to retrieve data via JDBC fails with error Transliteration failed, reason: invalid UTF8 data"
 Why is there a difference of behavior in handling a foreign language character using SQL client versus using ABL client?