Upgrade to OpenEdge 11.6.2 Service Pack, 11.7.0 or later where a new option is provided in PKIUTIL to generate either a pkcs5 or pks8 encrypted key:
- -pkcs5 (force pivate key to be pkcs5) [This is default in 11.6.2]
- -pkcs8 (force private key to be pkcs8)
For example:
A pkcs5 format encrypted key is the default for 11.6.2:
$ pkiutil -keysize 2048 -newreq testalias
Or by explicitly setting
-pkcs5 as a parameter
$ pkiutil -keysize 2048 -pkcs5 -newreq testalias
Should a pkcs 8 encrypted key be needed, then use -pkcs8:
$ pkiutil -keysize 2048 -pkcs8 -newreq testalias
On OpenEdge 11.6 when using a pkcs8 private key format:
- An OpenEdge Database using a pkcs8 certificate will start without error
- An AppServer using a pkcs8 certificate will fail, which is expected in OpenEdge 11.6.x:
SSL Server initialization error com.rsa.ssl.SSLException: java.lang.NullPointerException
at com.rsa.asn1.AlgorithmID.a(Unknown Source)
This is because the Broker (java.exe) uses BSAFE RSA SSLJ lib for underlying SSL, which has problem with pkcs8.
In OE 11.7.0 RSA SSLJ has been upgraded to JSSE, which will resolve the problem