Salesforce

AdminServer slow to start and configure on Linux

« Go Back

Information

 
TitleAdminServer slow to start and configure on Linux
URL NameAdminServer-slow-to-start-on-Linux
Article Number000168906
EnvironmentProduct: OpenEdge
Version: 11.5,x, 11.6.x, 11.7.x, 12.x
OS: Linux
Other: AdminServer, Progress AppServer for OpenEdge, OEM/OEE
Question/Problem Description
AdminServer takes a long time to start on Linux
proadsv -q shows "starting" but looks like it does not start
AdminServer takes at least 30 minutes to start on Ubuntu.
On Red Hat Linux the AdminServer can take 4 mins or longer to start.

PASOE commands take long to respond, seems to hang but eventually respond

When accessing OEM/OEE Browser UI for the first time post-install and saving the new password the browser loops for at least 30 minutes before the password is changed and standard settings are set: http://localhost:9090/management/firstuse.jsp  :
o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadDetected 
    - Thread [catalina-exec-3] (id=[125]) has been active for [608.792] milliseconds (since [25.11.20, 16:43]) to serve the same request for [http://localhost:9090/management/firstusevalidation.jsp] 
StuckThreadDetectionValve.notifyStuckThreadDetected - Thread [catalina-exec-3]
at java.base@11.0.9/sun.security.provider.NativePRNG$Blocking.engineNextBytes(NativePRNG.java:268)
at java.base@11.0.9/java.security.SecureRandom.nextBytes(SecureRandom.java:751)
at java.base@11.0.9/java.security.SecureRandom.next(SecureRandom.java:808)
at java.base@11.0.9/java.util.Random.nextLong(Random.java:424)
    at com.progress.isq.ipqos.resources.security.PasswordUtil.generateSalt(PasswordUtil.java:163)
    at com.progress.isq.ipqos.plugin.FathomConfiguration.storeFathomPassword(FathomConfiguration.java:1759)
    at com.progress.isq.ipqos.plugin.FathomConfiguration.save(FathomConfiguration.java:207)
[Catalina-utility-7] o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadCompleted 
- Thread [catalina-exec-3] (id=[125]) was previously reported to be stuck but has completed. It was active for approximately [1.218.462] milliseconds.

On the AdminServer side the delay seems to occur after the ActiveMQ broker configuration and before the log4j warnings:
[ActiveMQS] The optional ActiveMQ broker configuration file /dlc/properties/activemq.xml does not exist.  Default configuration will be used. (17212)
[STDERR]    log4j:WARN No appenders could be found for logger (org.apache.shiro.io.ResourceUtils)
 
Steps to Reproduce
Clarifying Information
Does not happen in OE 10.2x release.
Can happen on a physical server, but also in a Virtual Machine.
Antivirus software is not checking OpenEdge Explorer's files during startup, which is known to corrupt the %OEM%\work\cachedb\*.* and %OEM%\work\configdb\*.* databases
Error Message
Defect Number
Enhancement Number
Cause
Since OpenEdge shipped with Java8 and requires JDK11 since 12.2,  this issue affects java processes that require using encryption seed data for "NativePRNG", "SHA1PRNG" and "DRBG" crypto mechanisms from file:/dev/random.  

Linux needs entropy when working with cryptography. Entropy is randomness collected from the pseudo-device /dev/random from data generated by OS device drivers and services. The data sent to /dev/random is known as the entropy pool. Crypography uses a blocking function that waits until entropy reaches a certain level before providing its result.  From a practical standpoint, this means that programs using /dev/random can take longer to complete and this can additionally affect timeouts or stop the process from working if the pool is empty.

This is not specific to OpenEdge, for additional vendor specific information regarding random being slow, is provided in the Notes section below.
Resolution

To determine if the Operating System exhibits this behaviour at the time:

Try to display a portion of these files from a shell prompt and note the time difference:

$   head -n 10 /dev/random
$   head -c 10 file:/dev/urandom

or, due to a defect in certain distributions:
$   head -c 10 ///dev/urandom [ or ] head -c 10 /dev/./urandom 

An OpenEdge 12.2 catalina.log demonstrates the delay in the: o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadDetected & eventually o.a.c.v.StuckThreadDetectionValve.notifyStuckThreadCompleted messages.

To use /dev/urandom generator instead

This does not make cryptography less secure. The following Blog summarises why:
Myths about /dev/urandom
https://www.2uo.de/myths-about-urandom/   

Configure the JVM to use /dev/urandom

The java.security file, used for security management is found in the following locations:

$   sudo find / -iname java.security

  • Java 8, and earlier versions:  $JAVA_HOME/jre/lib/security/java.security. For OpenEdge 11 which ships with java, this would be: $DLC/jre/lib/security/java.security
  • Java 11: $JAVA_HOME/conf/security/java.security
Open the java.security file in a text editor, as root user:

Change the following line:
securerandom.source=file:/dev/random

to read:

securerandom.source=file:/dev/urandom
After Saving the update, restart the AdminServer.

Another method is to use the RNGD Service

The rngd service is part of the rng-tools package on both Debian and Red Hat / CentOS based systems.

As a quick verification, run the following command as root, then start the AdminServer which will no longer experience the delay on startup:
This command can be added to /etc/rc.local or crontab to be run at system startup.

$   rngd -r /dev/urandom 
$   proadsv -start

The following information is provided as a guideline, refer to OS documentation for details specific to your Linux distribution.

Install “rng-tools” if not exists. Login with root privileges and run:

UBUNTU:  apt-get install rng-tools
Update "/etc/default/rng-tools" file to add “HRNGDEVICE=/dev/urandom”
Start rng-tools:
 /etc/init.d/rng-tools start

CentOS: yum install rng-tools

Edit the service unit file: /usr/lib/systemd/system/rngd.service
[Service]
ExecStart=/sbin/rngd -f -r /dev/urandom

Reload the unit file, start the service and check it's status:
$   systemctl enable rngd
$   systemctl daemon-reload
$   systemctl start rngd
$   systemctl status rngd


.
Workaround
Notes
References to Other Documentation:

The Right Way to Use SecureRandom
https://tersesystems.com/blog/2015/12/17/the-right-way-to-use-securerandom/   

java random number generator drains /dev/random on Linux
https://bugs.openjdk.java.net/browse/JDK-8143166   

Avoiding JVM Delays Caused by Random Number Generation
https://docs.oracle.com/cd/E13209_01/wlcp/wlss30/configwlss/jvmrand.html   

How to deal with a slow SecureRandom generator?
https://stackoverflow.com/questions/137212/how-to-deal-with-a-slow-securerandom-generator   

rngd(8) - Linux man page
https://linux.die.net/man/8/rngd   

/dev/random Extremely Slow?
http://stackoverflow.com/questions/4819359/dev-random-extremely-slow.

 
Keyword Phrase
Last Modified Date8/22/2022 7:38 PM

Powered by