This Perl script will generate protrace files for all BUSY agents of a given broker and save it in a certain directory. We find it useful, so I decided to share.
#!/usr/bin/perl # 6/21/2019 Dmitri Levin # This Perl script will collect Progress protrace files for web brokers use strict; # variables must be declared my $main_dir="Enter here a Directory to store protrace files"; if (! -d $main_dir ) { print "Directory $main_dir does not exist\n"; exit 0 } &main_proc; # Main Procedure sub main_proc { my $broker_name="your broker name"; my $broker_temp_dir="workDir for that broker"; &create_wtbman_out( $broker_name ); &check_broker( $broker_name, $broker_temp_dir ); } =item create_wtbman_out() procedure to create wtbman output =cut sub create_wtbman_out { print "process $_[0]\n"; # if directory does not exist then create it if (! -d "$main_dir/$_[0]" ) { system("mkdir $main_dir/$_[0]"); } # you need to have the correct proenv to set $DLC and $PATH here system(". proenv; wtbman -i $_[0] -q >/tmp/wtbman_$_[0]"); } =item check_broker() procedure check_broker will work with a given webspeed broker =cut sub check_broker { my $i; my $pos=0; my $PID; my $filename="/tmp/wtbman_$_[0]\n"; open(FILE,$filename) || die "Can't open $filename : $!\n"; my @lines=<FILE>; # all lines my $count=@lines; # We skip first 22 lines and loop the rest looking for BUSY for ($i=22;$i<=$count;$i++){ if ($lines[$i] =~ m/BUSY/) { $pos = index $lines[$i]," "; $PID= substr $lines[$i],0,$pos; # just to be safe I check that we do not send kill 30 to 1 or small number if ($PID > "1000" ) { print "$PID\n"; system("kill -USR1 $PID"); system("cp $_[1]/protrace.$PID $main_dir/$_[0]"); } } } close(FILE); };
As soon as we started collecting protrace statistics for BUSY agents we start getting Memory violation errors in database log.
[2019/06/26@19:30:15.000+0000] P-39846366 T-1 I WSAGE1235: (49) SYSTEM ERROR: Memory violation.
When we stopped collecting protrace, the Memory violation stopped. We collected protrace 1078 times and received 46 Memory violation errors in 2 days.
So about 1 Memory violation in 25 calls.
Apparently each kill -USR1 signal creates a small memory leak. The leak may depend on how deep is the procedure call stack reported by protrace.
Some agents reported Memory violation (49) after 3-4 kill -USR1 signals, others took longer time. So if you use kill -USR1, use it once or twice. It is a dangerous call.
Article: proGetStack / kill -SIGUSR1 causes memory leak, eventually crash in AppServer
knowledgebase.progress.com/.../000035325
Defect PSC00248179 / OE00225436
Some libraries will not be unmapped before closing the respective files.
Upgrade to OpenEdge 11.6.3, 11.7 or later where library files that do not have a symbol table are unmapped properly.
As far as I know, you're running 11.7.1. So the issue is NOT yet fixed in 11.7
By the way, enabling Statement Cache also causes Memory violation in 11.7 but only on AIX. Issue OCTA-14000. Workaround is to set a statement cache directory (using promon).
What platform and release are these crashes? Does the protrace file look like it was complete for the last SIGUSR1, or does it end abruptly?
That knowledgebase's linked defect affected Linux only, and it looks like it was fixed in 11.7.0 and 11.6.3 for most executables based on looking at the code. It wasn't fixed for _sqlsrv2, rpagent/rpserver/dsrutil, and dbtool until 11.7.5, because those executables share pieces of a different protrace generation mechanism.
>What platform and release are these crashes?
IBM AIX 7.2
Progress version 11.7.1
>Does the protrace file look like it was complete for the last SIGUSR1, or does it end abruptly?
protrace looks it is complete.
There are 46 memory violations happened. And first two protraces were always fine, it is either the 3d or the 4th or later when memory violation happens.