OpenEdge 11.7, 32-bit
We are running a process through Jenkins, which will periodically restore a database from backup. However, the 'prorest db-name db-name.bak' command is intermittently failing with the:error message:
OpenEdge Release 11.7.3 as of Fri Apr 27 17:02:34 EDT 2018
** stget: out of storage. (6498)
When the restore fails, the database log file contains the following:
[2019/01/07@09:56:39.910+1000] P-8388 T-3168 I DBUTIL : (451) prostrct create session begin for svc-jenkins on batch. [2019/01/07@09:56:40.240+1000] P-8388 T-3168 I DBUTIL : (12862) Structure file db-name.st used to create database with blocksize 4096. [2019/01/07@09:56:40.250+1000] P-8388 T-3168 I DBUTIL : (334) prostrct create session end.
The database is being restored on a Windows Server 2016 machine, using the default block size (4K). There is 16G of RAM on the machine (5G in use before restoring the database), and there is sufficient disk space on all drives. The database backup file is 1.5G, with Type II storage areas.
Has anyone seen a prorest command raise this error before? I've only ever seen this message from 4GL code.
stget: out of storage errors like to occur when you either run out of memory, or run over your address space.
Since you say you're using 32-bit OpenEdge, hitting the address space limits is the more likely scenario because you'll be bound by the classic 2GB limit.
The stget errors are not limited to ABL client processes either; a "proutil sports2000 -C idxbuild -B 123456789" or indeed "prorest sports2000 sports2000.bk -B 123456789" will also trigger one a 32-bit OE. (Because a 100GB+ buffer pool isn't going to fit the address space.)
Check the exact startup parameters used to launch the prorest sub-process for starters. (Including what's in your %DLC%\startup.pf).
I'm also not sure if WIndows imposes additional limits when one process is spawned as a child of another, if it does you'll need to take those into account as well.
Is all the disk attached physically, or is it network attached at all? Is Jenkins running with Administrator privileges? What else is happening at the time of the errors?
Thanks frank.meulblok - the tip about the startup.pf file was the solution. There was a -B 100000 and a -L 40000 sitting there. The process itself (when it worked) was only hitting 1.2G of RAM, so it's still strange the error was intermittent. I've reduced those figures and the process runs with 600MB allocated now. I'll monitor it and see if it will stay stable now.
Memory allocations can fail before all of memory is exhausted because over time memory space of a process becomes fragmented as memory is allocated and freed. Eventually you can end up in a situation where there may be, say, a total of 500MB of free memory but the largest contiguous block of free memory is only 20MB or less. At that point an allocation of 21MB will fail even though there is 500MB free.