In some cases there may be a need to know what Shared Memory Segments are attached to a Progress OpenEdge Database.
Use-Cases:
- After a database crash where the shared memory was not released and the database fails to start with 'shared memory already in use' errors.
- When a database is shutdown and can not be restarted due to error: Shared memory in use by another process (1260)
- When a database cannot be restarted after an application hang needs to be resolved by shutting the database down to release orphaned or dead-locked connections
- When an undetermined shared-memory issue prevents the database from being shutdown
How to determine which shared memory segments are being used by OpenEdge databases running on this machine:
On UNIX platforms, PROUTIL with the dbipcs option can be used.
- Notice there is no dbname specified
- When not logged in as the root user, SUDO must be used
$ proutil -C dbipcs
Sample DBIPCS output: In the following example, the production 'sports' database owns 3 shmsegs {6,7,8}
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
0 - - - (not PROGRESS)
1 - - - (not PROGRESS)
6 10213 0 Yes /db/prod/sports.db
7 10213 1 - /db/prod/sports.db
8 10213 2 - /db/prod/sports.db
10 10213 0 No /db/test/sports.db
13 64010213 - - (unsupported shared memory version)
Where:
- ID - Indicates the shared-memory ID.
- ShMemVer - Specifies the OpenEdge shared memory version.
- Seg# - Indicates the shared-memory segment number. One database can own more than one shared-memory segment.
- Database - Represents full path name to the database.
This column will also represent:
- Non-OpenEdge (not PROGRESS)
- Different versions of OpenEdge databases (unsupported shared memory version)
In the example above, if proutil -C dbipcs were run with 10.1C 64-bit instead of the 32-bit executable, then all the other ShMemVer=10213 lines would read (unsupported shared memory version) and this line would read:
13 64010213 0 Yes /db/dev/sports.db
- InUse - Specifies whether the segment is in use.
- YES or NO are displayed only if the segment # is 0.
- All other segments show a dash(-)
A. When DBIPCS shows: InUse = NO
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
10 10213 0 No /db/test/sports.db
Identify the process id:
$ ipcs -mp | grep 10
------ Shared Memory Creator/Last-op --------
shmid owner cpid lpid
10 root 3283 3315
1. Terminate the process holding the shared memory segment.
A shared memory segment marked for termination will release its memory only once all application pid's attached to this memory segment are terminated.
Example:
$ kill 3283 << that's the cpid from ipcs -mp
2. Remove the shared memory segment with: ipcrm -m <id>
Example:
$ ipcrm -m 10 << that's the ID_from_DBIPCS
B. When DBIPCS shows: InUse = YES
When InUse is YES, one or more processes have not exited properly and are still attached to shared memory.
1. Determine which processes are locking this shared memory.
Use a
ps command that provides the complete process command line, otherwise some database parameters (like the database name) may be truncated and therefore the grep won't find them. Take care when there are multiple databases running with the same name in different directories.
- Linux = ps -auxww
- Solaris = /usr/ucb/ps auxww
- AIX = ps -ef -auxww
Example: In the following example DBIPCS shows the database is sports which is still holding onto the shared memory.
PROGRESS SHARED MEMORY STATUS
ID ShMemVer Seg# InUse Database
6 10213 0 Yes /db/prod/sports.db
List processes that contain the word "sports" and determine whether the returned processes are the correct ones from the output listed.
$ ps -ef -auxww | grep sports > ps.out
UID PID PPID C STIME TTY TIME CMD
root 409680 1 0 13:04:40 - 0:01 /dlc/bin/_mprosrv -m3 sports
root 442502 479414 0 14:06:13 pts/1 0:00 /dlc/bin/_progres sports -b
Or use the UNIX command lsof to see if a process is associated with a particular database.
Example: Use LSOF to list all processes that have file handles that contain the word "sports".
$ lsof -f +d /db/prod | grep sports >lsof.out
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
_mprosrv 409680 root 6u VREG 45,2 655360 294957 /db/prod/sports.db
_progres 442502 root 8u VREG 45,2 655360 294957 /db/prod/sports.db
2. Terminate all processes holding the shared memory segment with SIGKILL
- Make sure the PID has no children – this is often overlooked depending on the os utility used to find these processes.
- Never pass 1 as the PID, –1 is the “init” process that defunct processes are orphaned (bound to INIT PID 1) In these situations a full server reboot is often the only recourse.
3. Remove the shared memory segment with: ipcrm -m <id>
Example:
$ ipcrm -m 6 << that's the ID_from_dpics
Final sanity check: Verify no semaphores the database was using remain
$ ipcs -s
$ ipcrm -s id
On Windows platforms, DBIPCS is only available on UNIX systems. One method to identify processes with handles still holding the shared-memory locked, is a 3rd party tool from Microsoft:
Using Microsoft Process Explorer, find handles to a database:
- Select View, Enable "show lower pane" (if it is not already enabled)
- Select View, Lower pane view > handles
- Select Find, Find Handle or Dll, > search for the database name
- Verify that any references returned relate to the database in question
- Processes listed must first be terminated with TASKILL or ENDPROC
- If processes cannot be terminated, a full machine reboot is required: C:\Windows\System32\shutdown.exe /s /f /t 0
Getting Databases running again:
If all processes associated with the database have been terminated and old shared memory segments were released or removed, or after the full system reboot, remove the database .lk file that was left behind.
Truncate the bi files of all databases, then start databases multi-user as required.
$ proutil dbname -C truncate bi -crStatus 10 -crTXDisplay
Application environments that require no access until all databases have started, another option is to start a single-user session against each database instead. This action will initiate crash recovery and lock the database to single-user until it is ready to be started multi-user after disconnecting the single-user session.
If no errors are encountered while truncating the BI file the database will be ready to start normally, otherwise a backup needs to be restored.
When further analysis is needed, gather the following:
1. Obtain a detailed process list using ps -elf or similar command showing the process that is holding onto the shared memory segment.
$ ps -elf | grep <process id>
Example: Using the cpid field from the ipcs -p example output above>
$ ps -elf | grep 3283 > ps.out
2. Obtain a stack trace from the process holding the shared memory segment.
Once the processes that are holding on to shared memory for a database have been identified, first obtain a stack trace from these processes before terminating them.
3. Retain the OpenEdge and OS logfiles
