Linux Database crashes when semaphores get removed

Database crashes when its semaphores get removed.

Whenever any command is run against the database using as the same user account that started the databases, semaphores get removed when the user logs off and the database crashes.

The database crashes with semaphore errors every time a user account that runs a database administrative task cron session finishes

The database is started with a user account where the corresponding semaphores are owned by this user:

As long as the same user account that owns the semaphores does not log off the database remains up and running.

The database starts with appropriate semaphores sets (-semsets) and the kernel has been tuned with SEMMSL 250, SEMMNI 256 and SEMMNS 32000. Article Calculating SEMMSL, SEMMNS and SEMMNU  

The default value for RemoveIPC in RHEL 7.2 is yes

• The Linux parameter RemoveIPC=yes in etc/systemd/logind.conf is causing the issue when the user is the non-system user.
• The RemoveIPC= logic only applies to non-system users. It is disabled entirely for system users

When user accounts start Applications using IPC (semaphores, shared memory, message queues) do not take this into account, they will experience problems after update to RHEL 7.2 or later when that user account logs off. 

There is a difference in behavior between system and regular users. IPC objects of the root user and system users are excluded from the effect of this setting, otherwise it is a configurable setting as documented in the Linux man pages. Any user that starts processes & daemons requiring communication & synchronization mechanisms need to be started by system users which are excluded from IPC removal (shared memory and semaphores). Otherwise the default behavior for systemd is to “clean slate on logout” ensuring the user no longer consumes IPC resources.
 

Set up accounts that start databases appropriately.

OPTION 1: Create the user as a system user.

The RemoveIPC logic only applies to non-system users. It is disabled entirely for system users determined at systemd compile time from the SYS_UID_MAX setting in /etc/login.defs. Ensure the user is registered properly as system user (i.e. with a UID < 1000) instead of regular user.

A user created with the system option (adduser -r or adduser --system)

OPTION 2: For regular user accounts prevent semaphores being removed when the user account logs out.

Edit /etc/systemd/logind.conf, change RemoveIPC=no, save and reboot the system or restart systemd-logind service.     

Additional Login manager configuration considerations:

Another configuration setting is KillUserProcesses, which controls whether all processes of a user's session should be killed when the user logs out of that session.

KillUserProcesses defaults to yes since systemd version 230 and will close background processes started from a user shell for example.
The logind.conf man page explains how to configure the default behavior with: KillOnlyUsers, KillExcludeUsers, KillUserProcesses, loginctl enable-linger, systemd-run. 

• Login manager configuration files: https://www.freedesktop.org/software/systemd/man/logind.conf.html

Progress Article:
Database stops abnormally after showing the error (10839)