We have been discussing this for a long time now and it looks like if there was a mean to time out rollback operation that BaseConnection class does while cleaning up of resources, it could certainly most of the time save system from crashing.
Just to clarify on the situation, when the system is heavily bombed with bulk request(mostly on particular days, if it is a vertical solution), there are chances of high lock contention that might lead to more threads waiting. Obviously there are threads that consumes more memory than usual threads, what if these threads are more in number? In the situation of high request volume and more memory consuming threads, JVM calls for the GC. Unfortunately, if there are some explicit close call made on connection which has associated transaction pending, BaseConnection would try rolling it back and you would see traces like below:
Now, that this call was not getting a a signal back for proper rollback from DB, it is still waiting there holding up the finalizer thread where rest of isolated and GC qualified objects eating up the memory remain in the heap because the corresponding cleanup calls are waiting in the queue(to get hold of finalizer thread), eventually leading to memory starvation and crashing the server.
Assuming this to be one scenario, if there was a mean to tell BaseConnection class that rollback didn't get a response back from DB(for example xyz-timeout period of time) and we know waiting on finalizer for this long is costly, just timeout and let others do their job in finalizer, it would save JVM from reaching OOM.
There is no workaround to this. Administrator and developers can just have JVM memory set to its best and work on reducing locks contention, which is mostly taken care of as far as possible.