IdStory Identity Manager

Memory on project machine:

                     total        used        free      shared  buff/cache   available
Mem:           7633        6341         139         568        1151         450

jmap  -dump:format=b,file=/tmp/aopk_heap.bin <PID>

• Remainder: 491,9 MB,
• org.codehaus.groovy.reflection.ClassInfo: 141,3 MB,
• java.beans.ThreadGroupContext: 128,9 MB,
• java.lang.ref.Finalizer: 108,2 MB.

• org.codehaus.groovy.reflection.ClassInfo: 16.24%,
• java.beans.ThreadGroupContext: 14,81%,
• java.lang.ref.Finalizer: 12,44%.

After search in open tickets in groovy project found this: https://issues.apache.org/jira/browse/GROOVY-6704
I have created the same test as in ticket: after 30min of running unit test java consume still only about ~200mb and 7000 interactions and still continuing.

Ondřej Kopr wrote:

During synchronization I dump again memory and verified the behavior during synchronization:

Memory during synchronization:
[...]

for detail info see pics.

I recommend to get the workflow that caused the out of memory error from Filip and run it again and see why it crashed if it happen again.

I've tried to limit metaspace to 1G -> resulted to outofmemory exception.

Maybe is problem still in Groovy scripts (as Ondra wrote). We have minimal three places where we using Groovy scripts:

• GroovyScirptService - but here is now using cache for same scirpts.
• Workflow - for example process for change user permissions use many of Groovy scripts and there is not any cache for same scripts. In standard situation (on AOPK may be yes), is not call any workflow after save identity.
• JDBC scripted connector - This is my favorite, because here is not cache for script too and connector is used after identity is saved.

Can be, thanks for hints. My personal tip is the JAXB we use for deserializing XMLs. JAXB itself uses WeakHashMap and such things. Those, when handled improperly, can very easily create a leak.
But first I need to collect the dump - IdM has been chugging memory away almost whole day and I do not want to throw the invested time away. :)

After some discussion with colleagues, we think problem may be in the script connector. For our scripts is cache (from Filip), but for script connector isn't. Please try deactivate system with script connector.

Ondřej Kopr wrote:

I get hit from Alca about connector configuration reloadScriptOnExecution when this configuration is set to true is all scripts loaded and compiled every time when connector call script, but on project is this configuration property set to false :(

So should we do something on projects? If so, please discuss with relevant people...

Thx

Marcel Poul wrote:

Ondřej Kopr wrote:

I get hit from Alca about connector configuration reloadScriptOnExecution when this configuration is set to true is all scripts loaded and compiled every time when connector call script, but on project is this configuration property set to false :(

So should we do something on projects? If so, please discuss with relevant people...

Thx

This configuration property is by default turn off, so it not necessary set anything on any projects.

Ondřej Kopr wrote:

This configuration property is by default turn off, so it not necessary set anything on any projects.

Yes, it is turned off, but does it really work? In the log it looked like the scripts were compiled every time. Definitely when we made the changes to groovy scripts, they were loaded immediately.

Alena Peterová wrote:

Ondřej Kopr wrote:

This configuration property is by default turn off, so it not necessary set anything on any projects.

Yes, it is turned off, but does it really work? In the log it looked like the scripts were compiled every time. Definitely when we made the changes to groovy scripts, they were loaded immediately.

On my local build i set up new Scripted SQL Connector, create all necessary scripts (create, update and search) and debug this class: https://github.com/Tirasa/ConnIdDBBundle/blob/db-2.2.4/scriptedsql/src/main/java/net/tirasa/connid/bundles/db/scriptedsql/ScriptedSQLConnector.java

and check every place that has similar if statement like this:

 if (config.isReloadScriptOnExecution()) {
     searchExecutor = getScriptExecutor(config.getSearchScript(), config.getSearchScriptFileName());
     LOG.ok("Search script loaded");
}

if is this configuration property turn off it not reached body of this if statement, so this property works well.

Then maybe the problem is in multiple calling the Init method:

Thread Id: 52   Time: 2017-11-02 11:02:07.797   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:155)     Level: OK       Message: 
Create script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.207   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:158)     Level: OK       Message: 
Update script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.352   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:161)     Level: OK       Message: 
Delete script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.649   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:164)     Level: OK       Message: Search script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.650   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:167)     Level: OK       Message: Sync script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.650   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:170)     Level: OK       Message: Sync script loaded
Thread Id: 52   Time: 2017-11-02 11:02:08.651   Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector    Method: init(ScriptedSQLConnector.java:173)     Level: OK       Message: Test script loaded

I did some tests and prototype for using pool in JDBC scripted connector.
When is pool used and connector facade are cached in IC module and checkbox 'reloadScriptOnExecution' is unchecked, then are GroovyScriptExecutors reusing (only GroovyScriptExecutor.execute is calling everytime).
I can`t now confirm that problem with memory leak will be definitally solved (I tested it only on 1000 accounts), but I think this is correct way.

I created task #816 for implement pool and cache in IC module. I will consulted with Zdeněk, when will be this task implemented.