https://redmine.czechidm.com/https://redmine.czechidm.com/themes/purplemine2/favicon/favicon.ico?16339658642017-07-02T14:47:30ZIdStory Identity ManagerIdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23042017-07-02T14:47:30ZMarcel Poulmarcel.poul@bcvsolutions.eu
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-2 priority-default prio-name-normal closed" href="/issues/561">Task #561</a>: resave user performance</i> added</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23062017-07-02T14:47:40ZMarcel Poulmarcel.poul@bcvsolutions.eu
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/2306/diff?detail_id=3419">diff</a>)</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23072017-07-02T14:48:33ZMarcel Poulmarcel.poul@bcvsolutions.eu
<ul><li><strong>Subject</strong> changed from <i>Memory management, possilbe leaks</i> to <i>Memory management, possible leaks</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23282017-07-03T09:29:49ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/63">result.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/63/result.png">result.png</a> added</li><li><strong>File</strong> <a href="/attachments/64">dominator_tree.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/64/dominator_tree.png">dominator_tree.png</a> added</li><li><strong>File</strong> <a href="/attachments/65">threads.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/65/threads.png">threads.png</a> added</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23312017-07-03T09:31:30ZRadek Tomiškaradek.tomiska@bcvsolutions.eu
<ul></ul><p>Memory on project machine:<br /><pre>
total used free shared buff/cache available
Mem: 7633 6341 139 568 1151 450
</pre><br />After dump memory<br /><pre>
jmap -dump:format=b,file=/tmp/aopk_heap.bin <PID>
</pre></p>
file has ~2.3GB, after analyze in Memory Analyzer in eclise get this result (See pic result) memory ussage: 870,4 MB:
<ul>
<li>Remainder: 491,9 MB,</li>
<li>org.codehaus.groovy.reflection.ClassInfo: 141,3 MB,</li>
<li>java.beans.ThreadGroupContext: 128,9 MB,</li>
<li>java.lang.ref.Finalizer: 108,2 MB.</li>
</ul>
<strong>Dominator Tree (children alive by node)</strong> (see dominator_tree pic)
<ul>
<li>org.codehaus.groovy.reflection.ClassInfo: 16.24%,</li>
<li>java.beans.ThreadGroupContext: 14,81%,</li>
<li>java.lang.ref.Finalizer: 12,44%.</li>
</ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23322017-07-03T09:54:23ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/66">result_unreachable.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/66/result_unreachable.png">result_unreachable.png</a> added</li><li><strong>File</strong> <a href="/attachments/67">domitaro_tree_unreachable.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/67/domitaro_tree_unreachable.png">domitaro_tree_unreachable.png</a> added</li><li><strong>File</strong> <a href="/attachments/68">max_class_usage.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/68/max_class_usage.png">max_class_usage.png</a> added</li></ul><p>Compare with unreachable object:<br />pics: result_unreachable, domitaro_tree_unreachable, max_class_usage</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23342017-07-03T11:11:31ZOndřej Kopr
<ul></ul><p>After search in open tickets in groovy project found this: <a class="external" href="https://issues.apache.org/jira/browse/GROOVY-6704">https://issues.apache.org/jira/browse/GROOVY-6704</a><br />I have created the same test as in ticket: after 30min of running unit test java consume still only about ~200mb and 7000 interactions and still continuing.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23362017-07-03T13:34:53ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/69">output_complete.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/69/output_complete.png">output_complete.png</a> added</li></ul><p>Output from jemalloc</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23432017-07-04T06:43:52ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/70">groovy_test_leak.java</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/70/groovy_test_leak.java">groovy_test_leak.java</a> added</li></ul><p>Test for memory leak in groovy script - test did not find any errors or leaks.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23482017-07-04T08:14:56ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/71">overview_sync.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/71/overview_sync.png">overview_sync.png</a> added</li><li><strong>File</strong> <a href="/attachments/72">histogram_sync_idm_objects.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/72/histogram_sync_idm_objects.png">histogram_sync_idm_objects.png</a> added</li><li><strong>File</strong> <a href="/attachments/73">histogram_sync_idm_objects.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/73/histogram_sync_idm_objects.png">histogram_sync_idm_objects.png</a> added</li><li><strong>File</strong> <a href="/attachments/74">dominator_sync.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/74/dominator_sync.png">dominator_sync.png</a> added</li><li><strong>Status</strong> changed from <i>New</i> to <i>Needs feedback</i></li><li><strong>Priority</strong> changed from <i>High</i> to <i>Normal</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>90</i></li></ul><p>During synchronization I dump again memory and verified the behavior during synchronization:</p>
<p>Memory during synchronization:<br /><pre>
total used free shared buff/cache available
Mem: 7633 4918 133 568 2582 1858
Swap: 0 0 0
</pre></p>
<p>for detail info see pics.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23492017-07-04T08:15:38ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/75">histogram_sync.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/75/histogram_sync.png">histogram_sync.png</a> added</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=23572017-07-04T10:56:02ZMarcel Poulmarcel.poul@bcvsolutions.eu
<ul></ul><p>Ondřej Kopr wrote:</p>
<blockquote>
<p>During synchronization I dump again memory and verified the behavior during synchronization:</p>
<p>Memory during synchronization:<br />[...]</p>
<p>for detail info see pics.</p>
</blockquote>
<p>I recommend to get the workflow that caused the <em>out of memory error</em> from Filip and run it again and see why it crashed if it happen again.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24202017-07-12T13:49:57ZOndřej Kopr
<ul><li><strong>Status</strong> changed from <i>Needs feedback</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>90</i> to <i>100</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24252017-07-12T17:22:41ZJan Helbichjan.helbich@bcvsolutions.eu
<ul><li><strong>File</strong> <a href="/attachments/77">postgres-caching.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/77/postgres-caching.png">postgres-caching.png</a> added</li></ul><p>I think there might be something wrong with postgres configuration.<br />I 10 threads resave entity task it seems to consume even more memory then configured - see attached pic.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24262017-07-12T18:30:28ZJan Helbichjan.helbich@bcvsolutions.eu
<ul><li><strong>Assignee</strong> changed from <i>Ondřej Kopr</i> to <i>Petr Fišer</i></li></ul><p>Postgres memory seems to be holding on reported level. Configuration:<br /><pre>
max_connections = 100
shared_buffers = 256MB
effective_cache_size = 768MB
work_mem = 2621kB
maintenance_work_mem = 64MB
min_wal_size = 1GB
max_wal_size = 2GB
checkpoint_completion_target = 0.7
wal_buffers = 7864kB
default_statistics_target = 100
</pre></p>
<p>It seems like every postmaster process has its own shared_buffers cache or something like that.<br />Or am I wrong here?<br />With described setting the database actually takes over 3.5G of memory, which is definitely not supposed to...<br />Petr, can you take a look at this? Thx</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24272017-07-12T18:57:37ZRadek Tomiškaradek.tomiska@bcvsolutions.eu
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>In Progress</i></li><li><strong>% Done</strong> changed from <i>100</i> to <i>70</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24282017-07-12T22:44:26ZJan Helbichjan.helbich@bcvsolutions.eu
<ul></ul><p>I've tried to limit metaspace to 1G -> resulted to outofmemory exception.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=24292017-07-13T06:47:33ZPetr Fišerpetr.fiser@bcvsolutions.eu
<ul><li><strong>Assignee</strong> changed from <i>Petr Fišer</i> to <i>Jan Helbich</i></li></ul><p>Jan Helbich wrote:</p>
<blockquote>
<p>Postgres memory seems to be holding on reported level. Configuration:<br />[...]</p>
<p>It seems like every postmaster process has its own shared_buffers cache or something like that.</p>
</blockquote>
<p>I wouldn't expect Postgres to be a culprit here. Databases usually have those things very thoroughly tested. It <strong>is</strong> possible, but not very probable.<br />The total sum of shared_buffers does not make sense. As by name, this part of memory is <strong>shared</strong> between the processes, so there is only one instance of it. Because the "top" tool is process-based you can see the memory listed at every postgres process - it is true, after all, that every postgres process has those shared_buffers mapped into their memory range. But physically, there is only one "instance" of shared_buffers in the memory.</p>
<p>Looking at jeprof output from Ondra: <a class="external" href="https://redmine.czechidm.com/attachments/download/69/output_complete.png">https://redmine.czechidm.com/attachments/download/69/output_complete.png</a><br />The strange thing here is so much allocation in JVM_FindSignal, I have never seen an application have massive allocations on this. Also I managed to google couple of bugs related to JVM_FindSignal, (most of them would probably be regression bugs) in the JVM itself. This would also explain, why limiting heap and/or metaspace has no effect.</p>
<p>Currently we are running on os-packaged version of jdk, that means OpenJDK. The first thing I would to would be to change to OracleJDK to see if it behaves the same way.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=25492017-07-26T13:33:48ZJan Helbichjan.helbich@bcvsolutions.eu
<ul><li><strong>Assignee</strong> changed from <i>Jan Helbich</i> to <i>Petr Fišer</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=25812017-07-28T12:03:55ZPetr Fišerpetr.fiser@bcvsolutions.eu
<ul><li><strong>File</strong> <a href="/attachments/85">munin_classes_loaded.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/85/munin_classes_loaded.png">munin_classes_loaded.png</a> added</li><li><strong>File</strong> <a href="/attachments/86">munin_heap_alloc.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/86/munin_heap_alloc.png">munin_heap_alloc.png</a> added</li><li><strong>File</strong> <a href="/attachments/87">munin_metaspace_peak.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/87/munin_metaspace_peak.png">munin_metaspace_peak.png</a> added</li><li><strong>File</strong> <a href="/attachments/88">munin_nonheap_alloc.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/88/munin_nonheap_alloc.png">munin_nonheap_alloc.png</a> added</li><li><strong>File</strong> <a href="/attachments/89">run_2h50m_top.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/89/run_2h50m_top.png">run_2h50m_top.png</a> added</li><li><strong>File</strong> <a href="/attachments/90">visualvm_heap.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/90/visualvm_heap.png">visualvm_heap.png</a> added</li><li><strong>File</strong> <a href="/attachments/91">visualvm_metaspace.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/91/visualvm_metaspace.png">visualvm_metaspace.png</a> added</li></ul><p>I got the visualvm up and running on the remote and traced what is happening. Also added "-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/idm_oomdump.dmp" to the JAVA_OPTS so we will eventually have a dump from the time of crash.<br />I have "resave all" scheduled tasks running on roles and identities. I started the "resave all roles" again (like 5th time now) every time it finished. The "resave all identities" is still going through its first run.</p>
<p>Looks like we are leaking classes somewhere. I started with roughly 60k loaded classes in the morning.<br />By now, there are about 130k loaded classes. Almost none (~5k) got unloaded.<br />Heap seems to be holding fine - JVM tries really hard to make it fit into -Xmx:3500M limit we gave it. On the other hand, Metaspace grows steadily - which makes sense if we are really leaking classes. Munin monitoring, installed by Zdenda, confirms this.</p>
<p>Attached screenshots contain more info. Especially note correlation between non-heap memory and metaspace usage.</p>
<p>Now we know what is happenning, but I do not know the culprit yet.<br />I am going to let the JVM OOM so it will produce a dump. I hope I will find out more from the dump.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=25832017-07-28T12:49:31ZVít Švanda
<ul></ul><p>Maybe is problem still in Groovy scripts (as Ondra wrote). We have minimal three places where we using Groovy scripts:</p>
<ul>
<li>GroovyScirptService - but here is now using cache for same scirpts.</li>
<li>Workflow - for example process for change user permissions use many of Groovy scripts and there is not any cache for same scripts. In standard situation (on AOPK may be yes), is not call any workflow after save identity. </li>
<li>JDBC scripted connector - This is my favorite, because here is not cache for script too and connector is used after identity is saved.</li>
</ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=25842017-07-28T12:54:25ZPetr Fišerpetr.fiser@bcvsolutions.eu
<ul></ul><p>Can be, thanks for hints. My personal tip is the JAXB we use for deserializing XMLs. JAXB itself uses WeakHashMap and such things. Those, when handled improperly, can very easily create a leak.<br />But first I need to collect the dump - IdM has been chugging memory away almost whole day and I do not want to throw the invested time away. :)</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=25942017-07-31T10:36:31ZOndřej Kopr
<ul></ul><p>After some discussion with colleagues, we think problem may be in the script connector. For our scripts is cache (from Filip), but for script connector isn't. Please try deactivate system with script connector.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=26212017-08-01T12:57:16ZPetr Fišerpetr.fiser@bcvsolutions.eu
<ul><li><strong>File</strong> <a href="/attachments/93">dominator_cake.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/93/dominator_cake.png">dominator_cake.png</a> added</li><li><strong>File</strong> <a href="/attachments/92">dominator_drops.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/92/dominator_drops.png">dominator_drops.png</a> added</li><li><strong>File</strong> <a href="/attachments/95">leakhunter_accu_by_class.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/95/leakhunter_accu_by_class.png">leakhunter_accu_by_class.png</a> added</li><li><strong>File</strong> <a href="/attachments/94">leakhunter_class_dominator.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/94/leakhunter_class_dominator.png">leakhunter_class_dominator.png</a> added</li><li><strong>File</strong> <a href="/attachments/96">leakhunter_obj_dominator.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/96/leakhunter_obj_dominator.png">leakhunter_obj_dominator.png</a> added</li><li><strong>File</strong> <a href="/attachments/97">leakhunter_overview.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/97/leakhunter_overview.png">leakhunter_overview.png</a> added</li><li><strong>File</strong> <a href="/attachments/98">leakhunter_path_to_accupoint.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/98/leakhunter_path_to_accupoint.png">leakhunter_path_to_accupoint.png</a> added</li></ul><p>There is a bit better view on the issue. I started on IdM with ~30k loaded classes, I ran "resave all identities" and "resave all roles" until the count of loaded classes was about ~60k - this means potentially one half of 60k classes leaked. Then I examined heap dump with leakhunter from eclipse memory analyzer.<br />One identified issue which occupies about one half of the heap (so this neatly corresponds with "one half of loaded classes leaked"). See leakhunter_overview.png.</p>
<p>So there is obviously problem with <strong>org.codehaus.groovy.reflection.ClassInfo</strong> and (as by other screenshots) with <strong>groovy.lang.MetaClassImpl</strong> classes. This really points out that classes loaded through groovy do not get unloaded anywhere.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=26302017-08-02T13:29:08ZOndřej Kopr
<ul><li><strong>Assignee</strong> changed from <i>Petr Fišer</i> to <i>Ondřej Kopr</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=26672017-08-08T06:23:19ZOndřej Kopr
<ul><li><strong>File</strong> <a href="/attachments/102">classes_loaded_result.png</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/102/classes_loaded_result.png">classes_loaded_result.png</a> added</li><li><strong>% Done</strong> changed from <i>70</i> to <i>90</i></li></ul><p><strong>Possibly a final judgment:</strong> as you se in picture (classes_loaded_result.png) after disable scripted connector, or change this connector to table is loaded class apparently less than when we use scripted connector. <strong>So problem is definitely in scripted connector</strong>, we tried to find some solution, but we did not come to anything yet.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=27512017-08-15T13:42:00ZOndřej Kopr
<ul><li><strong>Assignee</strong> changed from <i>Ondřej Kopr</i> to <i>Jan Helbich</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=30762017-09-27T11:03:46ZJan Helbichjan.helbich@bcvsolutions.eu
<ul><li><strong>Assignee</strong> changed from <i>Jan Helbich</i> to <i>Ondřej Kopr</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36102017-11-02T09:41:39ZOndřej Kopr
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>90</i> to <i>100</i></li></ul><p>I get hit from Alca about connector configuration <strong>reloadScriptOnExecution</strong> when this configuration is set to true is all scripts loaded and compiled every time when connector call script, but on project is this configuration property set to false :(</p>
<p>Also exists new version of this connector 2.2.5 but in this version is only add connector configuration to script and another minor fixes.</p>
<p>Found some discussion on <a class="external" href="https://stackoverflow.com/questions/36407119/groovyshell-in-java8-memory-leak-duplicated-classes-src-code-load-test-pr">https://stackoverflow.com/questions/36407119/groovyshell-in-java8-memory-leak-duplicated-classes-src-code-load-test-pr</a></p>
<p>on our script evaluator is already similar cache exists, but in <a class="external" href="https://github.com/Evolveum/openicf/blob/release1.4.2.0/framework/java/connector-framework-internal/src/main/java/org/identityconnectors/common/script/groovy/GroovyScriptExecutorFactory.java#L67-L71">https://github.com/Evolveum/openicf/blob/release1.4.2.0/framework/java/connector-framework-internal/src/main/java/org/identityconnectors/common/script/groovy/GroovyScriptExecutorFactory.java#L67-L71</a> isn't.</p>
<p>If we forked connector-framework-internal it is possible update this class with similar cache like in our DefaultGroovyScriptService.</p>
<p>For now I recommend to use script connector on remote connector server, because when will be connector server shut down by memory leak is less problem than shut down IdM.</p>
<p>I write some resume of this ticket into documentation and for now I close this ticket (in anoher ticket is possible fork connector-framework-internal).</p>
<p>documentation: <a class="external" href="https://wiki.czechidm.com/devel/dev/system/supported-connectors#probable_memory_leak_in_scripted_sql_connector">https://wiki.czechidm.com/devel/dev/system/supported-connectors#probable_memory_leak_in_scripted_sql_connector</a></p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36112017-11-02T09:48:51ZMarcel Poulmarcel.poul@bcvsolutions.eu
<ul></ul><p>Ondřej Kopr wrote:</p>
<blockquote>
<p>I get hit from Alca about connector configuration <strong>reloadScriptOnExecution</strong> when this configuration is set to true is all scripts loaded and compiled every time when connector call script, but on project is this configuration property set to false :(</p>
</blockquote>
<p>So should we do something on projects? If so, please discuss with relevant people...</p>
<p>Thx</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36122017-11-02T09:51:46ZOndřej Kopr
<ul></ul><p>Marcel Poul wrote:</p>
<blockquote>
<p>Ondřej Kopr wrote:</p>
<blockquote>
<p>I get hit from Alca about connector configuration <strong>reloadScriptOnExecution</strong> when this configuration is set to true is all scripts loaded and compiled every time when connector call script, but on project is this configuration property set to false :(</p>
</blockquote>
<p>So should we do something on projects? If so, please discuss with relevant people...</p>
<p>Thx</p>
</blockquote>
<p>This configuration property is by default turn off, so it not necessary set anything on any projects.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36152017-11-02T09:55:18ZAlena Peterováalena.peterova@bcvsolutions.eu
<ul></ul><p>Ondřej Kopr wrote:</p>
<blockquote>
<p>This configuration property is by default turn off, so it not necessary set anything on any projects.</p>
</blockquote>
<p>Yes, it is turned off, but does it really work? In the log it looked like the scripts were compiled every time. Definitely when we made the changes to groovy scripts, they were loaded immediately.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36162017-11-02T10:03:00ZOndřej Kopr
<ul></ul><p>Alena Peterová wrote:</p>
<blockquote>
<p>Ondřej Kopr wrote:</p>
<blockquote>
<p>This configuration property is by default turn off, so it not necessary set anything on any projects.</p>
</blockquote>
<p>Yes, it is turned off, but does it really work? In the log it looked like the scripts were compiled every time. Definitely when we made the changes to groovy scripts, they were loaded immediately.</p>
</blockquote>
<p>On my local build i set up new Scripted SQL Connector, create all necessary scripts (create, update and search) and debug this class: <a class="external" href="https://github.com/Tirasa/ConnIdDBBundle/blob/db-2.2.4/scriptedsql/src/main/java/net/tirasa/connid/bundles/db/scriptedsql/ScriptedSQLConnector.java">https://github.com/Tirasa/ConnIdDBBundle/blob/db-2.2.4/scriptedsql/src/main/java/net/tirasa/connid/bundles/db/scriptedsql/ScriptedSQLConnector.java</a></p>
<p>and check every place that has similar if statement like this:</p>
<pre>
if (config.isReloadScriptOnExecution()) {
searchExecutor = getScriptExecutor(config.getSearchScript(), config.getSearchScriptFileName());
LOG.ok("Search script loaded");
}
</pre>
<p>if is this configuration property turn off it not reached body of this if statement, so this property works well.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36172017-11-02T10:05:12ZAlena Peterováalena.peterova@bcvsolutions.eu
<ul></ul><p>Then maybe the problem is in multiple calling the Init method:<br /><pre>
Thread Id: 52 Time: 2017-11-02 11:02:07.797 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:155) Level: OK Message:
Create script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.207 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:158) Level: OK Message:
Update script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.352 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:161) Level: OK Message:
Delete script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.649 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:164) Level: OK Message: Search script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.650 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:167) Level: OK Message: Sync script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.650 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:170) Level: OK Message: Sync script loaded
Thread Id: 52 Time: 2017-11-02 11:02:08.651 Class: net.tirasa.connid.bundles.db.scriptedsql.ScriptedSQLConnector Method: init(ScriptedSQLConnector.java:173) Level: OK Message: Test script loaded
</pre></p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36192017-11-02T10:36:21ZOndřej Kopr
<ul><li><strong>Assignee</strong> changed from <i>Ondřej Kopr</i> to <i>Vít Švanda</i></li><li><strong>% Done</strong> changed from <i>100</i> to <i>90</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36202017-11-02T11:18:53ZVít Švanda
<ul><li><strong>Status</strong> changed from <i>Closed</i> to <i>In Progress</i></li><li><strong>Target version</strong> set to <i>Forsterite (7.6.0)</i></li></ul><pre>
Alena Peterová wrote: Then maybe the problem is in multiple calling the Init method.
</pre><br />It makes sense, because pooling for ConnId (in IC modul) was not never implemented.<br />I will try to simulating the problem and use the pooling for this case. IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36392017-11-03T11:19:52ZVít Švanda
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-5 priority-2 priority-default prio-name-normal closed" href="/issues/816">Task #816</a>: Implement connector pooling</i> added</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=36402017-11-03T11:32:22ZVít Švanda
<ul></ul><p>I did some tests and prototype for using pool in JDBC scripted connector. <br />When is pool used and connector facade are cached in IC module and checkbox 'reloadScriptOnExecution' is unchecked, then are GroovyScriptExecutors reusing (only GroovyScriptExecutor.execute is calling everytime).<br />I can`t now confirm that problem with memory leak will be definitally solved (I tested it only on 1000 accounts), but I think this is correct way.</p>
<p>I created task <a class="issue tracker-2 status-5 priority-2 priority-default prio-name-normal closed" title="Task: Implement connector pooling (Closed)" href="https://redmine.czechidm.com/issues/816">#816</a> for implement pool and cache in IC module. I will consulted with Zdeněk, when will be this task implemented.</p> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=39692017-12-07T09:47:09ZVít Švanda
<ul><li><strong>Target version</strong> deleted (<del><i>Forsterite (7.6.0)</i></del>)</li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=51692018-05-18T10:22:22ZVít Švanda
<ul><li><strong>Assignee</strong> changed from <i>Vít Švanda</i> to <i>Ondřej Kopr</i></li></ul> IdStory Identity Manager - Task #563: Memory management, possible leakshttps://redmine.czechidm.com/issues/563?journal_id=51702018-05-18T11:01:36ZOndřej Kopr
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Closed</i></li><li><strong>% Done</strong> changed from <i>90</i> to <i>100</i></li></ul><p>Thanks Vitek for give me this hot apple! :)</p>
<p>The ticket will be probably solved by task <a class="issue tracker-2 status-5 priority-2 priority-default prio-name-normal closed" title="Task: Implement connector pooling (Closed)" href="https://redmine.czechidm.com/issues/816">#816</a>. Thank all of interested people for help and debug.</p>