Project

General

Profile

Actions
Copy link

Defect #3069

open

Potential memory leak in remote system initialization

Added by Ondřej Kopr almost 3 years ago. Updated almost 3 years ago.

Status:
In Progress
Priority:
Urgent
Assignee:
Peter Štrunc
Category:
Performance
Target version:
-
Start date:
03/05/2022
Due date:
% Done:

0%

Estimated time:
Affected versions:
10.8.3
Owner:

Description

In our environment is a big slowdown in the slice reconciliation (reconciliation provides only removing).

After the first analysis, we thought that there were to many records in the table sys_sync_item_long (about 3mil). So we executed delete on the table (300k records left) but in synchronization isn't time improvement.

But after restart whole application server reconciliation is again fast.

See this image:

Before restart 6hours and after restart 2hours.

We have heap dump from tomcat process. In dump exists many eu.bcvsolutions.idm.acc.dto.SysSystemDto that was for AD system, AD system isn't for synchronization but only for provisioning. The AD system is connected to IdM via remote connector. In dump also exists eu.bcvsolutions.idm.acc.dto.SysConnectorServerDto, eu.bcvsolutions.idm.acc.dto.SysConnectorKeyDto, eu.bcvsolutions.idm.acc.dto.SysSchemaObjectClassDto. See picture:

I'm not able add dump into the ticket (size) if you want it just write me.

We try vacuum on every big table, add more service task with removing old logs, but any attempts don't provides us better synchronization time, except restart whole application server.

IdM version: 10.8.3 (lts)

For our project is this very important, because temporary workarround with planed weakly restarts isn't acceptable for customer.

Please could you check this presumption for slowdown? It's only idea that we have.

Files

Download all files
before_restart.png (43.7 KB) before_restart.png Ondřej Kopr, 03/06/2022 07:26 PM
dump.png (243 KB) dump.png Ondřej Kopr, 03/06/2022 07:27 PM
Actions
Copy link
 #1

Updated by Roman Kučera almost 3 years ago

  • Status changed from New to In Progress
@kopro
I have few questions about environment.
  • How many provisioning operation are there per day to that AD system?
  • The connector is just simple AD conn or WinRM + AD?
  • Pooling in IdM is enabled?
  • Slowdown from 2 to 6 hours in in one week?
  • Is it possible to simulate it on customer test environment?

Send me the dump pls.

Actions
Copy link
 #2

Updated by Ondřej Kopr almost 3 years ago

Roman Kučera wrote in #note-1:

  • How many provisioning operation are there per day to that AD system?

About 1k in working day and 200 in weekend.

1.1.2022 - 3890
2.1.2022 - 120
15.1.2022 - 150
1.2.2022 - 960

  • The connector is just simple AD conn or WinRM + AD?

Remote connector, winrm + ad - power shell script creates users home directories

  • Pooling in IdM is enabled?

No. Is pooling implemented with winrm connector for power shell scripts?

  • Slowdown from 2 to 6 hours in in one week?

about 1 month

  • Is it possible to simulate it on customer test environment?

No, test HR system is different and there isn't any changes during days on the environment.

Send me the dump pls.

Dump is there: 172.31.255.96:/root/dump/dump-2-3-2022.bin

Actions
Copy link

Also available in: Atom PDF