Task #972
closed
Slow synchronization of Identities
20%
Description
Setup:
Win server (virtualized)
8GB RAM for backend
DB separated server (Win and 8GB too)
Synchronization of users took about 10,5 hours for 2680 identities. That is around 14sec for one identity. IdM was clean, no identity except basic admin.
It seemed to me that the first 100 users were synchronized really quickly - 1 minute or so. Then I checked again when about 1500 identities were synced and average was 9 sec per identity. At the end of the sync, average was 14 sec per identity.
Setup for identities sync. is very basic. 15 attributes in mapping (usually string) only 1 transformation used (login generation). ConnId DB Table connector used.
During the sync of identities (in 75%) I checked Windows task manager and CPU usage was 5% and memory 38%.
I also tried sync of Contracts and 500 contracts took several minutes (started during identities sync and then again for comparison after identity sync had ended). Always very quick.
Related issues
Updated by Vít Švanda about 6 years ago
- Assignee changed from Vít Švanda to Ondřej Kopr
Updated by Marcel Poul about 6 years ago
- Priority changed from Normal to Urgent
- average 24 seconds per one identity sync. (after 200 identities)
- the start was slow too
Alena told me that the same situation apply for other new project with current CzechIdM version 7.7
Currently it delays us from working on our project.
Updated by Marcel Poul about 6 years ago
Marcel Poul wrote:
It is getting worse. Another run:
- average 24 seconds per one identity sync. (after 200 identities)
- the start was slow too
Alena told me that the same situation apply for other new project with current CzechIdM version 7.7
Currently it delays us from working on our project.
After 14 hours, about half of the identities is synchronized and currently avarage sync speed is 30 seconds per identity. Only operation done is Create entity. No provisioning, no nothing.
Again I tried running sync. of Contracts next to Identity sync. and 200 Contracts took several seconds.
Updated by Ondřej Kopr about 6 years ago
- Status changed from New to In Progress
- % Done changed from 0 to 90
I checked one of your projects:
number of operations: 5429
diff between first two identites: 0,7 sec,
diff between two lasts identites: 32 sec,
audit is filled only by SysSyncIdentityConfig on this entity is changed only token and modified in ticket: #939 this behavior is removed (will be part of 8.0.0 release :() (maybe next hotfix?)
When I disable audit, synchronization with 5430 operations is done about 10 minutes (this is only temporary solution, in ticket #939 is removed token audit):
diff between first two identites: ~0,2 sec,
diff between two lasts identites: ~0,2 sec,
You can temporary turn off audit by property (this attribute can't be added by gui configuration and required server restart):
spring.jpa.properties.hibernate.listeners.envers.autoRegister=false
So final solution will be apeare with ticket #939. In this ticket will be removed audit from token and some audit improvements.
Updated by Alena Peterová about 6 years ago
Would it work, if I turn off audit during initial import and turn it on afterwards? If e.g. the lastname of identity changes, what will be shown in the audit log detail?
Updated by Ondřej Kopr about 6 years ago
Alena Peterová wrote:
Would it work, if I turn off audit during initial import and turn it on afterwards? If e.g. the lastname of identity changes, what will be shown in the audit log detail?
Audit in this case is very defensive, on some project we copied configuration except entities (without audit logs).
After your again turn on audit you haven't audit with create or update from synchronization, but next steps for audit will works correctly. For example:
- create entity by synchronization audit is turn off
- after synchronization turn on audit
- edit entity (do some modification)
- idm created audit log with type MOD, previous version doesn't exists, only 'change columns' attribute will not be calculate, everything else will works correctly
- edit entity again
- idm created audit log with correctly behavior, except 'change columns'.
Updated by Petr Hanák about 6 years ago
Same source.. reconcillation with audit - 5430 operations in 7 mins.
Updated by Marcel Poul about 6 years ago
- % Done changed from 90 to 20
almost done on my project - 2600/2800 after 24 hours of synchronization is average 40 seconds identity. So the synchronization is slowing down constantly to the extremity. Since the synchronization is started as Reconciliation, the audit improvement for synchronization token auditing will not work for our project. Tomorrow we are going to try 7.8.2 version with various improvements and we will see.
Updated by Ondřej Kopr about 6 years ago
- Related to Task #939: Add index for audit tables added
Updated by Alena Peterová about 6 years ago
Auditing isn't the problem here.
One more example (current version 7.8.1), where I:- started Synchronization of identities, it took cca 8s to update 1 identity
- stopped the synchronization
- stopped Tomcat
- turned off auditing in the application-production.properties
- started Tomcat
- changed synchronization to Reconciliation
- started Reconciliation. It took cca 8s to update 1 identity.
- checked that audit logs are really turned off (no new record in the GUI)
There are 16 000 identities, so I expect the synchronization to run around 2 days :-( When it was run on version 7.5.3, it took only 2 hours to create all identities. There are a few changes in the data, but not many. It was originally connected by CSV connector, now it runs on DatabaseTable, but that shouldn't make any difference.
After a few minutes, I continued with the synchronization of contracts. This got stuck after 5 contracts, so I can't tell if it is faster or not.
The Catalina log was full of exceptions about starting some Hr process and that it can't run in two instances. These exceptions were there even after stopping the contracts' synchronizations, so it looks like it's caused by identities' synchronization.
Updated by Vít Švanda about 6 years ago
Do you use some automatic role or defalut role on sync?
Do identities have a default contract?
- version 7.8.2
- audits turn on
- Source local DB table
- 20000 records in source table
- Identity have mapped 4 attributes and 2 EAV attributes.
- Default contract is not created.
- None default role is set by sync.
- I created 2755 identies / 45 min = cca 1 identity per 1s.
Updated by Alena Peterová about 6 years ago
Vít Švanda wrote:
Do you use some automatic role or defalut role on sync?
I have ~90 automatic roles for attributes, the automatic rules are all on CONTRACT_EAV.
I don't have default role.
Does identities have a default contract?
No, creating Default contract is turned off. All identities have 1 non-default contract, which was synchronized in the past
I have mapped 18 EAVs (all text without transformation) and these identity attributes:
username - transformation - padding by 0 from the left to 5 characters
email
firstname
lastname
titlesBefore
titlesAfter
telephone
--------------------------
If you run the synchronization again with Linked->Update entity, is it still fast?
Updated by Marcel Poul about 6 years ago
Vít Švanda wrote:
Do you use some automatic role or defalut role on sync?
no roles for me
Do identities have a default contract?
no default contract
Updated by Vít Švanda about 6 years ago
I tried update identities too and speed is same ~1/1s.
Updated by Marcel Poul about 6 years ago
Tomorrow, Ondra is going to upgrade on our project to 7.8.2 and we will see.
Updated by Ondřej Kopr about 6 years ago
- Related to Task #980: Optimization automatic role by attribute added
Updated by Ondřej Kopr about 6 years ago
It will be implemented optimization for automatic role attribute in ticket: #980
Updated by Marcel Poul about 6 years ago
- Priority changed from Urgent to High
.Ondra updated IdM on 7.8.2 version and it worked for me. Now 2680 identities takes about 15 minutes. There are some differences between this run and the slow one (update entity in sync and server restarted), but I hope It make no difference.
Updated by Ondřej Kopr about 6 years ago
- Status changed from In Progress to Closed
I close the ticket problem with slow synchronization is resolved by ticket #980 and change audit behavior