Task #1806
open
ClearDirtyStateForContractSliceTaskExecutor runs in single transaction when started after Synchronization
0%
Description
Version 9.7.3
ClearDirtyStateForContractSliceTaskExecutor runs in one single transaction when started after Synchronization. This is problematic when there are many time slices of contracts or many EAV's of contracts. It fails to finish. And when it fails, the processed dirty flags are not deleted from the DB => we are at the beginning.
When ClearDirtyStateForContractSliceTaskExecutor starts as a scheduled task, it processes every dirty flag separately. So when it fails, we can continue where we left.
Please change it so ClearDirtyStateForContractSliceTaskExecutor doesn't run in single transaction. And also please solve #1413, which slows it down very much.
Details:
We have 64919 slices after reconcilation, every contract (slice) has 10 EAVs. The ClearDirtyStateForContractSliceTaskExecutor after synchronization processed 27669 in 6 hours, but no change was commited. No entity event was visible in IdM and no provisioning was done (I don't know if they started or not...).
When I cancelled the task manually, it started to commit the big transaction. After 2 hours, it still wasn't commited. When I restarted IdM, all was lost.
During the 2 hours, the database queries almost always inserted something in idm_audit or selected next value from hibernate_sequence. The hibernate_sequence used cca 20 000 IDs.
Now I run ClearDirtyStateForContractSliceTaskExecutor from scheduled task. It has processed 24230 in 6,5 hours and everything is commited. The hibernate_sequence used cca 300 000 IDs.
There is evidently some problem with the big transaction for all slices - it's much slower than single transactions for every slice.
Some screenshots are attached - how the task was canceled and what happened in PostgreSQL during the commit.
Files
