Project

General

Profile

Task #1806

ClearDirtyStateForContractSliceTaskExecutor runs in single transaction when started after Synchronization

Added by Alena Peterová about 2 years ago. Updated about 1 month ago.

Status:
New
Priority:
High
Assignee:
Vít Švanda
Category:
Long running task
Target version:
Start date:
08/20/2019
Due date:
% Done:

0%

Estimated time:
Milestones:

Description

Version 9.7.3

ClearDirtyStateForContractSliceTaskExecutor runs in one single transaction when started after Synchronization. This is problematic when there are many time slices of contracts or many EAV's of contracts. It fails to finish. And when it fails, the processed dirty flags are not deleted from the DB => we are at the beginning.
When ClearDirtyStateForContractSliceTaskExecutor starts as a scheduled task, it processes every dirty flag separately. So when it fails, we can continue where we left.

Please change it so ClearDirtyStateForContractSliceTaskExecutor doesn't run in single transaction. And also please solve #1413, which slows it down very much.

Details:
We have 64919 slices after reconcilation, every contract (slice) has 10 EAVs. The ClearDirtyStateForContractSliceTaskExecutor after synchronization processed 27669 in 6 hours, but no change was commited. No entity event was visible in IdM and no provisioning was done (I don't know if they started or not...).
When I cancelled the task manually, it started to commit the big transaction. After 2 hours, it still wasn't commited. When I restarted IdM, all was lost.
During the 2 hours, the database queries almost always inserted something in idm_audit or selected next value from hibernate_sequence. The hibernate_sequence used cca 20 000 IDs.
Now I run ClearDirtyStateForContractSliceTaskExecutor from scheduled task. It has processed 24230 in 6,5 hours and everything is commited. The hibernate_sequence used cca 300 000 IDs.
There is evidently some problem with the big transaction for all slices - it's much slower than single transactions for every slice.

Some screenshots are attached - how the task was canceled and what happened in PostgreSQL during the commit.

hibernate_sequence_value.png (11.4 KB) hibernate_sequence_value.png Alena Peterová, 08/20/2019 03:21 PM
postgres_transaction.png (56.3 KB) postgres_transaction.png Alena Peterová, 08/20/2019 03:21 PM
cancel_task.png (16.9 KB) cancel_task.png Alena Peterová, 08/20/2019 03:21 PM
dirty_states_DB.png (6.38 KB) dirty_states_DB.png Alena Peterová, 08/20/2019 03:21 PM
scheduled_tasks.png (54.9 KB) scheduled_tasks.png Alena Peterová, 08/20/2019 03:21 PM
lock.xlsx (7.79 KB) lock.xlsx Ondřej Kopr, 05/20/2021 05:20 PM
query.xlsx (6.28 KB) query.xlsx Ondřej Kopr, 05/20/2021 05:20 PM
003.png (66.7 KB) 003.png Ondřej Kopr, 05/20/2021 05:22 PM
002.png (68.8 KB) 002.png Ondřej Kopr, 05/20/2021 05:22 PM
001.png (40.9 KB) 001.png Ondřej Kopr, 05/20/2021 05:22 PM
004.png (128 KB) 004.png Ondřej Kopr, 05/20/2021 05:25 PM
takes.png (74 KB) takes.png Ondřej Kopr, 05/21/2021 05:28 AM

History

#1 Updated by Radek Tomiška about 2 years ago

  • Assignee changed from Radek Tomiška to Ondřej Kopr

#2 Updated by Alena Peterová about 2 years ago

More numbers:
4181 dirty flags after synchronization - ClearDirtyStateForContractSliceTaskExecutor run 57 minutes - commit of the big transaction took another 32 minutes.

#3 Updated by Ondřej Kopr 5 months ago

The issues still remain even in new version 10.7.2.

After first initialization project was synchronize about 25k time slices. All behavior with creating new slices eavs was done about 40 minutes and then 4 hours running clear dirty state without progress information, or some informations.

From postgres activity can be obtain information about query that run from 2021-05-20 15:13:25 to 2021-05-20 19:13:54 and still running :(

More information in attached files.

Synchronization was started in [2021-05-20T14:46:25.838997+02:00[Europe/Prague]].
-------------------------
Synchronization is running as differential (entities will be updated only if least one attribute was changed).
-------------------------
Synchronization will use inner connector synchronization implementation.
-------------------------
Synchronization was correctly ended in [2021-05-20T15:28:20.933353+02:00[Europe/Prague]].

I will let the task run during night and we will see in the morning.

#4 Updated by Ondřej Kopr 5 months ago

  • File deleted (004.png)

#5 Updated by Ondřej Kopr 5 months ago

#6 Updated by Ondřej Kopr 5 months ago

All tasks connected with time slices was executed.

Synchronization was started in [2021-05-20T14:46:25.838997+02:00[Europe/Prague]].
-------------------------
Synchronization is running as differential (entities will be updated only if least one attribute was changed).
-------------------------
Synchronization will use inner connector synchronization implementation.
-------------------------
Synchronization was correctly ended in [2021-05-20T15:28:20.933353+02:00[Europe/Prague]].
-------------------------
After success sync have to be run clear dirty state for contract slices. We start him (synchronously) now [2021-05-20T15:28:20.940337+02:00[Europe/Prague]].
-------------------------
Warning - select corrent contract slice is not executed correctly, Returned operation result is null. Ended in [2021-05-21T01:58:16.517695+02:00[Europe/Prague]].
-------------------------
After success sync have to be run HR task [HrEnableContractProcess]. We start him (synchronously) now [2021-05-21T01:58:16.790967+02:00[Europe/Prague]]. LRT ID: [aa825e47-ef6c-4b72-ba0e-9f675f284c08]
-------------------------
HR task [HrEnableContractProcess] ended in [2021-05-21T02:01:27.439763+02:00[Europe/Prague]].
-------------------------
After success sync have to be run HR task [HrEndContractProcess]. We start him (synchronously) now [2021-05-21T02:01:27.529425+02:00[Europe/Prague]]. LRT ID: [2ae6a92a-c86a-4dd5-ae10-a793f49fadfd]
-------------------------
HR task [HrEndContractProcess] ended in [2021-05-21T02:02:42.103070+02:00[Europe/Prague]].
-------------------------
After success sync have to be run HR task [HrContractExclusionProcess]. We start him (synchronously) now [2021-05-21T02:02:42.157825+02:00[Europe/Prague]]. LRT ID: [35e37235-3b89-4693-b392-edfd5989038a]
-------------------------
HR task [HrContractExclusionProcess] ended in [2021-05-21T02:02:42.212361+02:00[Europe/Prague]].
-------------------------
After success sync have to recount automatic roles (by attribute and tree structure). We start recount automatic roles by attribute (synchronously) now [2021-05-21T02:02:42.236533+02:00[Europe/Prague]].
-------------------------
Recalculation of automatic roles by attributes ended in [2021-05-21T03:12:01.421944+02:00[Europe/Prague]], role requests will be processed asynchronously.
-------------------------
We start recount automatic roles by tree structure (synchronously) now [2021-05-21T03:12:01.422175+02:00[Europe/Prague]].
-------------------------
Recalculation of automatic roles by tree structure ended in [2021-05-21T03:29:24.798241+02:00[Europe/Prague]], role requests will be processed asynchronously.

Executing ClearDirtyStateForContractSliceTaskExecutor takes 12:30 hours.

#7 Updated by Vít Švanda about 1 month ago

  • Priority changed from Normal to High
  • Target version set to 11.3.0

Also available in: Atom PDF

Go to top