Project

General

Profile

Task #1291

Recalculate contract slices after synchronization of slices, before HR processes

Added by Alena Peterová 4 months ago. Updated about 20 hours ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Synchronization
Target version:
Start date:
10/03/2018
Due date:
12/28/2018
% Done:

100%


Description

Version 9.2.0-SNAPSHOT

After synchronization of contract slices ends, it should first recalculate the slices into contracts and after that start HR processes (otherwise HR processes run with old data). Please add option to run SelectCurrentContractSliceTaskExecutor after end of slice synchronization.

Real-world example:
One employee had contract to 30.9. The contract was prolonged to the next month. This information appeared in HR at last minute - on 30.9. So a new slice was created during the night 30.9.-1.10. The HR processes run after synchronization of slices, but before slices recalculation. So the user was disabled on 1.10., where he should be working. He was enabled again on 2.10., but that is too late.

provisioning_archive.png View (63.4 KB) Alena Peterová, 01/08/2019 04:07 PM

entity_event_IdmIdentityRole.png View (39.4 KB) Alena Peterová, 01/08/2019 04:11 PM

audit_IdmIdentityRole_Modify.png View (51.8 KB) Alena Peterová, 01/08/2019 04:13 PM

009.png View (52.1 KB) Ondřej Kopr, 01/09/2019 09:18 AM

008.png View (73.2 KB) Ondřej Kopr, 01/09/2019 09:18 AM

007.png View (79.1 KB) Ondřej Kopr, 01/09/2019 09:18 AM

006.png View (39.7 KB) Ondřej Kopr, 01/09/2019 09:18 AM

005.png View (69.4 KB) Ondřej Kopr, 01/09/2019 09:18 AM

004.png View (85.8 KB) Ondřej Kopr, 01/09/2019 09:18 AM

003.png View (52 KB) Ondřej Kopr, 01/09/2019 09:18 AM

002.png View (69.1 KB) Ondřej Kopr, 01/09/2019 09:18 AM

001.png View (78 KB) Ondřej Kopr, 01/09/2019 09:18 AM

clear_dirty_state_different_numbers.png View (104 KB) Alena Peterová, 01/17/2019 01:56 PM

History

#1 Updated by Vít Švanda 3 months ago

  • Assignee changed from Vít Švanda to Ondřej Kopr
  • Target version set to Onyx (9.3.0)

#2 Updated by Ondřej Kopr 2 months ago

  • Status changed from New to In Progress
  • Target version changed from Onyx (9.3.0) to Opal (9.4.0-rc.1)

This will be implemented as new option for contract slice synchronization, but implementing with test is little bit complicated so I moved it on next version.

#3 Updated by Ondřej Kopr 2 months ago

  • % Done changed from 0 to 20

After consult I decided that I'm not implement new checkbox for contract slice synchronization, because contract slice synchronization use same settings as contract synchronization (to much reimplementing and it will be non backward compatible).

So when is checked run HR process after correctly ended synchronization. The task select current contract will be started. @Alca are you OK with the solution?

#4 Updated by Alena Peterová 2 months ago

OK, this makes sense. Together with Peter we didn't find any real use case for starting only the slices recalculation, or only HR processes.
Just to be sure:
  • slices recalculation will be started only if it is synchronization of contract slices, not for synchronization of contracts?
  • if I have some project specific implementation (my own ContractSliceManager), it will be used for this recalculation started from the sync?

If this solution is simple, could you maybe put it into 9.2.3? :-) It would save us some time when scheduling tasks on our project.

#5 Updated by Ondřej Kopr 2 months ago

Alena Peterová wrote:

  • slices recalculation will be started only if it is synchronization of contract slices, not for synchronization of contracts?

Of course, task SelectCurrentContractExecutor is started only in contract slice synchronization.

Alena Peterová wrote:

  • if I have some project specific implementation (my own ContractSliceManager), it will be used for this recalculation started from the sync?

Your implementation of ContractSliceManager will be works as it is. Implementation is done only in ContractSliceSynchronizationExecutor (contract slice synchronization).

Thank you for answer. I will start with implementation.

#6 Updated by Ondřej Kopr 2 months ago

  • % Done changed from 20 to 80

I implement executing select current contract slice after synchronization of contract slices correctly ended.

commit: https://github.com/bcvsolutions/CzechIdMng/commit/2458f9e871bd9349e4247f8a76caf2906a86b062 (branch: okopr/1291-select-current-contract)

Branch will be merged after release version 9.3.0

Tests is included.

Documentation will be also updated after will be released new version of documentation.

#7 Updated by Ondřej Kopr 2 months ago

Branch was merged into develop and checked tests with current develop. Still missing documentation.

#8 Updated by Ondřej Kopr about 2 months ago

  • Status changed from In Progress to Needs feedback
  • Assignee changed from Ondřej Kopr to Vít Švanda
  • % Done changed from 80 to 90

#9 Updated by Alena Peterová about 1 month ago

!! Problem alert!!

I found out in the version 9.3.1 that the recalculation of slices is in fact done during the synchronization of slices after every slice. That could lead to removing manually assigned roles, deleting and recreating accounts. Consider this situation:
  • Slice 1 - slice is valid from 1.1.2018, the contract is valid from 1.1.2018 to infinity
  • HR changes on 3.12.2018:
    • slice 1 - valid till of the contract is changed to 30.11.2018
    • slice 2 is created, the slice is valid from 1.12.2018, the contract is valid from 1.12.2018 to infinity
  • Synchronization of slices will first update the slice 1 and so update the parent contract as invalid = valid till 30.11.2018. All roles and accounts are removed.
  • The same synchronization will create the slice 2 and so update the parent contract as valid. Only automatic roles are assigned again and their accounts are recreated.

Such changes are nothing strange in HRs - e.g. if the type of contract is changed, the HR may create a new slice in this way.

Synchronization of slices must skip recalculation of slices and do it only at the end (if the checkbox is checked).

Sorry about the misleading original description of this ticket. We thought the reason was missing recalculation, but the reason was probably deleting the original slice from HR which wasn't processed by synchronization.

#10 Updated by Alena Peterová about 1 month ago

Adding to the previous note:

I tried "workaround" - to disable contract-slice-save-recalculate-processor during synchronization. But then SelectCurrentContractSliceTaskExecutor run endlessly, probably due to invalid state of slices.
If fix of this problem (= recalculation of slices during synchronization) is complicated (or not available for 9.3.1), would you please try to think of some safe workaround for us?

#11 Updated by Vít Švanda about 1 month ago

  • Due date set to 12/28/2018
  • Status changed from Needs feedback to In Progress
  • Assignee changed from Vít Švanda to Ondřej Kopr
  • % Done changed from 90 to 50

Conclusion (goal of this ticket changed):

We need:

  • Skip slice recalculation during sync (not implemented now). IdmContractSliceService.SKIP_RECALCULATE_CONTRACT_SLICE should be help, but this skip was not designed directly for this purpose, so beware on conseqences :-) (SelectCurrentContractSliceTaskExecutor as Alca descripted).
  • Start recalculation of all slices (implemented in this task). Recalculation of silces is controlled by checbox for recalculation of HR processes (please extends the label for this information ... I know, same label will be see in contract sync and add big warning to changelog ... ).

We don't need this feature in version 9.4.0.rc.1, so please remove this from develop (sorry).

We need to release this feature as 9.3.2 and after that as 9.4.0.

#12 Updated by Vít Švanda about 1 month ago

  • Priority changed from Normal to High
  • Target version changed from Opal (9.4.0-rc.1) to Opal (9.4.0)

#13 Updated by Radek Tomiška about 1 month ago

  • Target version changed from Opal (9.4.0) to Onyx (9.3.2)

#15 Updated by Ondřej Kopr about 1 month ago

  • Target version deleted (Onyx (9.3.2))

The ticket will be solved in another hotfix (eq 9.3.3 or 9.4.0 final version).

Please move all changes in the ticket into some new branch.

#16 Updated by Alena Peterová about 1 month ago

Ondřej Kopr wrote:

The ticket will be solved in another hotfix (eq 9.3.3 or 9.4.0 final version).

Please not 9.4, we need this as hotifx for 9.3.x and we can't update to 9.4 yet.

#17 Updated by Ondřej Kopr about 1 month ago

  • Target version set to Onyx (9.3.3)

Alena Peterová wrote:

Ondřej Kopr wrote:

The ticket will be solved in another hotfix (eq 9.3.3 or 9.4.0 final version).

Please not 9.4, we need this as hotifx for 9.3.x and we can't update to 9.4 yet.

Not problem, I create next hotfix 9.3.3. Thank you.

#18 Updated by Ondřej Kopr about 1 month ago

  • Target version changed from Onyx (9.3.3) to Onyx (9.3.2)

#19 Updated by Ondřej Kopr about 1 month ago

  • Target version changed from Onyx (9.3.2) to Onyx (9.3.3)

#20 Updated by Ondřej Kopr 10 days ago

  • Status changed from In Progress to Needs feedback
  • Assignee changed from Ondřej Kopr to Alena Peterová

Today we tried simulate same problem on current develop:

Test scenario:
  • Synchronization 1:
  • creates one new slice[ID:1] with validFrom 1.1. 2019 and validTill: infity
  • after synchronization was added automatic roles by structure.
  • We manually added two roles to newly created contract
  • Synchronization 2:
  • updates slice[ID:1] validTill: 3.1. 2019
  • creates new slice slice[ID:2 ] with validFrom 1.1. 2019 and validTill: infinity
  • after synchronization user still has all roles. In audit exits only infomation about validity change.

The problem descibes in ticket description:

One employee had contract to 30.9. The contract was prolonged to the next month. This information appeared in HR at last minute - on 30.9. So a new slice was created during the night 30.9.-1.10. The HR processes run after synchronization of slices, but before slices recalculation. So the user was disabled on 1.10., where he should be working. He was enabled again on 2.10., but that is too late.

Can't be simulated, because now doesn't exists skip for slice recalculation in synchronization. So for every slices is done recalculation during synchronization immediate (in future we probably need some skip :)). Task SelectCurrentContractSliceTaskExecutor exits only for recalculate existing slices, that expire during time, not from synchronization.

Question is, if we really need some changes for hotfix 9.3.3? Please could you test your scenarios?

#21 Updated by Alena Peterová 10 days ago

OK, you're right that the roles aren't removed, only their validity is temporarily changed. But it's enough to delete and recreate the accounts :-(

I tried to debug it in my project and I think this happens:
  • IdmIdentityRole is updated - validTill is set to the past (see audit_IdmIdentityRole_Modify.png)
  • it causes IdmIdentityRole NOTIFY event, which runs identity-role-save-provisioning-processor (see entity_event_IdmIdentityRole.png)
  • the processor starts account management just when all roles are invalid:
    2019-01-08 16:41:24.224  INFO 20773 --- [task-executor-2] e.b.i.c.a.e.AbstractEntityEventProcessor : Processor [identity-role-save-provisioning-processor]([acc]) start for [CoreEvent [type
    : eu.bcvsolutions.idm.core.model.service.impl.DefaultEntityEventManager$$Lambda$111/42994890@144e55ff, content: eu.bcvsolutions.idm.core.api.dto.IdmIdentityRoleDto [id= a9df673d-8f87-4abd-
    8efe-4510a259e916], properties: {idm:parent-event-id=null, idm:super-owner-id=46ed6b7e-efc1-4e88-abcb-4dbfdeb2156a, idm:parent-event-type=UPDATE, idm:root-event-id=null, idm:event-id=3fd8a
    d24-afcb-4ddf-abfd-db9b67ebd311, idm:priority=NORMAL, skipCheckAuthorities=true, idm:execute-date=null}]] with order [1000].
    ......
    2019-01-08 16:41:24.734  INFO 20773 --- [task-executor-2] e.b.i.c.m.s.i.DefaultEntityEventManager  : Publishing event [IdentityAccountEvent [type: DELETE, content: eu.bcvsolutions.idm.acc.dto.AccIdentityAccountDto [id= c46f5e87-0b09-43e7-9070-7a7300ef911c], properties: {deleteTargetAccount=true, forceDeleteOfIdentityAccount=false}]]
    
  • ACM deletes the AccIdentityAccount, which deletes AccAccount, which causes provisioning Delete (see provisioning_archive.png - 3 accounts are deleted and later recreated, other 3 fall to account protection because the systems have it enabled).

#22 Updated by Ondřej Kopr 10 days ago

Thank you for your test scenario, I started test it again and there is screenshots from my tests:

  • STEP 1: I synchronized contract slice [id:1], after synchronization user recive automatic roles by treeStructure and I added one manually role:

  • STEP 2: In end system I change validity for contract slice [id: 1] to past and then I create second one slice that is valid [id: 2]
  • STEP 3: I start with new synchronization, during synchronization is actualized contract slice [id:1] and the slice and contract is now invalid (also roles), but account was deleted with also provisioning delete:

After synchronized the second slice [id:2] still exists roles with new validity and account with new provisioning operation was created:

So the main problem isn't with removing roles, but with remove account during this scenario. Do you agree?

#23 Updated by Alena Peterová 10 days ago

  • Assignee changed from Alena Peterová to Ondřej Kopr

Exactly, thank you for detailed screenshots.

#24 Updated by Ondřej Kopr 10 days ago

  • Status changed from Needs feedback to In Progress

Alena Peterová wrote:

Exactly, thank you for detailed screenshots.

I also thank you for confirmation, Now I try the skip recalculation for contract slices during synchronization.

#25 Updated by Ondřej Kopr 9 days ago

  • Status changed from In Progress to Needs feedback
  • Assignee changed from Ondřej Kopr to Vít Švanda
  • % Done changed from 50 to 90

Done, I implement new skip for contract slice recalculation (it is called set_dirty_state). Dirty state is calculated by new LRT ClearDirtyStateForContractSliceTaskExecutor. Task check all dirty states in entity states and then start recalculation for each slice.

Please Vitek could you make a review?

commit: https://github.com/bcvsolutions/CzechIdMng/commit/4e6872f94905a4efc43905381e4861801262b0bd
documentation: https://wiki.czechidm.com/devel/documentation/identities/dev/contractual-relationship-slice#slices_recalculation

Thank you for check. If you want I have prepared test data with test end system.

#26 Updated by Alena Peterová 1 day ago

I tested this in C... project and it works fine, thanks.

Only strange thing is that the ClearDirtyStateForContractSliceTaskExecutor has different count of processed items, please see the screenshot.

The total amount is correct - it corresponds to the number of synchronized slices. But the count of processed items is sometimes lower.

EDIT: There must be something wrong with paging, because next run of ClearDirtyStateForContractSliceTaskExecutor processed the remaining 100 dirty flags and didn't process my 3 new dirty flags.

#27 Updated by Vít Švanda 1 day ago

  • Status changed from Needs feedback to Resolved
  • Assignee changed from Vít Švanda to Ondřej Kopr
  • % Done changed from 90 to 100

I did review and tested sync of slices.
It works well, so thanks for nice new recalculation with dirty flags. I think this is good way how generally would be recalculation realized.

- I found problem when current using slice was changed (contract validity) and validity From of slice was changed too. On end of my analysis, the problem is not in the bulk recalculation, but in recalculation of slice (one by one).
I fixed this in the commit:
https://github.com/bcvsolutions/CzechIdMng/commit/79e4692cc10c99676ac6b1aa4df8a98ed8b8498f

#28 Updated by Ondřej Kopr 1 day ago

Alena Peterová wrote:

Only strange thing is that the ClearDirtyStateForContractSliceTaskExecutor has different count of processed items, please see the screenshot.

Thanks for check with more slices. I fix the pageable by removing it :D the main problem is with iterator and removing item from the result set.

Commit: https://github.com/bcvsolutions/CzechIdMng/commit/e53f4c6226e932320aeaa79c558bb4f0a8e4cd5a

#29 Updated by Ondřej Kopr about 20 hours ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF