Task #1162
closed
Delete virtual systems
100%
Description
Version 8.1.2
I need to delete cca 20 virtual systems.
The standard way to delete systems was slow in the customer's environment (3 days were not enough for one system).
- cca 14 000 "Entities in system managed in IdM"
- 0 - 2 implementers (identities, not by roles)
- no "Accounts in system managed in IdM" (I deleted them by synchronization Linked -> Remove link)
- no open requests (I canceled them, or they were implemented without confirmation).
- Required confirmation by the implementer: No
- Attributes (multi):
firstName
lastName
email
titleAfter
titleBefore
phone - schema, provisioning mapping - please see the screenshots
- When opening the detail of some entity on the system, then "Attributes on the target system" contained only NAME, ENABLE and one another attribute (phone or username, I don't remember). All the other attributes were missing.
- When I checked "required confirmation" and then resaved an identity with this virtual, a new request was created. The request contained duplicated attributes firstName, lastName, email - but only one of the line was filled with the real value. After implementing the request and resaving the identity again, the same request was created again. I know the values of these attributes were not in vs_account_form_value.
- In Form definitions of the VsAccount of the system, the attributes firstName, lastName, email, titleAfter, titleBefore were duplicated. E.g. "email" was there once with order 2 and once with order 0.
I wasn't able to create so "broken" virtuals in my local environment, so I don't know how to simulate it :-( These virtuals were created by admins, I don't know how (but surely not by standard Adding of virtual, because that works correctly).
Files
Updated by Vít Švanda almost 6 years ago
- Status changed from New to In Progress
- Target version set to Lapis (8.2.0)
Updated by Vít Švanda almost 6 years ago
- % Done changed from 0 to 40
- I created VS system with 6000 accounts.
- I discovered problem with delete. Problems mainly occures on delete a VsRequest where is in the VsRequestDto returned list of implementers (are shows in the request table and on the request detail). Time for search implementers are 1 sec for every request (on my ntb).
Updated by Vít Švanda almost 6 years ago
- % Done changed from 40 to 80
I successfuly simulated problems described above. I think all problems was caused with renaming the attributes. I suppose that somebody renamed attributes "firstName", "lastName", "email" to "firstName ", "lastName ", "email " or with another white character.
Use case:
- We have new virtual system (generated) with one account "vs_one".
- When we show detail of this account, we can see all attributes (firstName too).
- Change the connector configuration "attributes". Rename the "firstName" -> "firstName ". Click on "Test connector" (save the changes and push configuration to the VS connector).
- When we show detail of this account, we cannot se firstName attribute now. This is correct because old attribute "firstName" is not already in the connector schema and new attribute "firstName " not exists because was never provisioned.
- Go to the identity "vs_one" and resave it.
- Request with change of "firstName" attribute was created (again and again). That is "correct". Problem is in the IdM schema of virtual system. After rename the attributes in the threeth step, nobody clicked on the generate the schema from the connector. It means firstName is still mapped on the attribute wich does not exists in the connector schema. So during provisioning the connector allways returns "nothing" for "firstName" attribute (that is cause for request creating).
- Generate the IdM schema on virtual system. This creates new schema attribute with key "firstName ". Next we have to change mapping on this new attribute.
- Easier way is rename currently exists schema attribute "firstName" to "firstName ".
Updated by Vít Švanda almost 6 years ago
- I optimalized searching the implementers. Now is returned max 10 implementers over REST and 50 in the email notification.
Delete VS system with 6000 VsAccounts and 6000 VsRequests takes cca 2 minuts (on my NTB) now.
- I tried trun off system-delete processor in the VS module and delete SysSystem without data in the VS module. This is no good way, because VsRequests has hard reference to SysSystem.
Updated by Vít Švanda almost 6 years ago
- Status changed from In Progress to Needs feedback
- Assignee changed from Vít Švanda to Radek Tomiška
Updated by Ondřej Kopr almost 6 years ago
- Assignee changed from Radek Tomiška to Ondřej Kopr
Updated by Ondřej Kopr almost 6 years ago
- Status changed from Needs feedback to Resolved
- Assignee changed from Ondřej Kopr to Vít Švanda
- % Done changed from 80 to 100
AAAAaa I finally resolve this hell ticket! In first attempt I tested this in develop, so I tested behavior without and with your changes.
After I rebuild branch (vsvanda/1162-Delete-virtual-systems) and recover backup for DB I started with this review. I try load test with 20k identities and request to virtual system (prepare test data for this case was harder thing on review :)). I also resolve merge conflicts with develop.
Test system: (our virtual: http://172.31.255.96:8080/idm/)
Thanks for help with prepare test enviroment. Remove virtual system with ~20k archvied request and system entities takes about.
During delete virtual system was deleted about 70 000 entities (mapping, eavs, provisioning archive, system entities, vs request - all these entities generates audit logs). Processor from VS () starts 15:00:59 and ACC processors end 15:33:00 (in this mode was loged DEBUG for all loggers - hibernate and etc.) commit that contains all audit logs was ended in 22:08 (server logs debug messages - this probably slows down the process with create audit logs)
Remove virtual system by processor in VS was speeded up. Thanks for that.