Project

General

Profile

Actions

Task #1357

open

Create Export system Bulk-Action

Added by Marek Klement over 5 years ago. Updated about 5 years ago.

Status:
In Progress
Priority:
Normal
Assignee:
Marek Klement
Category:
Bulk operations
Target version:
-
Start date:
11/01/2018
Due date:
% Done:

60%

Estimated time:
Owner:

Description

- Create a bulk action which will export all entities from the selected system.

Technical design

  • Bulk action will extends from AbstractBulkAction<SysSystemDto,SysSystemFilter>
  • It will get system id/ids and from that particular system/systems it will export all entities
  • Exported data will be saved into CSV file
  • User will provide directory, where CSV files will be stored
  • Name of CSV will consist of system name and time stamp - this gives us the opportunity to export more systems and also to recognize current export by timestamp
  • Next get the configuration of the connector
  • Connect to the system and get all entities
  • Write entities to file chosen and creating header while doing so
  • Header cannot be created before all items are processed - some items have more properties than others in connector object

Related issues

Related to IdStory Identity Manager - Task #1230: Virtual system - make first snapshot of dataNewMarek Klement08/29/2018

Actions
Related to IdStory Identity Manager - Task #1286: Import entities to systemClosedVít Švanda10/03/2018

Actions
Related to IdStory Identity Manager - Defect #1540: Displaying bulk actions for systems require ROLE_READ and SYSTEM_DELETEClosedRadek Tomiška03/06/2019

Actions
Actions #1

Updated by Marek Klement over 5 years ago

  • Related to Task #1230: Virtual system - make first snapshot of data added
Actions #2

Updated by Marek Klement over 5 years ago

  • Related to Task #1286: Import entities to system added
Actions #3

Updated by Marek Klement over 5 years ago

  • Status changed from New to In Progress

- created class ExportSystemBulkAction in acc module
- Trying to get directory path from frontend

Actions #4

Updated by Marek Klement over 5 years ago

Actions #5

Updated by Marek Klement over 5 years ago

- now I am able to get the string with directory and dto of system
- will continue with the logic of export -

Actions #6

Updated by Marek Klement over 5 years ago

  • % Done changed from 0 to 30

First export from the system achieved!

File:
  • First of all bulk action validate a given directory
  • if the path does not exist, it will create it
  • then because of multiselection of systems, it will also create a folder with a name of the system
  • in that folder will file with name: <system-name><date><time-millis-of-day>.csv (SomeDifferentVirt2018-11-15-52111169.csv)
Header
  • header is created according to the schema of the system
  • it will take all attributes of schema and creates a header
Objects in System
  • then I handle objects in the system
  • iterate header and if there is value, it will put it on a new line, if not then it will put an empty string inside
  • multivalued attributes are separated by ","
Todo now
  • create ResultCode messages for some exceptions
  • connect these messages with localization
  • create test
  • create documentation
Actions #7

Updated by Marek Klement over 5 years ago

My questions:
  • if I get null from connector object for one particular item in the header, do I fill it ass empty string or null. The difference is in quotation chars - nothing against ""
  • I have a method for filling multivalued - it puts multivalued separator before every item except first - the question is: What if I get null for the last item of these values? it will put ',' in the end - is it even possible?
Actions #8

Updated by Marek Klement about 5 years ago

  • % Done changed from 30 to 60

- All messages in exceptions and names are in locates
- Exceptions are solved by ResultCode
- Test - creates a system, import accounts and export it - deletes all created files after
- Documentation - [[https://wiki.czechidm.com/tutorial/adm/systems_-_export_of_accounts_to_csv|here]]

- Also fixed counter problem in ImportFromCSVToSystemExecutor

Actions #9

Updated by Marek Klement about 5 years ago

- All messages in exceptions and names are in locates
- Exceptions are solved by ResultCode
- Test - creates a system, import accounts and export it - deletes all created files after
- Documentation - [doc](https://wiki.czechidm.com/tutorial/adm/systems_-_export_of_accounts_to_csv)

- Also fixed counter problem in ImportFromCSVToSystemExecutor

Actions #10

Updated by Marek Klement about 5 years ago

  • Status changed from In Progress to Needs feedback
  • Assignee changed from Marek Klement to Alena Peterová
Actions #11

Updated by Alena Peterová about 5 years ago

  • Status changed from Needs feedback to In Progress
  • Assignee changed from Alena Peterová to Marek Klement
I tested the functionality and made code and documentation review, it's nice. A few things to improve:
  • make DEFAULT_MULTIVALUED_SEPARATOR configurable (as "Separator of multivalued attributes"). The separator "," is not universal, e.g. when exporting AD, the values of groups contain "cn=groupname,ou=organization,dc=company,dc=tld". We would like to use "\n" (newline) for AD.
  • (optional) make DEFAULT_LINE_SEPARATOR configurable (as "Attributes separator"). Although ";" is quite universal
  • when the value contains " (double quote), it should be escaped by another double quote (escapechar in CSVWriter). Now it isn't escaped, which could cause some problems when parsing CSV.
  • Description - change to "Export all accounts from system"
  • file.getParentFile().mkdirs() - check the result. If false, return the ResultCodeException that the file cannot be created (same as when creating new file)
  • Permissions - export shouldn't require SYSTEM_UPDATE, SYSTEM_READ should be enough. Please write the required permission to the wiki. (Note: when testing, there is a problem in permissions of bulk actions #1540)
Specific to virtuals:
  • if there are some unresolved requests, the export has inconsistent data:
    • unresolved created accounts aren't exported -> correct
    • unresolved updates of accounts are exported already with the changes, which were not implemented yet -> problem, administrators would expect the real state on the system, which doesn't have the changes yet.
    • unresolved deletes cause NullPointerException. These accounts should be exported, because they were not deleted yet
    • after cancelling a delete request, the account is still exported -> correct
  • when a new virtual system is created with a default role, "Behavior of the default role for inactive identities" must be set in the specific settings of the synchronization. Otherwise the synchronization doesn't start. Please set it to DO_NOT_LINK.
Actions #12

Updated by Alena Peterová about 5 years ago

  • Related to Defect #1540: Displaying bulk actions for systems require ROLE_READ and SYSTEM_DELETE added
Actions

Also available in: Atom PDF