Task #1357
open
Create Export system Bulk-Action
Added by Marek Klement over 5 years ago.
Updated about 5 years ago.
Description
- Create a bulk action which will export all entities from the selected system.
Technical design
- Bulk action will extends from AbstractBulkAction<SysSystemDto,SysSystemFilter>
- It will get system id/ids and from that particular system/systems it will export all entities
- Exported data will be saved into CSV file
- User will provide directory, where CSV files will be stored
- Name of CSV will consist of system name and time stamp - this gives us the opportunity to export more systems and also to recognize current export by timestamp
- Next get the configuration of the connector
- Connect to the system and get all entities
- Write entities to file chosen and creating header while doing so
- Header cannot be created before all items are processed - some items have more properties than others in connector object
- Related to Task #1230: Virtual system - make first snapshot of data added
- Related to Task #1286: Import entities to system added
- Status changed from New to In Progress
- created class ExportSystemBulkAction in acc module
- Trying to get directory path from frontend
- now I am able to get the string with directory and dto of system
- will continue with the logic of export -
- % Done changed from 0 to 30
First export from the system achieved!
File:
- First of all bulk action validate a given directory
- if the path does not exist, it will create it
- then because of multiselection of systems, it will also create a folder with a name of the system
- in that folder will file with name: <system-name><date><time-millis-of-day>.csv (SomeDifferentVirt2018-11-15-52111169.csv)
Header
- header is created according to the schema of the system
- it will take all attributes of schema and creates a header
Objects in System
- then I handle objects in the system
- iterate header and if there is value, it will put it on a new line, if not then it will put an empty string inside
- multivalued attributes are separated by ","
Todo now
- create ResultCode messages for some exceptions
- connect these messages with localization
- create test
- create documentation
My questions:
- if I get null from connector object for one particular item in the header, do I fill it ass empty string or null. The difference is in quotation chars - nothing against ""
- I have a method for filling multivalued - it puts multivalued separator before every item except first - the question is: What if I get null for the last item of these values? it will put ',' in the end - is it even possible?
- % Done changed from 30 to 60
- Status changed from In Progress to Needs feedback
- Assignee changed from Marek Klement to Alena Peterová
- Status changed from Needs feedback to In Progress
- Assignee changed from Alena Peterová to Marek Klement
I tested the functionality and made code and documentation review, it's nice. A few things to improve:
- make DEFAULT_MULTIVALUED_SEPARATOR configurable (as "Separator of multivalued attributes"). The separator "," is not universal, e.g. when exporting AD, the values of groups contain "cn=groupname,ou=organization,dc=company,dc=tld". We would like to use "\n" (newline) for AD.
- (optional) make DEFAULT_LINE_SEPARATOR configurable (as "Attributes separator"). Although ";" is quite universal
- when the value contains " (double quote), it should be escaped by another double quote (escapechar in CSVWriter). Now it isn't escaped, which could cause some problems when parsing CSV.
- Description - change to "Export all accounts from system"
- file.getParentFile().mkdirs() - check the result. If false, return the ResultCodeException that the file cannot be created (same as when creating new file)
- Permissions - export shouldn't require SYSTEM_UPDATE, SYSTEM_READ should be enough. Please write the required permission to the wiki. (Note: when testing, there is a problem in permissions of bulk actions #1540)
Specific to virtuals:
- if there are some unresolved requests, the export has inconsistent data:
- unresolved created accounts aren't exported -> correct
- unresolved updates of accounts are exported already with the changes, which were not implemented yet -> problem, administrators would expect the real state on the system, which doesn't have the changes yet.
- unresolved deletes cause NullPointerException. These accounts should be exported, because they were not deleted yet
- after cancelling a delete request, the account is still exported -> correct
- when a new virtual system is created with a default role, "Behavior of the default role for inactive identities" must be set in the specific settings of the synchronization. Otherwise the synchronization doesn't start. Please set it to DO_NOT_LINK.
- Related to Defect #1540: Displaying bulk actions for systems require ROLE_READ and SYSTEM_DELETE added
Also available in: Atom
PDF
