IdStory Identity Manager

To correct product behavior, we simply need to generate new (and unique) IV for each encryption performed. Effectively: (re)encrypt data saved into confidential storage if changed.
Because IV does not need to be secret, we can store it along the ciphertext in the same database table. That way we will know which IV to use for which ciphertext during decryption.

• Saving new data into confidential storage
  1. Generate new random IV.
  2. Encrypt data using AES-CBC, key, IV.
  3. Store ciphertext and IV into the database.
• Updating data in the confidential storage
  1. Generate new random IV.
  2. Encrypt updated data using AES-CBC, key, IV.
  3. Update ciphertext and IV in the database, overwriting old values.
• Removing data from confidential storage.
  • Without change (no enc/dec operations are performed in this step).

Secret key will be stored in the app config (separated from the database) the same way as we do now.

Source for clarification: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38a.pdf (Appendix C)
To quote:

The CBC, CFB, and OFB modes require an initialization vector as input, in addition to the
plaintext. An IV must be generated for each execution of the encryption operation, and the same
IV is necessary for the corresponding execution of the decryption operation. Therefore, the IV, or
information that is sufficient to calculate the IV, must be available to each party to the
communication.

The IV need not be secret, so the IV, or information sufficient to determine the IV, may be
transmitted with the ciphertext.

For the CBC and CFB modes, the IVs must be unpredictable. In particular, for any given
plaintext, it must not be possible to predict the IV that will be associated to the plaintext in
advance of the generation of the IV. 

There are two recommended methods for generating unpredictable IVs. The first method is to
apply the forward cipher function, under the same key that is used for the encryption of the
plaintext, to a nonce. The nonce must be a data block that is unique to each execution of the
encryption operation. For example, the nonce may be a counter, as described in Appendix B, or
a message number. The second method is to generate a random data block using a FIPS-approved random number generator.

The SecureRandom Java class (https://docs.oracle.com/javase/8/docs/api/java/security/SecureRandom.html) conforms with FIPS 140-2 and can be used for our purposes (generating IVs).
But there is an important note:
Depending on the implementation, the generateSeed and nextBytes methods may block as entropy is being gathered, for example, if they need to read from /dev/random on various Unix-like operating systems.

We already have that covered by starting Tomcat with java.security.egd=/dev/./urandom because urandom is unblocking even when there is not enough entropy gathered. This construct also makes Java use SHA1PRNG (stronger) instead of NativePRNG.
(More info: https://docs.oracle.com/en/java/javase/11/docs/specs/security/standard-names.html#securerandom-number-generation-algorithms)

@kopro You wrote the original crypt service. Could you please look at this? I just need a quick estimate of what would it mean to change IdM behavior to conform with #2355#note-1. Tyvm. :)

• vector will be different for every item - must be saved somewhere - new column in confidential storage?,
• check if this changes will be compatible with another implementation for confidential storage - vault (maybe it is compatible but for sure check it),
• add random generation for vector,
• the whole process with initialization and reinitialization cipher with new vector will be slower, check how big is difference with old behavior,
• change algorithm for encrypt and decrypt,
• new behavior isnt backward compatible, some LRT must be create for the first reinitialization with dynamic vector.

Implementation will change DefaultIdmConfidentialStorage and DefaultCryptService. For crypt service the change will not be compatible backward compatible (api will be changed).

Implementation+test+documentation+checks is about ~4MD

Implementation spec

We need to correct the hardcoded confidential storage's cipher IV in a way that a random (and unique) IV is generated every time we store something in the confidential storage.
This means changing DefaultIdmConfidentialStorage and DefaultCryptService, the change should be internal to a product and probably will not propagate "outside" to modules and scripts (but analysis needed there!).

• Saving new data into confidential storage
  1. Generate new random IV.
  2. Encrypt data using AES-CBC, key, IV.
  3. Store ciphertext and IV into the database.
• Updating data in the confidential storage
  1. Generate new random IV.
  2. Encrypt updated data using AES-CBC, key, IV.
  3. Update ciphertext and IV in the database, overwriting old values.
• Removing data from confidential storage.
  • Without change (no enc/dec operations are performed in this step).
• Reading data from confidential storage.
  1. IdM reads both ciphertext and corresponding IV from database.
  2. IdM initializes cipher in decrypt mode using AES-CBC, key, IV.
  3. IdM decrypts the ciphertext.

• IV is always 16B long (because AES in CBC mode).
• IV must be generated by CSRNG, this means using java.security.SecureRandom.nextBytes(...). For each encryption, new IV must be generated.
• IV must be generated server-side. Client must not have any way to influence the IV.
• IV can (and will) be stored in the same database table as encrypted data.
• This implementation change must be backwards compatible on deployments.
  • We will create a flyway script that adds current IV into the confidential storage. We need to write the flyway anyway to add the db table column. (Other option to this is to have the original IV still in the Java source and use it only if there is not IV stored in the database - but this is ugly.)
  • Maybe something else will be needed, we will consult ad-hoc.
• The cipher's secret key is stored as part of app configuration as it already is; no change there.
• Already used AES-CBC-PKCS5Padding cipher stays; no change there.

Estimate for implementation is about 4MD according to Ondra #2355#note-5.
See other notes in this ticket / ask if something needs clarification.