Transparent Data Encryption Error Codes

The nuocmd update data-encryption subcommand causes all Admin Processes and all database processes for the specified database to be synchronously updated with the new storage password. Admin Processes require the storage password even if they are not managing any running database processes, so that they can supply it to database processes that are subsequently started.

The nuocmd update data-encryption subcommand issues a REST API request to an Admin Process, which performs the following actions:

  1. Verify the --current-password using the storage password verifier stored in the domain state managed by the Admin Processes, or check that there is no storage password verifier if --is-initial-password was specified.

  2. Update the storage password verifier in the domain state managed by the Admin Processes.

  3. Replicate the new storage password (--new-password), the current storage password (--current-password), and any existing storage passwords (--existing-passwords) to all Admin Processes in the domain.

  4. Wait for each database process in the specified database (--db-name) to be reconfigured by its connected Admin Process.

The nuocmd update data-encryption subcommand generates no output if the command is successful. If the command fails, an error message will indicate why it failed. The following subsections describe possible errors and how to address them.

Unable to verify password

The following error occurs when an incorrect --current-password is specified, or --is-initial-password is specified for a database that has TDE enabled.

$ nuocmd update data-encryption --db-name db \
    --current-password wrongPassword> \
    --new-password newPassword>
'update data-encryption' failed: Unable to update storage password for database db: Unable to verify password for dbName=db
Solution

Specify the correct current storage password.

No leader to commit command

The following error occurs when the correct current storage password is specified, but there is no Admin majority needed to update the storage password verifier stored in the domain state.

$ nuocmd update data-encryption --db-name db \
    --current-password currentPassword \
    --new-password newPassword
'update data-encryption' failed: Unable to update storage password for database db: No leader to commit command: KeyValueCAS{key=config/storagePasswordVerifier/db}
Solution

Make sure that all Admin Processes are running and able to communicate with each other. The nuocmd check servers subcommand can be used with the --check-leader argument to make sure that an Admin leader has been elected. Once an Admin leader has been elected, the command can be re-issued:

$ nuocmd check servers --check-leader --check-connected --timeout 30
$ nuocmd update data-encryption --db-name db \
    --current-password currentPassword \
    --new-password newPassword

Unable to get command response

The following error has the same underlying cause as the previous error (no Admin majority) but has a different symptom. This is because the Admin Process servicing the request was the most recent leader, but was unable to commit the command to a majority of Admin Processes.

$ nuocmd update data-encryption --db-name db \
    --current-password currentPassword \
    --new-password newPassword
'update data-encryption' failed: Unable to update storage password for database db: Unable to get command response
Solution

The solution is the same as the one for the previous error, but it is also possible that the previously issued command to update the storage password verifier is eventually committed. In this case, there will be a verification error when the nuocmd update data-encryption command is re-issued, because the storage password has been updated. To confirm that the storage password has been updated, the nuocmd check data-encryption subcommand can be used:

$ nuocmd check data-encryption --db-name db \
    --password newPassword

This command verifies the supplied storage password and makes sure that it is fully replicated.

nuocmd check data-encryption is actually shorthand for nuocmd update data-encryption with --new-password and --current-password set to the same value (--password).

Unable to propagate storage password …​ to all admin servers

The following error occurs when the storage password cannot be replicated to some Admin Process in the domain. This can be due to a connectivity issue or because some Admin Process is not running at all.

$ nuocmd update data-encryption --db-name db \
    --new-password secret --is-initial-password
'update data-encryption' failed: Unable to propagate storage password for database db to all admin servers:
  nuoadmin2: failed to connect to nuoadmin2:48005
Solution

Make sure that all Admin Processes are running and able to communicate with each other. The nuocmd check servers subcommand can be used with the --check-connected argument to make sure that all Admin Processes are communicating with each other. Once all Admin Processes are running and communicating with each other, the nuocmd check data-encryption command can be used to make sure that the storage password is replicated to all Admin Processes:

nuocmd check servers --check-leader --check-connected --timeout 30
nuocmd check data-encryption --db-name db ---password secret

Unable to propagate storage password …​ to all database processes

The following error occurs when some database processes for the specified database cannot be reconfigured with the storage password.

$ nuocmd update data-encryption --db-name db \
    --current-password secret --new-password secret
'update data-encryption' failed: Unable to propagate storage password for database db to all admin servers:
  nuoadmin0: org.apache.thrift.transport.TTransportException: java.net.SocketException: Socket closed
Unable to propagate storage password for database db to all database processes:
  0: Timed out updating process configuration
The Unable to propagate storage password …​ to all admin servers message is also generated. This is because the unresponsive database process blocked the Admin Process from acknowledging the storage password replication message.
Solution

Make sure that all database processes are responsive. The nuocmd check database subcommand can be used with the --check-liveness argument specified to make sure that all database processes have responded within the specified time interval. This situation is more likely to occur in containerized deployment environments such as Kubernetes, where Admin and database processes are running in different Pods, possibly on different hosts. The ping-timeout database option and the processLivenessCheckSec Admin property can be used to cause such processes to be evicted.

No valid storage password provided

The following error does not occur when issuing the nuocmd update data-encryption subcommand, but when starting the database. If the entire database, or several SMs in the database, do not start up when invoking nuocmd start database, then exited database processes should be inspected using the nuocmd show database subcommand. If some database archives are encrypted but the Admin layer is not configured with the storage password for that database, then the following message will appear:

$ nuocmd show database --db-name db
Database(default_options={}, default_region_id=0, incarnation=(3, 1), name=db, server_assignments={}, state=NOT_RUNNING)
  db [NOT_RUNNING]
  ...
    [SM] <UNKNOWN ADDRESS> [start_id = 4] [server_id = server0] [pid = ] [node_id = ] EXITED(CONFIGURED:UNKNOWN):The archive was encrypted: the storage password must be provided.: (2020-01-24T22:20:33.448+0000) Engine connection to Admin Process closed (5)

This can happen for a number of reasons. Since the storage password itself is only stored in memory (only the storage password verifier is stored on disk), the user is expected to supply it whenever there is a full restart of the Admin layer. It is also possible that an archive with encrypted data is created using the --restored flag of nuocmd create archive, but the user does not supply the storage password using nuocmd update data-encryption.

If some archives are encrypted with a different storage password from the current storage password configured in the Admin layer, the following message will appear:

$ nuocmd show database --db-name db
Database(default_options={}, default_region_id=0, incarnation=(5, 1), name=db, server_assignments={}, state=NOT_RUNNING)
  db [NOT_RUNNING]
  ...
    [SM] <UNKNOWN ADDRESS> [start_id = 6] [server_id = server0] [pid = ] [node_id = ] EXITED(CONFIGURED:UNKNOWN):No valid storage password provided: (2020-01-24T23:37:23.938+0000) Engine connection to Admin Process closed (5)

This is possible if an archive is not running while multiple nuocmd update data-encryption invocations are made without specifying --existing-passwords, causing the storage password needed to decrypt that archive to not be retained (this situation is described above).

Solution

If TDE has never been explicitly configured in the current domain for a database that has encrypted archives that have been created using nuocmd create archive …​ --restored, then the user must specify the storage password that the archive was encrypted with using nuocmd update data-encryption.

If TDE has been explicitly configured in the current domain, but the Admin layer has been restarted, the nuocmd check data-encryption can be used to ensure that all Admin Processes have the latest storage password.

If older versions of the storage password are needed to decrypt certain archives, then the following command can be invoked to ensure that all Admin Processes have the latest storage password and any other required storage passwords:

$ nuocmd update data-encryption --db-name <DB-name>
   --current-password <latest-password>
   --new-password <latest-password>
   --passwords <old-password-1> <...> <old-password-n>

Once all storage passwords needed to decrypt all archives for a database have been supplied to the Admin layer, it should possible to start the database.