Rotating Key Pair Certificates

When using TLS encryption, it is necessary to rotate key pair certificates before the certificates expire. To avoid downtime, NuoDB Admin Processes (APs) with the new certificates should be introduced in a rolling fashion. This section provides supporting information for performing key rotation for the two trust models (Shared Admin Key and Unique Admin Key) described in Enabling TLS Encryption.

About Key Rotation for the Shared Admin Key Trust Model

For the Shared Admin Key trust model, the same key pair certificate is used by all APs, and that certificate is trusted by all processes. This means that both the keystore and the truststore have to be updated for all APs, and by extension, all database processes. As stated, processes with the new certificates should be introduced in a rolling fashion to avoid downtime, which means that processes with the old key pair certificate can verify processes with the new key pair certificate. Since the new key pair certificates must be verified using a new trusted certificate, existing processes (admin and database) must have the new trusted certificate propagated to their truststores before the new key pair certificates can be introduced.

An overview of how key rotation is performed is as follows:

  1. A new shared admin new key pair certificate (and associated PEM file) is generated.

  2. The new certificate is added to the truststore of every AP and every database process.

  3. The new certificate is added to the truststore of every SQL client and NuoDB Command client.

  4. For each AP, the keystore is replaced. (Make sure that the new key pair certificate is in effect.)

  5. For each database process, the process is shut down, and a new process is started with the same options on the same host, so that its certificate chain is based on the new certificate.

  6. (Optional) The old certificate is removed from the truststore of every AP and every database process.

  7. (Optional) The old certificate is removed from the truststore of every SQL client and NuoDB Command client.

When a TLS-encrypted NuoDB domain is running, NuoDB Command’s show domain command returns the following:

nuocmd show domain
server version: 6.0-1-fc6a857de9, server license: Enterprise
server time: 2023-06-05T22:02:21.075, client token: 640dbf3d5e65f59761ba497bf5118dcc738d2540
Servers:
  [server0] server0:48005 (LEADER, Leader=server0) ACTIVE:Connected *
  [server1] server1:48005 (FOLLOWER, Leader=server0) ACTIVE:Connected
Databases:
  test [RUNNING]
    [SM] server0:48007 (Default) [start_id = 0] [server_id = server0] [pid = 283] [node_id = 1] MONITORED:RUNNING
    [SM] server1:48006 (Default) [start_id = 1] [server_id = server1] [pid = 94] [node_id = 3] MONITORED:RUNNING
    [TE] server0:48006 (Default) [start_id = 2] [server_id = server0] [pid = 288] [node_id = 2] MONITORED:RUNNING
    [TE] server1:48007 (Default) [start_id = 3] [server_id = server1] [pid = 98] [node_id = 4] MONITORED:RUNNING

To get certificate data, use NuoDB Command’s get certificate-info command

nuocmd --show-json get certificate-info
{
 "processCertificates": {
  "0": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=server0"
  },
  "1": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=server1"
  },
  "2": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=server0"
  },
  "3": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=server1"
  }
 },
"processTrusted": {
  "0": [
    "nuocmd",
    "nuoadmin"
   ],
   "1": [
     "nuocmd",
     "nuoadmin"
   ],
   "2": [
     "nuocmd",
     "nuoadmin"
   ],
   "3": [
     "nuocmd",
     "nuoadmin"
   ]
},
"serverCertificates": {
  "server0": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=*.domain"
   },
  "server1": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=*.domain"
   }
},
 "serverTrusted": {
   "server0": [
     "nuocmd",
     "nuoadmin"
   ],
   "server1": [
     "nuocmd",
     "nuoadmin"
   ]
},
 "trustedCertificates": {
   "nuoadmin": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 1591301400000,
     "issuerName": "CN=*.domain",
     "subjectName": "CN=*.domain"
  },
  "server1": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=*.domain"
  }
},
 "serverTrusted": {
   "server0": [
    "nuocmd",
    "nuoadmin"
   ],
   "server1": [
    "nuocmd",
    "nuoadmin"
  ]
},
"trustedCertificates": {
    "nuoadmin": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 1591301400000,
    "issuerName": "CN=*.domain",
    "subjectName": "CN=*.domain"
  },
  "nuocmd": {
    "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
    "expires": 4706647403000,
    "issuerName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US",
    "subjectName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US"
   }
  }
}

The trustedCertificates field above contains all trusted certificates for all processes in the domain as a mapping of alias to certificate. These certificate aliases are referenced in the serverTrusted and processTrusted fields. serverTrusted is a mapping of NuoDB Admin’s server ID property to a set of trusted certificate aliases, and processTrusted is a mapping of database process start ID to a set of trusted certificate aliases.

In the example above, there are two trustedCertificates listed. nuoadmin is the certificate associated with the (Shared Admin Key) trust model and nuocmd is the certificate used by NuoDB Command clients.

The nuoadmin certificate has issuerName and subjectName CN=*.domain, as does the certificate for both APs (server0) and (server1); they are the same certificate (the certificatePem field contains the complete PEM-encoded certificate and has been abbreviated in the output above for the purpose of readability.)

Rotating Keys for the Shared Admin Key Trust Model

1. Generate a new shared admin new key pair certificate and associated PEM file.

Generate new key pair certificate as follows:

nuocmd create keypair --keystore /tmp/nuoadmin.p12 --store-password "$PASSWD" --ca \
  --sub-altnames dns:server0,dns:server1 --dname "CN=*.domain,SERIALNUMBER=12345"

Create a PEM file for the certificate as follows:

nuocmd show certificate --keystore /tmp/nuoadmin.p12 --store-password "$PASSWD" --cert-only > /tmp/nuoadmin.cert

Ensure that the keystore has the same password (as specified by keystore-password property) in nuoadmin.conf so that the APs can reload the keystore without having to be reconfigured and restarted. In the example above, we specify the SERIALNUMBER field so that the new certificate has a different distinguished name from the current certificate. We also specify Subject Alternative Names (SAN) for the certificate so that clients can perform hostname verification when communicating with either AP.

2. Add the new certificate to the set of certificates trusted by SQL clients and NuoDB Command clients. This document assumes that the SQL client is NuoDB SQL, which uses the same truststore format as the NuoDB Command (PEM file).

cat /tmp/nuoadmin.cert >> /etc/nuodb/keys/nuoadmin.cert

This simply appends the newly-generated certificate to the existing certificate.

3. Add the new certificate to the truststore of all admin and database processes:

nuocmd add trusted-certificate --alias nuoadmin_prime --cert /tmp/nuoadmin.cert --timeout 10

This adds the new certificate under the alias nuoadmin_prime. It is not possible to replace the nuoadmin certificate because all processes in the domain (admin and database) are verified with it. This single command invocation causes the trusted certificate to be propagated to all APs and all database processes. The --timeout 10 argument specifies the amount of time to wait for the new certificate to be propagated to the truststore of every process in the domain.

/tmp/nuoadmin.cert is specified which contains the new certificate only, and not /etc/nuodb/keys/nuoadmin.cert, which contains both the old and the new certificates generated in step 1 (nuoadmin and nuoadmin_prime).

To verify that the new certificate is in the truststore of every AP, use NuoDB Command’s get certificate-info command:

nuocmd --show-json get certificate-info
{
 "trustedCertificates": {
   "nuoadmin": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 1591301400000,
     "issuerName": "CN=*.domain",
     "subjectName": "CN=*.domain"
   },
   "nuoadmin_prime": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 1591321638000,
     "issuerName": "CN=*.domain, SERIALNUMBER=12345",
     "subjectName": "CN=*.domain, SERIALNUMBER=12345"
   },
   "nuocmd": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 4706647403000,
     "issuerName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US",
     "subjectName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US"
   }
 }
 ...
 "processTrusted": {
   "0": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ],
   "1": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ],
   "2": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ],
     "3": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ]
 },
"serverTrusted": {
   "server0": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ],
   "server1": [
     "nuoadmin_prime",
     "nuocmd",
     "nuoadmin"
   ]
 },
 ...
}

4. For each AP, replace the keystore with the keystore containing the certificate.

Back up the existing keystore:

cp /etc/nuodb/keys/nuoadmin.p12 /etc/nuodb/keys/nuoadmin.p12.bak

Change owner to nuodb and move keystore to location specified in nuoadmin.conf with the keystore property

chown nuodb:nuodb /tmp/nuoadmin.p12
mv /tmp/nuoadmin.p12 /etc/nuodb/keys/nuoadmin.p12

The AP reloads the keystore periodically, so it is not necessary to restart the AP. The period is specified with the nuoadmin.conf property keystoreUpdateIntervalMs (which defaults to 5000 (milliseconds)).

In the first step, the keystore was created with the same password as used for the existing keystore (specified using the keystore-password property in nuoadmin.conf). If the password for the new keystore is different, then you must update nuoadmin.conf with the new password and restart the AP. Avoid such a scenario by using the same password.

To verify that the new certificates have been loaded by all APs, use NuoDB Command’s get certificate-info command.

nuocmd --show-json get certificate-info
{
 ...
 "serverCertificates": {
   "server0": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 1591321638000,
     "issuerName": "CN=*.domain, SERIALNUMBER=12345",
     "subjectName": "CN=*.domain, SERIALNUMBER=12345"
   },
   "server1": {
     "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
     "expires": 1591321638000,
     "issuerName": "CN=*.domain, SERIALNUMBER=12345",
     "subjectName": "CN=*.domain, SERIALNUMBER=12345"
   }
 },
 ...
}

5. For each database process, shutdown the process and start an identical process in its place.

Restart TEs can be restarted as follows:

Shut down the TE on server1 and wait for it to exit

nuocmd shutdown process --start-id 3
nuocmd check process --start-id 3 --check-exited --timeout 10

Restart the TE on server1 and wait for it to become RUNNING

nuocmd start process --db-name test --engine-type TE --server-id server1
nuocmd check database --db-name test --check-running --num-processes 4 --timeout 10

Restart the SM to restart the non-running archive

nuocmd start database --incremental

Shut down the SM and wait for it to exit

nuocmd shutdown process --start-id 1
nuocmd check process --start-id 1 --check-exited --timeout 10

Start an SM on the non-running archive and wait for it to become RUNNING

nuocmd start database --db-name test --incremental
nuocmd check database --db-name test --check-running --num-processes 4 --timeout 10

Verify that all database processes are using certificates issued by the new admin certificate

nuocmd --show-json get certificate-info
{
  ...
  "processCertificates": {
    "4": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 1591321638000,
      "issuerName": "CN=*.domain, SERIALNUMBER=12345",
      "subjectName": "CN=server1"
    },
    "5": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 1591321638000,
      "issuerName": "CN=*.domain, SERIALNUMBER=12345",
      "subjectName": "CN=server0"
    },
    "6": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 1591321638000,
      "issuerName": "CN=*.domain, SERIALNUMBER=12345",
      "subjectName": "CN=server1"
    },
    "7": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 1591321638000,
      "issuerName": "CN=*.domain, SERIALNUMBER=12345",
      "subjectName": "CN=server0"
    }
  }
  ...
}
At this point, key rotation is complete. Steps 5 and 6 below are optional, clean-up steps which remove the old certificate from the truststore for all processes (APs and database processes) and for all clients.

6. Remove the old certificate from the truststore of every AP and every database process.

nuocmd remove trusted-certificate --alias nuoadmin

NuoDB Command’s remove trusted-certificate command is a single command invocation that propagates changes to all the truststore of every admin and database process.

You may then use the get certificate-info command to verify that the certificate has been removed from the truststore for all admin and database processes.

nuocmd --show-json get certificate-info
{
  ...
  "processTrusted": {
    "4": [
      "nuoadmin_prime",
      "nuocmd"
    ],
    "5": [
      "nuoadmin_prime",
      "nuocmd"
    ],
    "6": [
      "nuoadmin_prime",
      "nuocmd"
    ],
    "7": [
      "nuoadmin_prime",
      "nuocmd"
    ]
  },
  "serverTrusted": {
    "server0": [
      "nuoadmin_prime",
      "nuocmd"
    ],
    "server1": [
      "nuoadmin_prime",
      "nuocmd"
    ]
  },
  "trustedCertificates":
   {"nuoadmin_prime": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 1591321638000,
      "issuerName": "CN=*.domain, SERIALNUMBER=12345",
      "subjectName": "CN=*.domain, SERIALNUMBER=12345"
    },
    "nuocmd": {
      "certificatePem": "-----BEGIN CERTIFICATE-----...-----END CERTIFICATE-----",
      "expires": 4706647403000,
      "issuerName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US",
      "subjectName": "CN=nuocmd, OU=Eng, O=NuoDB, L=Boston, ST=MA, C=US"
    }
 }
 ...
}

7. Remove the old certificate from the truststore of every SQL client and NuoDB Command client. This is done by replacing the PEM file used by nuosql and nuocmd with the PEM file generated in step 1 (the PEM file that only contains the nuoadmin_prime certificate).

Back up the existing PEM file

cp /etc/nuodb/keys/nuoadmin.cert /etc/nuodb/keys/nuoadmin.cert.bak

Change the owner to nuodb and move the PEM file containing only the new certificate to a location specified by NUOCMD_VERIFY_SERVER

chown nuodb:nuodb /tmp/nuoadmin.cert
mv /tmp/nuoadmin.cert /etc/nuodb/keys/nuoadmin.cert

Key Rotation for the Unique Admin Key Trust Model

Where the CA certificate is expired, the key rotation process for the Unique Admin Key trust model is the same as that for the Shared Admin Key trust model, except that the trusted certificate is the CA certificate rather than the certificate associated with the Shared Admin Key.

Where there is no requirement to rotate the CA certificate, the key rotation process for the Unique Admin Key trust model is much simpler than that for the Shared Admin Key trust model. The only steps required are step 3 ( replacing the keystore for all APs) and step 4 (restarting all database processes).

Recovering From Expired Certificates

To avoid potential downtime, the key rotation process described should be performed well in advance of certificates expiring. If key rotation is not performed before either the trusted certificate or admin certificates expire, then the key rotation process has be modified to account for the inability of NuoDB Command clients to verify the admin certificates.

In the Unique Admin Key trust model, trusted certificates and admin certificates are one and the same, that is, there is a single admin certificate, which is trusted.

If certificates expire while database processes are running, then these will shut themselves down in accordance with the value set for certificate-poll-interval-ms. The certificate-poll-interval-ms option controls the interval at which database processes check the validity of their own certificates. The default value is 10000 (milliseconds). Users can disable this behavior by setting certificate-poll-interval-ms to 0, but they should still expect downtime or degraded service when a domain is running with expired certificates because NuoDB Command clients and SQL clients will not be able to establish new connections to the processes affected.

APs do not automatically shut themselves down when certificates expire because they are capable of loading new certificates as described above.

Shared Admin Key

If working with the Shared Admin Key trust model, recovering a domain with an expired certificate is similar to the key rotation process, except that steps 2 and 3 must be modified to account for the inability of NuoDB Command clients to verify the admin certificate. This can be done by disabling server verification for NuoDB Command invocations until the new certificate is installed and loaded (step 3) using the --no-verify argument.

To add the new certificate to the truststore for all processes without performing server verification, use the following command:

nuocmd --no-verify add trusted-certificate --alias nuoadmin_prime --cert /tmp/nuoadmin.cert --timeout 10

To check that the keystore containing the new admin key pair certificate has been loaded, use the following command:

nuocmd --no-verify --show-json get certificate-info

Unique Admin Key

If working with the Unique Admin Key trust model, recovering a domain with expired admin certificates is similar to the key rotation process, except that there will be downtime from the point of view of NuoDB Command clients and SQL clients communicating with the APs that have expired certificates.

If the CA certificate has expired, the recovery process is identical to that for the Shared Admin Key trust model except that the CA certificate is the certificate that has to be propagated to all truststores and each AP is provisioned with a different key pair certificate signed by the CA key.