Rolling Upgrade: Databases Remain Available

The rolling upgrade process moves a NuoDB domain, which might include multiple databases, to a new NuoDB release without shutting down the databases. Rolling upgrades ensure continuous database availability throughout the upgrade process.

All processes on a given host must be stopped to upgrade the software. Therefore, a no-downtime upgrade is only possible if all databases in the domain have sufficient redundancy:

Every database must have more than one Storage Manager (SM) in every Storage Group, and these SMs must be on different hosts.
Every database must have Transaction Engines (TEs) on more than one host.

There must be more than one Admin Process (AP) on different hosts.

It is best practice to keep an AP in the durable domain configuration while its host is being upgraded. However in a two-AP domain, in the unusual event that an AP does not restart, AP quorum is lost. Without AP quorum, operations such as the addition of a new host are not allowed. See Admin Process (AP) Quorum.

In this situation, remove the AP from domain membership and regain AP quorum.

Steps for Rolling Upgrade

During rolling upgrade, the NuoDB release software is updated on one host in the domain at a time. This ensures that all the hosts in a database never go offline simultaneously.

To upgrade NuoDB using rolling upgrade, follow these steps on each host.

Shut down the NuoDB database processes.
Shut down all NuoDB SMs and TEs for all databases running on the host machine using the nuocmd shutdown process command. For more information, see the shutdown process command.

The nuocmd shutdown process command shuts down the database processes "gracefully" which allows client connections to successfully complete their transactions. If a process refuses to shutdown, use nuocmd shutdown process --kill or nuocmd shutdown process --evict on the specific process.

Shut down the AP.
For more information, see Starting and Stopping NuoDB Admin.
Install the new NuoDB software release.
Install the new NuoDB software release on each host in the domain. For more information, see Installing NuoDB on Linux or Installing NuoDB on Windows.
Prepare the NuoDB configuration files.
The new NuoDB release might introduce new properties in the sample configuration files that are distributed with each release. To take advantage of the new features or changes to features, the files need to be manually merged so that the new or changed configuration parameters are moved into the existing configuration files.

If NuoDB is installed using the RPM package manager utility, the old configuration files in $NUODB_CFGDIR won’t be replaced with those included in the new package. Some of the new configuration files might be created with a .rpmnew suffix by the RPM package manager.

If NuoDB is installed using a TAR package, the $NUODB_CFGDIR for the new release might point to a different location. Merge and copy all configuration files to the correct directories under the new $NUODB_CFGDIR directory. For more information, see Pre-Upgrade Best Practices.

The default value of $NUODB_CFGDIR is different for NuoDB installation using a TAR package and an RPM package manager. For more information, see Environment Variables.
Retain NuoDB domain state.
The NuoDB domain keeps track of current state information for both NuoDB Admin and its databases under management. This state information is persisted in the $NUODB_VARDIR/raftlog file.

If NuoDB is installed using a TAR package, the $NUODB_VARDIR for the new release might point to a different location. Copy the raftlog file to the new $NUODB_VARDIR directory.

The default value of $NUODB_VARDIR is different for NuoDB installation using a TAR package and an RPM package manager. For more information, see Environment Variables.
Restart the NuoDB admin services.
Start the AP. For more information, see Starting and Stopping NuoDB Admin.
Start the NuoDB database processes (SMs and TEs).
Start all NuoDB SMs and TEs for all databases running on the host using nuocmd start process. For more information, see the start process command.

Repeat these steps for each host in the domain. It does not matter in which order you choose to upgrade the hosts. However, we recommend to restart the engines in order of decreasing start_id.

Verifying the Rolling Upgrade

During the rolling upgrade process, you may monitor version information using nuocmd show database-versions. For more information, see the show database-version command. Here is an example after one TE has been updated:

nuocmd show database-versions --db-name mydb

effective version ID: 1376256, effective version: 4.1|4.1.1|4.1.2, max version ID: 1376256
Available versions:
Process versions:
  version ID: 1376256, version: 4.1|4.1.1|4.1.2, release: 4.1.2-6-a9cf2cd579
    [SM] host1:48006 [start_id = 0] [server_id = nuoadmin-0] [pid = 98234] [node_id = 1] [last_ack =  4.83] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 1] [server_id = nuoadmin-1] [pid = 45632] [node_id = 2] [last_ack =  8.80] MONITORED:RUNNING
    [TE] host3:48006 [start_id = 2] [server_id = nuoadmin-2] [pid = 10873] [node_id = 3] [last_ack =  3.22] MONITORED:RUNNING
  version ID: 1441792, version: 4.2, release: 4.2-1-c96c3d1a26
    [TE] host4:48006 [start_id = 4] [server_id = nuoadmin-3] [pid = 82572] [node_id = 5] [last_ack =  2.92] MONITORED:RUNNING

At the end of rolling upgrade, run nuocmd show database-versions to check whether all database processes have been updated. It may also show that a database protocol upgrade is required. For more information, see Upgrade the Database Protocol.

nuocmd show database-versions --db-name mydb

effective version ID: 1376256, effective version: 4.1|4.1.1|4.1.2, max version ID: 1376256
Available versions:
  version ID: 1441792, version: 4.2
Process versions:
  version ID: 1441792, version: 4.2, release: 4.2-1-c96c3d1a26
    [SM] host1:48006 [start_id = 5] [server_id = nuoadmin-0] [pid = 25439] [node_id = 6] [last_ack =  4.12] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 6] [server_id = nuoadmin-1] [pid = 65232] [node_id = 7] [last_ack =  3.76] MONITORED:RUNNING
    [TE] host3:48006 [start_id = 7] [server_id = nuoadmin-2] [pid = 86244] [node_id = 8] [last_ack =  5.43] MONITORED:RUNNING
    [TE] host4:48006 [start_id = 4] [server_id = nuoadmin-3] [pid = 82572] [node_id = 5] [last_ack =  2.11] MONITORED:RUNNING