Rolling Upgrade: Databases Remain Available

The rolling upgrade process upgrades a NuoDB domain, which might include multiple databases, to a new NuoDB database software release without shutting down the databases. Rolling upgrade ensure continuous database availability throughout the upgrade process.

During the upgrade, all the database processes running on a given host must be stopped. Therefore, a no-downtime upgrade is only possible if all databases in the domain have sufficient redundancy. The redundancy requirements for rolling upgrade are:

  • Every database must have more than one Storage Manager (SM) in every Storage Group, and these SMs must be running on different hosts.

  • Every database must have Transaction Engines (TEs) running on more than one host.

  • There must be Admin Processes (APs) running on more than one host.

    It is best practice to keep an AP in the durable domain configuration while its host is being upgraded. However, in a two-AP domain, if an AP does not restart, the AP quorum will be lost. Without an AP quorum, operations such as the addition of a new host are not allowed. In such cases, remove the AP from domain membership and regain the AP quorum.

Upgrade NuoDB Using Rolling Upgrade

During rolling upgrade, update the NuoDB database software on one host at a time to ensure that no two hosts in a domain go offline simultaneously.
The order in which the hosts are upgraded does not matter. However, it is more efficient to upgrade the host running the SM or TE with the highest start_id value first.

To upgrade the existing NuoDB database software using rolling upgrade, follow these steps for each host in the domain.

  1. Shut down all the NuoDB database processes running on the host.
    Shut down all the SMs and TEs for all databases running on the host using the nuocmd shutdown process command. The nuocmd shutdown process command shuts down the database processes "gracefully", allowing client connections to complete the transactions successfully. For more information, see the shutdown process command.

    To forcefully shut down a database process, use nuocmd shutdown process --kill or nuocmd shutdown process --evict on the specific process.

  2. Shut down the AP running on the host.
    For more information, see Starting and Stopping NuoDB Admin.

  3. Install the new NuoDB database software on the host.
    For more information, see Installing NuoDB on Linux or Installing NuoDB on Windows.

  4. Prepare the NuoDB configuration files.
    The new NuoDB software release may introduce new properties in the sample configuration files that are distributed with each release. To utilize these new features or changes, manually merge the files to update the existing configuration files.

    If NuoDB is installed using RPM, the old configuration files in $NUODB_CFGDIR are not replaced by the new files included in the new package. However, RPM might create a few new configuration files with a .rpmnew suffix.

    If NuoDB is installed using TAR, the $NUODB_CFGDIR for the new release might point to a different location.

    Depending on the installation method, identify the changes in configuration parameters. Merge and copy all the updated configuration files to the appropriate directories under the new $NUODB_CFGDIR. For more information, see Pre-Upgrade Best Practices.

    The default value of $NUODB_CFGDIR differs depending on whether NuoDB was installed using TAR or RPM. For more information, see Environment Variables.

  5. Retain the NuoDB domain state.
    The NuoDB domain tracks the current state information for both the AP and the databases managed by the AP. This domain state information is persisted in the $NUODB_VARDIR/raftlog file.

    If NuoDB is installed using TAR, $NUODB_VARDIR for the new database software may point to a different location. Copy the raftlog file to the new $NUODB_VARDIR directory.

    The default value of $NUODB_VARDIR differs depending on whether NuoDB was installed using TAR or RPM. For more information, see Environment Variables.

  6. Restart the AP.
    For more information, see Starting and Stopping NuoDB Admin.

  7. Restart all the database processes.
    Start all SMs and TEs for all databases running on the host, using nuocmd start process.

  8. Ensure that all the processes are RUNNING.
    Use nuocmd show domain to check the state of all the processes running in a domain.

  9. Verify the upgrade on the host.
    Run nuocmd show database-versions to verify the upgrade. For more information, see the show database-version command. For example:

    nuocmd show database-versions --db-name mydb
    effective version ID: 1376256, effective version: 4.1|4.1.1|4.1.2, max version ID: 1376256
Available versions:
Process versions:
  version ID: 1376256, version: 4.1|4.1.1|4.1.2, release: 4.1.2-6-a9cf2cd579
    [SM] host1:48006 [start_id = 0] [server_id = nuoadmin-0] [pid = 98234] [node_id = 1] [last_ack =  4.83] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 1] [server_id = nuoadmin-1] [pid = 45632] [node_id = 2] [last_ack =  8.80] MONITORED:RUNNING
    [TE] host3:48006 [start_id = 2] [server_id = nuoadmin-2] [pid = 10873] [node_id = 3] [last_ack =  3.22] MONITORED:RUNNING
  version ID: 1441792, version: 4.2, release: 4.2-1-c96c3d1a26
    [TE] host4:48006 [start_id = 4] [server_id = nuoadmin-3] [pid = 82572] [node_id = 5] [last_ack =  2.92] MONITORED:RUNNING

  10. Repeat step 1 to step 9 for each host in the domain.

Verify the Upgrade

After completing the upgrade, run nuocmd show database-versions to confirm all the database processes have been updated.

For example:

nuocmd show database-versions --db-name mydb
effective version ID: 1376256, effective version: 4.1|4.1.1|4.1.2, max version ID: 1376256
Available versions:
  version ID: 1441792, version: 4.2
Process versions:
  version ID: 1441792, version: 4.2, release: 4.2-1-c96c3d1a26
    [SM] host1:48006 [start_id = 5] [server_id = nuoadmin-0] [pid = 25439] [node_id = 6] [last_ack =  4.12] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 6] [server_id = nuoadmin-1] [pid = 65232] [node_id = 7] [last_ack =  3.76] MONITORED:RUNNING
    [TE] host3:48006 [start_id = 7] [server_id = nuoadmin-2] [pid = 86244] [node_id = 8] [last_ack =  5.43] MONITORED:RUNNING
    [TE] host4:48006 [start_id = 4] [server_id = nuoadmin-3] [pid = 82572] [node_id = 5] [last_ack =  2.11] MONITORED:RUNNING

The database version information is listed under Process versions. If a new database protocol version is available, this information is listed under Available versions. In that case, Upgrade the Database Protocol.