Restoring Data From Backup Sets

A simple hot copy creates an identical copy of the archive copied, which can be used directly to start a new SM - see here for details.

In all other cases, hot copy backs up into backup sets. A backup set is the directory created when initiating a full hot copy. For more background on backup sets, see Using Backup Sets.

  • To use the data hot copied into backup sets, it must be restored into a new database archive using nuoarchive.

Important Considerations

Be aware of the following:

  • nuoarchive restore is an offline operation.

  • Do not restore into any archive of a running database.

    • Neither the source backup set nor the target archive(s) can be in use by a running database.

  • nuoarchive restore creates a new archive and you can then use this archive:

    1. To start a new SM for the same database the hot copy came from.

    2. To restart an SM on an existing host that has been offline for a long time (to reduce archive synchronization time).

    3. To create a new database (a copy of the original).

  • In the event of lost or corrupted data due to an application or human error:

    • Create a brand new database by restoring to a point-in-time just before the error happened.

    • Then use SQL to copy the lost or corrupted data from the new, restored database back to the original database.

  • Ensure the correct ownership of the restored archive directory and its contents.

    • root user access or sudo elevated permissions access are required to perform this task:

      • Bare metal/VM: Run chown nuodb:nuodb <dir>

      • In a Container: Permissions should be set for you, but if necessary run chown nuodb:root <dir> (note the group is root not nuodb).

      • See also the restore Helm chart which can automate this process under Kubernetes.

  • The journal(s) will be found in a subdirectory of the restored archive. You may wish to move them to a separate directory (typically to their own volume).

    • If so, ensure the same ownership as above for the journal directory and its contents.

    • Use --journal-dir to specify its location when running nuocmd create archive.

  • If using storage groups you will probably need to restore from multiple backup sets in order to restore all storage groups. NuoDB does not track this and you need to know which backup sets, from which SMs, contain which storage groups.

    • One solution is to incorporate the storage group names in the backup set names, for example: testdb-2022-09-21-archive-0-sg1-sg2.

For more information on backup sets and journals, see Using Backup Sets.

Performing a Restore

In the descriptions below, "using a backup" means using one or more backup sets to restore an archive (depending on whether you are using storage groups or not).

The basic steps are:

  1. Restore from a backup to create a new archive, repeat for multiple hosts (multiple SMs) as desired.

  2. On each host:

    1. Optionally copy the journal to its own volume.

    2. Ensure correct ownership of the archive and journal.

    3. Initialize each archive using nuocmd create archive …​ --restored

  3. Start the SM process(es):

    • Explicitly using nuocmd start process (existing database) on each host.

    • Or by using nuocmd create database (new database).

See Worked Examples - Single Backup Set below for examples of performing step 1 (the restore).

Restoration Scenarios

Why might you restore from a backup?

Creating a New Database

  1. Use a backup to create a copy of the original database. Restore the archive on the first SM host, then copy it to all the other new SM hosts.

  2. As (1) but restore from backup to a specific point-in-time or to a specific backup-element.

New SM for an Existing Database

To reduce archive SYNC time, restore the archive from latest backup and use this to initialize the SM. The SM need only SYNC changes since the backup was taken.

Restoring a Failed SM

If an SM has failed due to a disk or other hardware error. Once fixed, use the latest backup to restore its archive (essentially the same scenario as the previous case).

Restoring a Database to a Known State

Stop the existing database, restore from the backup and use it to replace the archive and journal on every SM host, then restart the database.

Restoring a Database to a Specific Known State

This can only be done by taking the database offline and replacing its archives and journals. However, if you are careful, it can be done without deleting the database and recreating it. This is typically not a production requirement, but is often needed during development and testing. Possibile scenarios:

  1. Doing a globally consistent restore using locally available backup sets.

  2. Doing a local restore on one host and relying on that host syncing other newly started SMs.

    • All SMs but the first will start with an empty archive and journal, and so must SYNC from the first.

  3. Doing a local restore on one host and copying the archive to seed other SMs (to avoid syncing overhead).

    • Restore on one host to the exact state required and starting the SM on it.

    • Then do a local restore on the other hosts to seed them and allow them to sync with the first SM.

    • Since they all have the exact same archive and journal, syncing will be minimal.

Steps required for a local restore:

  1. Restore an archive to a temporary directory using any of the techniques described in this section.

  2. Shutdown the database and replace all its archives and journals with the restored archive and its journal.

    • How to do this is described under simple hot copy - note the warning in this section.

  3. Restart the database.

As this is most commonly required for development and/or testing, we suggest the following approach for creating the necessary backup:

  1. Create your database and populate it with the desired data.

  2. To take the backup:

    • Stop all clients so no changes to the database can occur.

    • Perform a simple hot copy on any host, multiple hosts if using storage groups.

  3. Retain the backup(s) until needed.

Worked Examples - Single Backup Set

In these examples, three separate disks are assumed:

  • /volumes/archives For database archive storage.

  • /volumes/backups For backup storage.

  • /volumes/journals For database journal storage.

Restoring a Single SM

Suppose your SM has been offline during a planned (or unplanned) outage. Using a backup taken since the SM went offline may allow it to become usable more quickly. The trade-off is whether syncing the existing archive will take longer than restoring the backup.

If the archive and/or journal have be lost, perhaps due to disk failure, synching a new, empty, archive will be required. This places a load on the other SMs and can be slow. Restoring the archive first from the most recent backup is usually quicker.

The simplest restore assumes you take periodic incremental and/or journal backups:

# On SM host, remove out-of-date archive
rm -rf /volumes/archives/test

# Replace from the backup
nuoarchive restore --restore-dir /volumes/archives/test /volumes/backups/BackupSets/2022-04-18

# By default the journal is a subdirectory of the archive.
# Otherwise, you need to replace the old journal too.
rm -rf /volumes/journal
cp -R /volumes/archives/test/journal /volumes/journal/test

chown -R nuodb:nuodb /volumes/archives/test
chown -R nuodb:nuodb /volumes/journal/test

# Restart the SM
nuocmd start process --db-name test --type SM --archive-id 0 --this-server

  • In this example we are restoring the SM for archive with id=0 of database test.

    • If you aren’t sure of the archive id, check using nuocmd show archives.

  • The restore will restore the archive to the state as at the most recent incremental or journal backup in the backup set.

Restoring to Specific Backup Elements

When using incremental (or journal) backups, you can restore to a specific element using element IDs.

In this example assume there are two hosts, host1 and host2, corresponding to the Admin Processes (servers) with ids nuoadmin-1 and nuoadmin-2 repectively. The domain looks like this:

nuocmd show domain
server version: ..., server license: Enterprise
server time: ..., client token: ...
Servers:
  [nuoadmin-0] host1:48005 [last_ack = 7.57] ACTIVE (LEADER, Leader=nuoadmin-0, log=199574/154/154) Connected *
  [nuoadmin-1] host2:48005 [last_ack = 7.57] ACTIVE (FOLLOWER, Leader=nuoadmin-0, log=199574/154/154) Connected
Databases:
  test [state = RUNNING]
    [SM] host1:48006 [start_id = 10] [server_id = nuoadmin-0] [pid = 640] [node_id = 1] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 12] [server_id = nuoadmin-0] [pid = 357] [node_id = 3] [last_ack =  8.08] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 11] [server_id = nuoadmin-1] [pid = 207] [node_id = 2] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 13] [server_id = nuoadmin-1] [pid = 291] [node_id = 4] [last_ack =  9.17] MONITORED:RUNNING

To restore to a specific backup element, do the following:

  1. Identify a backup set to be restored. In this example each SM is backing up to a local /volumes/backups directory on its host. Copying locally is usually faster, but this is not a restriction.

  2. Identify the directory where the backup set is located. In this example, each host is using the backup set /volumes/backups/BackupSets/2022-04-18. We can use either one, the procedure is the same.

  3. Using NuoDB Archive's --report-backups command, generate a summary of your chosen backup set. For example:

    nuoarchive restore --report-backups /volumes/backups/BackupSets/2022-04-18
    <BackupSet id="16df36f3-afb1-bf45-5e87-a6b6d3e44fdf" database="test" databaseId="efa117e7-8b5c-5449-14ab-80f77e81334b" collectionId="fb7f28df-
405f-488e-b7e4-7307e373be7f" archiveId="9e1a097b-e310-2f4a-38ab-f04f4eaeb1c8">
 <BackupElements>
   <BackupElement id="63bdb6de-5382-459a-8397-c786f16817d4" type="incremental" startDate="2022-04-18 14:20:45" endDate="2022-04-18 14:20:46"/>
   <BackupElement id="4d7338fe-f3c9-4041-aaa1-77ba730807f5" type="journal" startDate="2022-04-18 14:20:37" endDate="2022-04-18 14:20:39"/>
   <BackupElement id="9be268f4-4b11-4913-b7af-cc27bd970d0e" type="incremental" startDate="2022-04-18 14:20:30" endDate="2022-04-18 14:20:31"/>
   <BackupElement id="fb7f28df-405f-488e-b7e4-7307e373be7f" type="full" startDate="2022-04-18 14:20:24" endDate="2022-04-18 14:20:24"/
 </BackupElements>
</BackupSet>

    • Note that the backups are listed most recent first.

  4. Restore from the backup set using the desired backup element ID . The following example demonstrates restoring of the backup set to the first incremental backup element - the element with id 9be268f4-4b11-4913-b7af-cc27bd970d0e.

    nuoarchive restore --backup-element-id 9be268f4-4b11-4913-b7af-cc27bd970d0e --restore-dir /volumes/archives/test2 /volumes/backups/BackupSets/2022-04-18 /tmp/sm2-backupset.xml

  5. Ensure correct ownership:

    sudo chown -R nuodb:nuodb /volumes/archives/test2

  6. Optionally, move the journal to its own volume. If you do this:

    • Check to make sure ownership is still correct.

    • Use --journal-dir when invoking nuocmd create archive.

  7. Start a new database, with a new SM serving the restored archive. For example, start a database named test2 using the restored archives from step 5 running on a new host host3 (server id nuoadmin2):

    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-2
    nuocmd create database --db-name test2 --dba-user ... --dba-password ... --te-server-ids nuoadmin-2
    After database test2 is running, the DBA user and password will be whatever they were in the original database we restored from.

  8. Now the domain looks like this:

    nuocmd show domain
    server version: ..., server license: Enterprise
server time: ..., client token: ...
Servers:
  [nuoadmin-0] host1:48005 [last_ack = 4.16] ACTIVE (LEADER, Leader=nuoadmin-0, log=199574/163/163) Connected *
  [nuoadmin-1] host2:48005 [last_ack = 4.16] ACTIVE (FOLLOWER, Leader=nuoadmin-0, log=199574/163/163) Connected
Databases:
  test [state = RUNNING]
    [SM] host1:48006 [start_id = 10] [server_id = nuoadmin-0] [pid = 640] [node_id = 1] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 12] [server_id = nuoadmin-0] [pid = 357] [node_id = 3] [last_ack =  8.08] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 11] [server_id = nuoadmin-1] [pid = 207] [node_id = 2] [last_ack =  9.84] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 13] [server_id = nuoadmin-1] [pid = 291] [node_id = 4] [last_ack =  9.17] MONITORED:RUNNING
  test2 [state = RUNNING]
    [SM] host1:48006 [start_id = 0] [server_id = nuoadmin-0] [pid = 827] [node_id = 1] [last_ack =  9.92] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 1] [server_id = nuoadmin-0] [pid = 831] [node_id = 2] [last_ack =  9.21] MONITORED:RUNNING

Restoring to Specific Transactions

When using journal backups, you can restore to a specific transaction using timestamps, a point-in-time restore. In this example, assume we are running on a host with server id nuoadmin-1.

  1. Identify the backup set folder. This was the folder specified when invoking the hot copy command.

    • In this example: /volumes/backups/BackupSets/2022-04-18

  2. Using NuoDB Archive (nuoarchive), find available transactions. For example:

    nuoarchive restore --report-timestamps --format simple --start-time "2022-04-18T19:00" --end-time "2022-04-18T19:59" /volumes/backups/BackupSets/2022-04-18
    2022-04-18T19:12:37  3970
2022-04-18T19:12:45  4738

    In the output above, timestamps are to the left and transaction IDs to the right.

  3. Restore a new copy from the backup set by using its ID. In this example we are restoring to the 2022-04-18T19:12:37 timestamp using ID 3970.

    nuoarchive restore --restore-snapshot 3970 --restore-dir /volumes/archives/test2
    /volumes/backups/BackupSets/2022-04-18

  4. Ensure correct ownership

    sudo chown -R nuodb:nuodb /volumes/archives/test2

  5. Optionally, move the journal to its own volume. If you do this:

    • Check to make sure ownership is still correct.

    • Use --journal-dir when invoking nuocmd create archive.

  6. Start a new SM from the restored directory:

    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-1

nuocmd create database --db-name test2 --dba-user ... --dba-password ... --te-server-ids nuoadmin-1
    After database test2 is running, the DBA user and password will be whatever they were in the original database we restored from at that particular point in time.

Worked Examples - Multiple Backup Sets

This is required if you are using storage groups distributed across multiple SMs so that different storage groups are backed up into different backup-sets. See here for an example.

We use the term backup collection to refer to the multiple backup sets we need to restore from.

In these examples, three separate disks are assumed:

  • /volumes/archives For database archive storage.

  • /volumes/backups For backup storage.

  • /volumes/journals For database journal storage.

Restoring An Archive with Storage Groups

When restoring the database from multiple backup sets to a globally consistent state, XML-based summaries are used as "stand-ins" for backup sets located on remote hosts. Global information available in these XML files enables NuoDB Archive to determine the latest common backup element to be used for the restore.

In this example each SM is backing up to a local /volumes/backups directory on its host. Copying locally is usually faster, but this is not a restriction.

  1. Identify the directory where each backup set in the backup collection is located. In this example, the backup collection contains two backup directories, one on each host at /volumes/backups/BackupSets/2022-04-18.

  2. Using NuoDB Archive's --report-backups command, generate summaries of each backup set. Each summary should be stored in a file. For example:

    On host1:

    nuoarchive restore --report-backups /volumes/backups/BackupSets/2022-04-18 > /tmp/sm1-backupset.xml

    On host2:

    nuoarchive restore --report-backups /volumes/backups/BackupSets/2022-04-18 > /tmp/sm2-backupset.xml

  3. Copy summaries to all hosts involved in the restore so that each summary is available on each host. For example, use scp to copy the summary generated on host1 to host2, and vice versa.

    scp host1:/tmp/sm1-backupset.xml host2:/tmp/
    scp host2:/tmp/sm2-backupset.xml host1:/tmp/

  4. Restore a consistent copy of the archive from each backup set by combining the summaries with each backup set. In the following example, we will restore the backup set on each host to produce a consistent copy of the database consisting of two restored archives on two hosts (host1:/volumes/archives/test2 and host2:/volumes/archives/test2).

    On host1:

    nuoarchive restore --restore-dir /volumes/archives/test2 /volumes/backups/BackupSets/2022-04-18 /tmp/sm2-backupset.xml

    On host2:

    nuoarchive restore --restore-dir /volumes/archives/test2 /volumes/backups/BackupSets/2022-04-18 /tmp/sm1-backupset.xml

  5. Ensure correct ownership. On each machine run:

    sudo chown -R nuodb:nuodb /volumes/archives/test2

  6. Optionally, move the journal to its own volume. If you do this:

    • Check to make sure ownership is still correct.

    • Use --journal-dir when invoking nuocmd create archive.

  7. Start a new database, with SMs serving the restored archives. For example, to start a database named test2 with two SMs serving the restored archives from step 5:

    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-1
    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-2
    nuocmd create database --db-name test2 --dba-user ... --dba-password ... --te-server-ids nuoadmin-1 nuoadmin-1

Restoring to Specific Backup Elements

When using incremental (or journal) backups, you can restore to a specific element using element IDs.

In this example assume there are two hosts, host1 and host2, corresponding to the Admin Processes (servers) nuoadmin-1 and nuoadmin-2 repectively. The domain looks like this:

nuocmd show domain
server version: ..., server license: Enterprise
server time: ..., client token: ...
Servers:
  [nuoadmin-0] host1:48005 [last_ack = 7.57] ACTIVE (LEADER, Leader=nuoadmin-0, log=199574/154/154) Connected *
  [nuoadmin-1] host2:48005 [last_ack = 7.57] ACTIVE (FOLLOWER, Leader=nuoadmin-0, log=199574/154/154) Connected
Databases:
  test [state = RUNNING]
    [SM] host1:48006 [start_id = 10] [server_id = nuoadmin-0] [pid = 640] [node_id = 1] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 12] [server_id = nuoadmin-0] [pid = 357] [node_id = 3] [last_ack =  8.08] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 11] [server_id = nuoadmin-1] [pid = 207] [node_id = 2] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 13] [server_id = nuoadmin-1] [pid = 291] [node_id = 4] [last_ack =  9.17] MONITORED:RUNNING

To restore to a specific backup element, do the following:

  1. Identify a backup collection to be restored. In this example each SM is backing up to a local /volumes/backups directory on its host. Copying locally is usually faster, but this is not a restriction.

  2. Identify the directory where each backup set in the backup collection is located. In this example, the backup collection contains two backup directories, one on each host at /volumes/backups/BackupSets/2022-04-18.

  3. Using NuoDB Archive's --report-backups command, generate summaries of each backup set. For example:

    On host1:

    nuoarchive restore --report-backups /volumes/backups/BackupSets/2022-04-18 | tee /tmp/sm1-backupset.xml
    <BackupSet id="16df36f3-afb1-bf45-5e87-a6b6d3e44fdf" database="test" databaseId="efa117e7-8b5c-5449-14ab-80f77e81334b" collectionId="fb7f28df-
405f-488e-b7e4-7307e373be7f" archiveId="9e1a097b-e310-2f4a-38ab-f04f4eaeb1c8">
 <BackupElements>
   <BackupElement id="63bdb6de-5382-459a-8397-c786f16817d4" type="incremental" startDate="2022-04-18 14:20:45" endDate="2022-04-18 14:20:46"/>
   <BackupElement id="4d7338fe-f3c9-4041-aaa1-77ba730807f5" type="journal" startDate="2022-04-18 14:20:37" endDate="2022-04-18 14:20:39"/>
   <BackupElement id="9be268f4-4b11-4913-b7af-cc27bd970d0e" type="incremental" startDate="2022-04-18 14:20:30" endDate="2022-04-18 14:20:31"/>
   <BackupElement id="fb7f28df-405f-488e-b7e4-7307e373be7f" type="full" startDate="2022-04-18 14:20:24" endDate="2022-04-18 14:20:24"/
 </BackupElements>
</BackupSet>

    On host2:

    nuoarchive restore --report-backups /volumes/backups/BackupSets/2022-04-18 | tee /tmp/sm2-backupset.xml
    <BackupSet id="785ba1e7-9946-8b48-dd9f-d579a5c4fad5" database="test" databaseId="efa117e7-8b5c-5449-14ab-80f77e81334b" collectionId="fb7f28df-
405f-488e-b7e4-7307e373be7f" archiveId="3203d1ed-2c55-2940-df88-da01c6819cce">
 <BackupElements>
   <BackupElement id="63bdb6de-5382-459a-8397-c786f16817d4" type="incremental" startDate="2022-04-18 14:20:45" endDate="2022-04-18 14:20:46"/>
   <BackupElement id="4d7338fe-f3c9-4041-aaa1-77ba730807f5" type="journal" startDate="2022-04-18 14:20:37" endDate="2022-04-18 14:20:39"/>
   <BackupElement id="9be268f4-4b11-4913-b7af-cc27bd970d0e" type="incremental" startDate="2022-04-18 14:20:30" endDate="2022-04-18 14:20:31"/>
   <BackupElement id="fb7f28df-405f-488e-b7e4-7307e373be7f" type="full" startDate="2022-04-18 14:20:24" endDate="2022-04-18 14:20:24"/>
 </BackupElements>
</BackupSet>

  4. Copy summaries to all hosts involved in the restore so that each summary is available on each host. For example, use scp to copy the summary generated on host1 to host2, and vice versa:

    scp host1:/tmp/sm1-backupset.xml host2:/tmp/
    scp host2:/tmp/sm2-backupset.xml host1:/tmp/

  5. Restore a consistent copy of the archive from each backup set using the same backup element ID on each host. The following example demonstrates restoring of the backup set on each host to the first incremental backup element ( /volumes/archives/test2 on host1 and /volumes/archives/test2 on host2).

    On host1:

    nuoarchive restore --backup-element-id 9be268f4-4b11-4913-b7af-cc27bd970d0e --restore-dir /volumes/archives/test2 /volumes/backups/BackupSets/2022-04-18 /tmp/sm2-backupset.xml

    On host2:

    nuoarchive restore --backup-element-id 9be268f4-4b11-4913-b7af-cc27bd970d0e --restore-dir /volumes/archives/test2 /volumes/backups/BackupSets/2022-04-18 /tmp/sm1-backupset.xml

  6. Ensure correct ownership. On each host run:

    sudo chown -R nuodb:nuodb /volumes/archives/test2

  7. Optionally, move the journal to its own volume. If you do this:

    • Check to make sure ownership is still correct.

    • Use --journal-dir when invoking nuocmd create archive.

  8. Start a new database, with Storage Managers (SMs) serving the restored archives. For example, to start a database named test2 with two SMs serving the restored archives from step 5:

    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-1
    nuocmd create archive --db-name test2 --restored --archive-path /volumes/archives/test2 --server-id nuoadmin-2
    nuocmd create database --db-name test2 --dba-user ... --dba-password ... --te-server-ids nuoadmin-1 nuoadmin-1

  9. Now the domain looks like this:

    nuocmd show domain
    server version: ..., server license: Enterprise
server time: ..., client token: ...
Servers:
  [nuoadmin-0] host1:48005 [last_ack = 4.16] ACTIVE (LEADER, Leader=nuoadmin-0, log=199574/163/163) Connected *
  [nuoadmin-1] host2:48005 [last_ack = 4.16] ACTIVE (FOLLOWER, Leader=nuoadmin-0, log=199574/163/163) Connected
Databases:
  test [state = RUNNING]
    [SM] host1:48006 [start_id = 10] [server_id = nuoadmin-0] [pid = 640] [node_id = 1] [last_ack = 10.18] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 12] [server_id = nuoadmin-0] [pid = 357] [node_id = 3] [last_ack =  8.08] MONITORED:RUNNING
    [SM] host2:48006 [start_id = 11] [server_id = nuoadmin-1] [pid = 207] [node_id = 2] [last_ack =  9.84] MONITORED:RUNNING
    [TE] host1:48007 [start_id = 13] [server_id = nuoadmin-1] [pid = 291] [node_id = 4] [last_ack =  9.17] MONITORED:RUNNING
  test2 [state = RUNNING]
    [SM] host1:48008 [start_id = 0] [server_id = nuoadmin-0] [pid = 827] [node_id = 1] [last_ack =  9.92] MONITORED:RUNNING
    [TE] host1:48009 [start_id = 2] [server_id = nuoadmin-0] [pid = 831] [node_id = 3] [last_ack =  9.21] MONITORED:RUNNING
    [SM] host2:48008 [start_id = 1] [server_id = nuoadmin-1] [pid = 306] [node_id = 2] [last_ack = 10.05] MONITORED:RUNNING
    [TE] host1:48009 [start_id = 3] [server_id = nuoadmin-1] [pid = 315] [node_id = 4] [last_ack =  9.53] MONITORED:RUNNING