Removing a Storage Manager

On occasion, you may need to remove a Storage Manager (SM) from a NuoDB database. To remove an SM, do the following:

1. From the OpenShift master node, run the kubectl get pod command to identify the node you wish to remove a SM from.

kubectl get pod,STATUS:.status.phase,NODE:.spec.nodeName
NAME                           STATUS    NODE
a-0                            Running   ip-10-0-1-218.ec2.internal
a-1                            Running   ip-10-0-1-142.ec2.internal
a-2                            Running   ip-10-0-1-146.ec2.internal
b-0                            Running   ip-10-0-2-153.ec2.internal
b-1                            Running   ip-10-0-2-250.ec2.internal
b-2                            Running   ip-10-0-2-217.ec2.internal
nuodb-insights                 Running   ip-10-0-1-146.ec2.internal
te-a-eqvsj-1-jjbhz             Running   ip-10-0-1-142.ec2.internal
te-a-eqvsj-1-twn25             Running   ip-10-0-1-137.ec2.internal
te-b-negmj-1-4cnjq             Running   ip-10-0-2-153.ec2.internal
te-b-negmj-1-g8str             Running   ip-10-0-2-250.ec2.internal
test-storage-wo-backup-25w2s   Running   ip-10-0-1-53.ec2.internal
test-storage-wo-backup-5kkxb   Running   ip-10-0-1-218.ec2.internal
test-storage-wo-backup-67mnv   Running   ip-10-0-2-217.ec2.internal
test-storage-wo-backup-hkmqb   Running   ip-10-0-2-43.ec2.internal
ycsb-load-a-toqge-sq7b9        Running   ip-10-0-1-142.ec2.internal

2. Remove the storage label for the node (for example, ip-10-0-2-43.ec2.internal) and it will also remove the storage manager pod (test-storage-wo-backup-hmqb) from the database.

oc label node ip-10-0-2-43.ec2.internal

3. To verify that the SM was removed, typing the following in a (NuoDB Admin Service pod) Terminal tab in the OpenShift Web UI:

nuocmd show domain

Note: Under Databases:, instead of RUNNING the status for the test database is now [AWAITING_ARCHIVE_HISTORIES_INC]. This is normal and indicates that the database is still RUNNING but that a SM was removed and its archive entry in NuoDB and disk storage on pod ip-10-0-2-43.ec2.internal still remains.

5. To confirm the archive entry that is no longer running, type nuocmd show archives:

nuocmd show archives
Archive: [0] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-67mnv/ (Default) [sid = 0] [server = b-2] MONITORED:RUNNING
Archive: [1] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-25w2s/ (Default) [sid = 2] [server = a-2] MONITORED:RUNNING
Archive: [2] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-5kkxb/ (Default) [sid = 1] [server = a-0] MONITORED:RUNNING
Archive: [3] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] NOT_RUNNING
[SM] test-storage-wo-backup-hkmqb/ (Default) [start_id = 3] [server_id = a-2] [pid = 44] [node_id = 3] EXITED(REQUESTED_SHUTDOWN:RUNNING):Gracefully shutdown engine (?)

The output displays Archive: [3]... which is the archive that belonged to the removed SM.

6. To delete this archive entry, type the following:

nuocmd delete archive archive-id 3

7. To verify that the exited archive was deleted and the database has returned to a status of RUNNING, type the following:

nuocmd show domain

8. Finally, to delete the archive storage on disk, ssh into the node that was hosting the removed SM and remove the nuodb archive directory.

ssh ip-10-0-2-43.ec2.internal
[centos@storage03 ~]$ rm -rf /mnt/local-storage/ebs/disk0/nuodb