Removing a Storage Manager

On occasion, you may need to remove a Storage Manager (SM) from a NuoDB database. To remove an SM, do the following:

1. From the OpenShift master node, run the kubectl get pod command to identify the node you wish to remove a SM from.

kubectl get pod -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName
NAME                           STATUS    NODE
a-0                            Running   ip-10-0-1-218.ec2.internal
a-1                            Running   ip-10-0-1-142.ec2.internal
a-2                            Running   ip-10-0-1-146.ec2.internal
b-0                            Running   ip-10-0-2-153.ec2.internal
b-1                            Running   ip-10-0-2-250.ec2.internal
b-2                            Running   ip-10-0-2-217.ec2.internal
nuodb-insights                 Running   ip-10-0-1-146.ec2.internal
te-a-eqvsj-1-jjbhz             Running   ip-10-0-1-142.ec2.internal
te-a-eqvsj-1-twn25             Running   ip-10-0-1-137.ec2.internal
te-b-negmj-1-4cnjq             Running   ip-10-0-2-153.ec2.internal
te-b-negmj-1-g8str             Running   ip-10-0-2-250.ec2.internal
test-storage-wo-backup-25w2s   Running   ip-10-0-1-53.ec2.internal
test-storage-wo-backup-5kkxb   Running   ip-10-0-1-218.ec2.internal
test-storage-wo-backup-67mnv   Running   ip-10-0-2-217.ec2.internal
test-storage-wo-backup-hkmqb   Running   ip-10-0-2-43.ec2.internal
ycsb-load-a-toqge-sq7b9        Running   ip-10-0-1-142.ec2.internal

2. Remove the storage label for the node (for example, ip-10-0-2-43.ec2.internal) and it will also remove the storage manager pod (test-storage-wo-backup-hmqb) from the database.

oc label node ip-10-0-2-43.ec2.internal nuodb.com/nuodb.test-

3. To verify that the SM was removed, typing the following in a (NuoDB Admin Service pod) Terminal tab in the OpenShift Web UI:

bash
nuocmd show domain

Note: Under Databases:, instead of RUNNING the status for the test database is now [AWAITING_ARCHIVE_HISTORIES_INC]. This is normal and indicates that the database is still RUNNING but that a SM was removed and its archive entry in NuoDB and disk storage on pod ip-10-0-2-43.ec2.internal still remains.

5. To confirm the archive entry that is no longer running, type nuocmd show archives:

nuocmd show archives
Archive: [0] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-67mnv/10.130.2.8:48006 (Default) [sid = 0] [server = b-2] MONITORED:RUNNING
Archive: [1] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-25w2s/10.129.4.8:48006 (Default) [sid = 2] [server = a-2] MONITORED:RUNNING
Archive: [2] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] RUNNING
[SM] test-storage-wo-backup-5kkxb/10.130.4.5:48006 (Default) [sid = 1] [server = a-0] MONITORED:RUNNING
Archive: [3] <NO VALUE> : /var/opt/nuodb/archive/nuodb/test [db = test] NOT_RUNNING
[SM] test-storage-wo-backup-hkmqb/10.130.0.8:48006 (Default) [start_id = 3] [server_id = a-2] [pid = 44] [node_id = 3] EXITED(REQUESTED_SHUTDOWN:RUNNING):Gracefully shutdown engine (?)
bash-4.2$

The output displays Archive: [3]... which is the archive that belonged to the removed SM.

6. To delete this archive entry, type the following:

nuocmd delete archive archive-id 3

7. To verify that the exited archive was deleted and the database has returned to a status of RUNNING, type the following:

bash
nuocmd show domain

8. Finally, to delete the archive storage on disk, ssh into the node that was hosting the removed SM and remove the nuodb archive directory.

ssh ip-10-0-2-43.ec2.internal
[centos@storage03 ~]$ rm -rf /mnt/local-storage/ebs/disk0/nuodb