Removing an Unreachable Admin Server from the Durable Domain

If an Admin Process (AP) has become unreachable and cannot be restarted, for example when the machine that the AP was running on is terminated, there is a risk that this could affect the forming of an AP quorum and the overall operation of the domain and/or the database processes that the AP was connected to. For information on AP quorum, and what operations are not possible when quorum is not established, see Admin Process (AP) Quorum.

The domain and databases in the domain can function without APs running and connected for all of the admin servers in the durable domain. If it is determined that an AP will never be restarted the admin server it is associated with should be removed from the durable domain. The steps to remove an admin server when there is a valid quorum formed for the domain are outlined here. (For the steps to re-establish a valid AP quorum, see Re-establishing Admin Process (AP) Quorum.)

It may not be possible to re-provision an admin server with the same server ID into the domain.

1. Determine the server ID of the admin server to be removed by running the show domain command and reviewing the list of servers. In the example output below the server r0db2 is Disconnected.

nuocmd show domain
...
Servers:
  [r0db0] 172.31.43.237:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected
  [r0db1] 172.31.46.156:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db1, log=95/143/143) Connected
  [r0db2] 172.31.34.187:48005 [last_ack = 159.32] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Disconnected
  [r0db3] 172.31.35.152:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected

2. If it is established that the AP on admin server r0db2 cannot be restarted or reconnected to the domain, the admin server can be removed from the durable domain using nuocmd delete server --server-id r0db2. When the show domain command is run again r0db2 will no longer be in the list of admin servers:

nuocmd show domain
...
Servers:
  [r0db0] 172.31.43.237:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected
  [r0db1] 172.31.46.156:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db1, log=95/143/143) Connected
  [r0db3] 172.31.35.152:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected *

If the issue that caused you to delete a server ID from the domain is resolved, you may wish add the admin server back into the durable domain.

The show domain command is issued using NuoDB Command (nuocmd). For more information on NuoDB Command and other command line tools, see Command Line Tools.