Removing an Unreachable Admin Server from the Durable Domain

If an admin process has become unreachable and cannot be restarted, for example when the machine that the admin process was running on is terminated, there is a risk that this could affect the forming of an admin process quorum and the overall operation of the domain and/or the database processes that the admin process was connected to. For information on admin process quorum, and what operations are not possible when quorum is not established, see About Admin Process Quorum.

The domain and databases in the domain can function without admin processes running and connected for all of the admin servers in the durable domain. If it is determined that an admin process will never be restarted the admin server it is associated with should be removed from the durable domain. The steps to remove an admin server when there is a valid quorum formed for the domain are outlined here. (For the steps to re-establish a valid admin process quorum, see Re-establishing Admin Process Quorum.)

Note: It may not be possible to re-provision an admin server with the same server ID into the domain.

1. Determine the server ID of the admin server to be removed by running the show domain command and reviewing the list of servers. In the example output below the server r0db2 is Disconnected.

nuocmd show domain
...
Servers:
  [r0db0] 172.31.43.237:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected
  [r0db1] 172.31.46.156:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db1, log=95/143/143) Connected
  [r0db2] 172.31.34.187:48005 [last_ack = 159.32] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Disconnected
  [r0db3] 172.31.35.152:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected

2. If it is established that the admin process on admin server r0db2 cannot be restarted or reconnected to the domain, the admin server can be removed from the durable domain using nuocmd delete server --server-id r0db2. When the show domain command is run again r0db2 will no longer be in the list of admin servers:

nuocmd show domain
...
Servers:
  [r0db0] 172.31.43.237:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected
  [r0db1] 172.31.46.156:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db1, log=95/143/143) Connected
  [r0db3] 172.31.35.152:48005 [last_ack = 1.00] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db1, log=95/143/143) Connected *

If the issue that caused you to delete a server ID from the domain is resolved, you may wish add the admin server back into the durable domain. For more information, see Reprovisioning an Admin Server.

 

Note: The show domain command is issued using NuoDB Command (nuocmd). For more information on NuoDB Command and other command line tools, see Command Line Tools.