Determining if Admin Process (AP) Quorum Exists

As discussed in Admin Process (AP) Quorum, a majority of the admin servers in the domain must have running Admin Processes (APs) available to the other running APs to make changes to the durable domain configuration. Since these are tasks that you might need to perform to resolve a failure, it is important to know if AP quorum exists.

The show domain command displays the state of the domain and database as seen by the AP you are running the command against.

show domain is run using NuoDB Command (nuocmd). For more information on NuoDB Command and other command line tools, see Command Line Tools.

Since network disruptions could affect different processes in different manners the output from the command may be different depending on which AP you are connected to. In the example below APs r0db2, r0db3 and r0db4 are disconnected from r0db0 and r0db1. In addition, there is no leader identified in the connected nodes. Since only two of the five members of the domain are connected and there is no leader established, there is not an AP quorum that includes host r0db0 and r0db1.

nuocmd show domain
...
Servers:
  [r0db0] 172.31.45.7:48005 [last_ack = 9.32] [member = ADDED]  [raft_state = ACTIVE] (FOLLOWER, Leader=<NO VALUE>, log=5/99/100)  Connected *
  [r0db1] 172.31.44.101:48005 [last_ack = 9.32] [member = ADDED]  [raft_state = ACTIVE] (FOLLOWER, Leader=<NO VALUE>, log=5/99/100)  Connected
  [r0db2] 172.31.42.100:48005 [last_ack = 119.32] [member = ADDED]  [raft_state = ACTIVE] (FOLLOWER, Leader=r0db4, log=5/98/98)  Disconnected
  [r0db3] 172.31.47.31:48005 [last_ack = 59.32] [member = ADDED]  [raft_state = ACTIVE] (FOLLOWER, Leader=r0db4, log=5/98/98)  Disconnected
  [r0db4] 172.31.47.176:48005 [last_ack = 29.32] [member = ADDED]  [raft_state = ACTIVE] (LEADER, Leader=r0db4, log=5/99/99)  Disconnected

Even if a set of APs have agreed upon an elected leader this does not mean that quorum is established. In the example below the six APs identified by server IDs r1db0, r1db1, r1db2, r1db3, r1db4 and r1db5 are all connected and agree that r1db1 is the leader. However since the six APs do not represent a majority of the 12 admin servers in the domain they do not form an AP quorum.

...
Servers:
  [r0db0] 172.31.40.33:48005 [last_ack = 1239.537] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db0, log=5/215/215) Disconnected
  [r0db1] 172.31.33.174:48005 [last_ack = 1229.537] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db0, log=5/215/215) Disconnected
  [r0db2] 172.31.34.204:48005 [last_ack = 1219.537] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=None, log=6/216/216) Disconnected
  [r0db3] 172.31.47.96:48005 [last_ack = 1219.537] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=None, log=6/216/216) Disconnected
  [r0db4] 172.31.42.146:48005 [last_ack = 1209.536] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r0db4, log=8/217/217) Disconnected
  [r0db5] 172.31.33.106:48005 [last_ack = 1209.537] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r0db4, log=8/217/217) Disconnected
  [r1db0] 172.31.26.105:48005 [last_ack = 53.430] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r1db1, log=9/218/224) Connected *
  [r1db1] 172.31.28.109:48005 [last_ack = 53.429] [member = ADDED] [raft_state = ACTIVE] (LEADER, Leader=r1db1, log=9/218/224) Connected
  [r1db2] 172.31.19.203:48005 [last_ack = 53.431] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r1db1, log=9/218/224) Connected
  [r1db3] 172.31.24.211:48005 [last_ack = 53.431] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r1db1, log=9/218/224) Connected
  [r1db4] 172.31.17.17:48005 [last_ack = 63.927] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r1db1, log=9/218/224) Connected
  [r1db5] 172.31.19.127:48005 [last_ack = 63.926] [member = ADDED] [raft_state = ACTIVE] (FOLLOWER, Leader=r1db1, log=9/218/224) Connected