Re-establishing Admin Process Quorum

When loss of a NuoDB Admin process causes loss of admin process quorumIn a domain, a quorum is the minimum number of brokers that must acknowledge a change to the durable domain configuration for that change to be made. Usually, this is a simple majority. Many operations in the management tier require acknowledgement from a majority of brokers in the domain in order to ensure safety (never returning an incorrect result) in case of failure. and you cannot restart that admin process then you must re-establish admin process quorum. The following procedure re-establishes admin process quorum by ensuring that the lost admin process is not counted when determining whether a majority of admin processes in the durable domain configuration is available. To re-establish admin process quorum:

  1. To make it easy to obtain the stable ID of the lost admin process host, invoke the NuoDB Manager set property command to show the stable ID in domain status output. For example:

    set property showHostStableId value true        
  2. Obtain the stable ID of the lost admin process host by invoking the NuoDB Manager show domain summary command.
  3. Shut down all NuoDB processes.
  4. On each host that was running a surviving admin process, with the admin process shut down, in the NUODB_HOME/bin directory, invoke the agent-tool utility to ban the unavailable admin process. This prevents the admin process of the unavailable admin process from continuing to be a resource in the domain. For example, if the stable ID of the unavailable admin process host is 123456789 then you would enter a command such as the following:

    /opt/nuodb/bin: agent-tool membership --ban uuid:123456789
  5. You must do this on each host of a surviving admin process. Banning an admin process means that the admin process can never use the same stable ID to peer into the domain. When the host machine is repaired and you want to start an admin process on that host then you must start the admin process with the --reset-broker-state option. This removes the old stable ID and the old Raft data. When the admin process starts, it gets the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each NuoDB Admin process in the domain by means of a Raft log. from another admin process.

  6. Restart each admin process that was available before you shut down the domain's resources. When a admin process starts, it immediately removes any banned admin process from the durable domain configuration. The re-started admin process prevent any banned admin process from being counted when determining the required number of admin processes for admin process quorum.

With admin process quorum established, any banned admin processes are removed from the durable domain configuration. The banned admin processes cannot use the same stable ID to re-peer into the domain.