Re-establishing Broker Quorum

When loss of a broker causes loss of broker quorumIn a domain, a quorum is the minimum number of brokers that must acknowledge a change to the durable domain configuration for that change to be made. Usually, this is a simple majority. Many operations in the management tier require acknowledgement from a majority of brokers in the domain in order to ensure safety (never returning an incorrect result) in case of failure. and you cannot restart that broker then you must re-establish broker quorum. The following procedure re-establishes broker quorum by ensuring that the lost broker is not counted when determining whether a majority of brokers in the durable domain configuration is available. To re-establish broker quorum:

  1. To make it easy to obtain the stable ID of the lost broker's host, invoke the NuoDB Manager set property command to show the stable ID in domain status output. For example:

                        set property showHostStableId value true
                    
  2. Obtain the stable ID of the lost broker's host by invoking the NuoDB Manager show domain summary command.
  3. Shut down all NuoDB processes, brokers and agents.
  4. On each host that was running a surviving broker, with the broker shut down, in the NUODB_HOME/bin directory, invoke the agent-tool utility to ban the unavailable broker. This prevents the host of the unavailable broker from continuing to be a resource in the domain. For example, if the stable ID of the unavailable broker's host is 123456789 then you would enter a command such as the following:

    /opt/nuodb/bin: agent-tool membership --ban uuid:123456789

    You must do this on each host of a surviving broker. Banning a broker means that the broker can never use the same stable ID to peer into the domain. When the host machine is repaired and you want to start a broker on that host then you must start the broker with the --reset-broker-state option. This removes the old stable ID and the old Raft data. When the broker starts it gets the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each broker in the domain by means of a Raft log. from another broker.

  5. Restart each broker that was available before you shut down the domain's resources. When a broker starts it immediately removes any banned broker(s) from the durable domain configuration. The re-started brokers prevent any banned brokers from being counted when determining the required number of brokers for the broker quorum.

With broker quorum established, any banned brokers are removed from the durable domain configuration. The banned brokers cannot use the same stable ID to re-peer into the domain.