Open topic with navigation
Re-establishing Broker Quorum
When loss of a broker causes loss of broker quorumIn a domain, a quorum is the minimum number of brokers that must acknowledge a change to the durable domain configuration for that change to be made. Usually, this is a simple majority. Many operations in the management tier require acknowledgement from a majority of brokers in the domain in order to ensure safety (never returning an incorrect result) in case of failure. and you cannot restart that broker then you must re-establish broker quorum. The following procedure re-establishes broker quorum by ensuring that the lost broker is not counted when determining whether a majority of brokers in the durable domain configuration is available. To re-establish broker quorum:
To make it easy to obtain the stable ID of the lost broker's host, invoke the NuoDB Manager
set property command to show the stable ID in domain status output. For example:
set property showHostStableId value true
show domain summarycommand.
On each host that was running a surviving broker, with the broker shut down, in the
NUODB_HOME/bin directory, invoke the
agent-tool utility to ban the unavailable broker. This prevents the host of the unavailable broker from continuing to be a resource in the domain. For example, if the stable ID of the unavailable broker's host is
123456789 then you would enter a command such as the following:
agent-tool membership --ban uuid:123456789
You must do this on each host of a surviving broker. Banning a broker means that the broker can never use the same stable ID to peer into the domain. When the host machine is repaired and you want to start a broker on that host then you must start the broker with the
--reset-broker-state option. This removes the old stable ID and the old Raft data. When the broker starts it gets the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each broker in the domain by means of a Raft log. from another broker.
With broker quorum established, any banned brokers are removed from the durable domain configuration. The banned brokers cannot use the same stable ID to re-peer into the domain.