Open topic with navigation
To ensure safety (never returning an incorrect result), any change to the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each broker in the domain by means of a Raft log. requires a broker quorum. A broker quorum exists when a majority of the brokers in the domain are available to the other running brokers. Typically, the minimum number required for a majority is easy to identify. You can also apply one of the following formulas according to whether there is an odd number or an even number of brokers in the domain membership. Suppose that there are
n brokers in the domain.
|Number of Brokers in the Domain Membership||Minimum Number of Available Brokers Required for Quorum|
For example, if there are
5 brokers in a domain then there is a quorum when there are at least
3 available brokers.
At any given moment, the leader broker (see About Brokers and Peering) determines whether there is a quorum. It does not matter whether there is more than one region. The leader broker determines the number of available brokers in the domain regardless of region.
To perform any of the following tasks, there must be a broker quorum because each of these tasks entails an update to the durable domain configuration, which is maintained by each broker:
Note: Without a broker quorum, a database may continue to run and serve clients. See Monitoring Database Operation and Obtaining Availability Status for Client Connections.
Brokers implement the Raft Consensus Algorithm. The durable domain configuration is implemented by using domain state machines (DSM) that use a Raft log. See About the Durable Domain Configuration.
When a domain contains an even number of brokers then a quorum requires that at least half plus one of those brokers are available. If a domain contains 2 brokers then both brokers must be running for there to be a quorum. The following table shows the minimum number of brokers that are requred for a quorum according to the numbers of brokers in the durable domain configuration. It also shows the maximum number of brokers in a domain that can fail without limiting performance of domain tasks.
Number of Brokers in the Durable Domain Configuration
Minimum Number of Brokers Required for Quorum
Maximum Number of Brokers That Can Fail Without Limiting Operations
A domain with 3 brokers (as in the example in About Brokers and Peering) is fully operational as long as 2 of the 3 brokers are available. If one broker's host machine is disconnected from the network, the other brokers continue to allow safe operations in the domain. Also, they continue to ping and try to reconnect to the missing broker until it is back online. Upon reconnect, the third broker safely catches up by synchronizing its durable domain configuration with the durable domain configuration of the other 2 brokers.
In a domain that has 3 brokers, the minimum number of available brokers required for a quorum is 2. In a domain that has 2 brokers, the minimum number of available brokers required for a quorum is also 2. However, an important difference is that in the domain with 3 brokers, one broker can disconnect from the domain and the domain continues to operate safely. If a broker disconnects from the domain with 2 brokers then the domain no longer has a broker quorum. None of the tasks that change the durable domain configuration and that therefore require a broker quorum can be performed until a broker quorum is restored. While a domain with 2 brokers provides durability if 1 broker fails, a domain must have at least 3 brokers to continue normal operations if 1 broker fails.
As you configure your domain, you should consider what portion of your domain can become disconnected without preventing domain operations. In your domain, if you assign brokers to multiple regions, remember that it is the number of available brokers in the domain, and not in a particular region, that determines whether there is a quorum. For example, suppose there is a domain with the following configuration:
B3are in the
B5are in the
B6is in the
The following figure illustrates this configuration:
This domain has 6 brokers, which means that a quorum requires a minimum of 4 brokers to be available. If the
east region becomes disconnected, then 1 broker becomes unavailable, as shown in the following figure:
The domain has a broker quorum as long as at least 4 of the 5 brokers in the
central regions are available. If only the
central region is disconnected, then the domain loses 2 brokers as shown in the following figure:
east regions have 4 brokers. As long as they are all available, the domain has broker quorum. However, if the
east regions are connected but the
west region becomes disconnected, a broker quorum is no longer possible. Together, the
east regions have only 3 available brokers, as shown in the following figure:
The loss of the
west region is a potential, single point of failure with regard to tasks that change the durable domain configuration. This is the case even though 2 other regions remain connected.