Open topic with navigation
To ensure safety (never returning an incorrect result), any change to the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each NuoDB Admin process in the domain by means of a Raft log. requires an admin process quorum.
An admin process quorum exists when the admin processes that are available to the other running admin processes are on a majority of the admin servers in the domain. Typically, the minimum number required for a majority is easy to identify. You can also apply one of the following formulas according to whether there is an odd number or an even number of admin servers in the domain membership. Suppose that there are
n admin servers in the domain.
|Number of Admin Servers in the Domain Membership||Minimum Number of Available Admin Processes Required for Quorum|
For example, if there are
5 admin servers in a domain then there is a quorum when there are at least
3 available admin processes.
At any given moment, the leader admin process (see About Admin Processes and Peering) determines whether there is a quorum. It does not matter whether there is more than one region. The leader admin process determines the number of available admin processes in the domain regardless of region.
To perform any of the following tasks, there must be an admin process quorum because each of these tasks entails an update to the durable domain configuration, which is maintained by each admin process:
Adding or removing an admin server from the domain
Note: Without an admin process quorum, a database may continue to run and serve clients. See Monitoring Database Operation.
Admin processes implement the Raft Consensus Algorithm. The durable domain configuration is implemented by using domain state machines (DSM) that use a Raft log. See About the Durable Domain Configuration.
When a domain contains an even number of admin servers then a quorum requires that admin processes are available for one half plus one of the admin servers.
If a domain contains two admin servers then admin processes must be running for each admin server for there to be a quorum. The following table shows the minimum number of admin processes that are required for a quorum according to the numbers of admin servers in the durable domain configuration. It also shows the maximum number of admin processes in a domain that can fail without limiting performance of domain tasks.
Number of Admin Servers in the Durable Domain Configuration
Minimum Number of Admin Processes Required for Quorum
Maximum Number of Admin Processes That Can Fail Without Limiting Operations
A domain with three admin servers (as in the example in About Admin Processes and Peering) is fully operational as long as two admin processes are available. If one admin process host machine is disconnected from the network, the other admin processes continue to allow safe operations in the domain. Also, they continue to ping and try to reconnect to the missing admin process until it is back online. Upon reconnection, the third admin process safely catches up by synchronizing its durable domain configuration with the durable domain configuration of the other two admin processes.
In a domain that has three admin servers, the minimum number of available admin processes required for a quorum is two. In a domain that has two admin servers, the minimum number of available admin processes required for a quorum is also two. However, an important difference is that in the domain with three admin servers, one admin process can disconnect from the domain and the domain continues to operate safely. If an admin process disconnects from the domain with two admin servers then the domain no longer has an admin process quorum. None of the tasks that change the durable domain configuration and that therefore require an admin process quorum can be performed until an admin process quorum is restored. While a domain with two admin servers provides durability if one admin process fails, a domain must have at least three admin servers to continue normal operations if one admin process fails.
As you configure your domain, you should consider what portion of your domain can become disconnected without preventing domain operations. In your domain, if you assign admin servers to multiple regions, remember that it is the number of available admin processes in the domain, and not in a particular region, that determines whether there is a quorum. For example, suppose there is a domain with the following configuration:
AP3are in the
AP5are in the
AP6is in the
The following figure illustrates this configuration:
This domain has six admin servers, which means that a quorum requires a minimum of four admin processes to be available. If the
east region becomes disconnected, then one admin processes becomes unavailable, as shown in the following figure:
The domain has an admin process quorum as long as at least four of the five admin processes in the
central regions are available. If only the
central region is disconnected, then the domain loses two admin processes as shown in the following figure:
east regions have four admin processes. As long as they are all available, the domain has an admin process quorum. However, if the
east regions are connected but the
west region becomes disconnected, an admin process quorum is no longer possible. Together, the
east regions have only three available admin processes, as shown in the following figure:
The loss of the
west region is a potential, single point of failure with regard to tasks that change the durable domain configuration. This is the case even though two other regions remain connected.