About Admin Process Quorum

To ensure safety (never returning an incorrect result), any change to the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each NuoDB Admin process in the domain by means of a Raft log. requires an admin process quorum. An admin process quorum exists when a majority of the admin processes in the domain are available to the other running admin processes. Typically, the minimum number required for a majority is easy to identify. You can also apply one of the following formulas according to whether there is an odd number or an even number of admin processes in the domain membership. Suppose that there are n admin processes in the domain.

Number of admin processes in the Domain Membership Minimum Number of Available Admin Processes Required for Quorum
Even number (n/2) + 1
Odd number (n/2) rounded up

For example, if there are 5 admin processes in a domain then there is a quorum when there are at least 3 available admin processes.

At any given moment, the leader admin process (see About Admin Processes and Peering) determines whether there is a quorum. It does not matter whether there is more than one region. The leader admin process determines the number of available admin processes in the domain regardless of region.

To perform any of the following tasks, there must be admin process quorum because each of these tasks entails an update to the durable domain configuration, which is maintained by each admin process:

Note: Without an admin process quorum, a database may continue to run and serve clients. See Monitoring Database Operation and Obtaining Availability Status for Client Connections.

Admin rocesses implement the Raft Consensus Algorithm. The durable domain configuration is implemented by using domain state machines (DSM) that use a Raft log. See About the Durable Domain Configuration.

When a domain contains an even number of admin processes then a quorum requires that at least half plus one of those admin processes are available. If a domain contains 2 admin processes then both admin processes must be running for there to be a quorum. The following table shows the minimum number of admin processes that are required for a quorum according to the numbers of admin processes in the durable domain configuration. It also shows the maximum number of admin processes in a domain that can fail without limiting performance of domain tasks.

Number of Admin Processes in the Durable Domain Configuration

Minimum Number of Admin Processes Required for Quorum

Maximum Number of Admin Processes That Can Fail Without Limiting Operations

1 1 0
2 2 0
3 2 1
4 3 1
5 3 2
6 4 2
7 4 3
8 5 3
9 5 4
10 6 4

A domain with three admin processes (as in the example in About Admin Processes and Peering) is fully operational as long as two of the three admin processes are available. If one admin process host machine is disconnected from the network, the other admin processes continue to allow safe operations in the domain. Also, they continue to ping and try to reconnect to the missing admin process until it is back online. Upon reconnect, the third admin process safely catches up by synchronizing its durable domain configuration with the durable domain configuration of the other two admin processes.

In a domain that has three admin processes, the minimum number of available admin processes required for a quorum is two. In a domain that has two admin processes, the minimum number of available admin processes required for a quorum is also two. However, an important difference is that in the domain with three admin processes, one admin process can disconnect from the domain and the domain continues to operate safely. If a admin process disconnects from the domain with two admin process then the domain no longer has an admin quorum. None of the tasks that change the durable domain configuration and that therefore require an admin process quorum can be performed until an admin process quorum is restored. While a domain with two admin processes provides durability if one admin process fails, a domain must have at least three admin processes to continue normal operations if one admin process fails.

As you configure your domain, you should consider what portion of your domain can become disconnected without preventing domain operations. In your domain, if you assign admin processes to multiple regions, remember that it is the number of available admin processes in the domain, and not in a particular region, that determines whether there is a quorum. For example, suppose there is a domain with the following configuration:

The following figure illustrates this configuration:

This domain has six admin processes, which means that a quorum requires a minimum of four admin processes to be available. If the east region becomes disconnected, then one admin processes becomes unavailable, as shown in the following figure:

The domain has an admin quorum as long as at least four of the five admin processes in the west and central regions are available. If only the central region is disconnected, then the domain loses two admin processes as shown in the following figure:

Together, the west and east regions have four admin processes. As long as they are all available, the domain has an admin process quorum. However, if the central and east regions are connected but the west region becomes disconnected, an admin quorum is no longer possible. Together, the central and east regions have only three available admin processes, as shown in the following figure:

The loss of the west region is a potential, single point of failure with regard to tasks that change the durable domain configuration. This is the case even though two other regions remain connected.