A NuoDB Domain is defined by a set of coordinating APs peered together.
Each AP uses a local file (its raftlog) to hold all the information about its domain, both its expected and actual state. The APs are completely redundant, any one of them has a fully up-to-date knowledge of the domain and all APs and databases defined within it, running or otherwise.
Changes to the domain are only possible while a majority (over 50%) of all known APs are running. If the number of APs falls below that number the domain goes into read-only mode. Databases will keep running but no changes to the domain or database processes are possible and no new connections will be given to clients. This mechanism comes into play if network connectivity fails between some APs or if the AP processes themselves fail or are taken down.
Once sufficient APs are running again, majority can be restored. Majority decisions are handled using the RAFT algorithm, hence the name of the AP’s local file.
Once the APs are running and peering successfully, the domain can be managed via the AP’s REST API, typically using the
This is the only way to create archives and databases and to start and stop database processes.
|A NuoDB database is not just files stored on disk. It is also the engine processes (TEs and SMs) that make it work.|
In a bare-metal or VM-based deployment, each TE or SM is started by an AP process, as specified in the start process command. Conveniently, the command can be issued to any AP which, if necessary, will forward the request to the AP on the desired host.
Under Kubernetes, the engines are started inside their own pod by Kubernetes itself.
Once the engine has started, it is handed off to an AP to be managed.
A dedicated script,
nuodocker, is installed in our container for this purpose —
nuodocker is an extension of
nuocmd with support for containerization.
Client Connection Brokering and Load Balancing
When a client requests a connection, the AP it is talking to will choose a TE and reply with that TE’s address By default connections are assigned randomly across all available TEs for a given database in order to balance the load.
This default can be changed and APs configured to force connections to, or away from, a specific TE or subset of TEs. Hence, if certain connections can be restricted to querying only a subset of tables, connections can also be restricted to a subset of TEs. Only those TEs will have that subset of tables cached in their local memory. In this way the cache requirements of different TEs can be managed by restricting the data they will cache. Alternatively, different TEs can be restricted to specific types of queries, for example OLTP or OLAP.
See the dedicated section on load-balancing for full details.
The Admin Process (AP)
The AP is implemented by
nuoadmin, a Java application.
It is run via this script:
$NUODB_HOME/etc/nuoadmin (Linux) or
In bare-metal/VM environments, it needs to be configured before it will run and can be setup as a service on its host. It is a requirement to have an AP on every host with a TE and/or an SM on it. For more details see Domain Operations.
In Kubernetes environments the APs are started in their own pods and it is normal to only have enough for redundancy and to ensure majority, 3 or 5 is typical. The APs are configured by values passed to the Helm admin chart which deploys them. For details refer to the Admin Helm Chart on github.