Host Properties (nuoadmin.conf)

To configure a host, set the required values for properties in nuoadmin.conf. You can find nuoadmin.conf in the following locations:

Installation type

Location

RPM / DEB

/etc/nuodb

TAR

$NUODB_HOME/etc

Windows MSI %NUODB_HOME%/etc
{
// For more information about modifying NuoDB Admin parameters, see the
// documentation in the NuoDB Admin reference section for the nuoadmin.conf:
// http://doc.nuodb.com/Latest/Default.htm#Nuoadmin-Host-Properties.htm
"ThisServerId": "nuoadmin-0",
// address to use instead of hostname or DNS name for the local server;
// consider removing "altAddr" if a stable address can be assigned to each
// admin server; NOTE: in production, "altAddr" should NOT be set to
// "localhost", which is only used to support single-host development
// use-cases
"altAddr": "localhost",
// a map of servers IDs to admin processes in the initial membership of the
// domain; if one of the server IDs matches the value of "ThisServerId", then
// the local server can bootstrap the domain (NOTE: the address specified in
// the "transport" field of the "initialMembership" entry for "ThisServerId"
// must have the same as the hostname as "altAddr" if an "altAddr" is
// specified); otherwise, the local server will attempt to enter the domain
// through one of the servers in the initial membership, assuming that the
// local server has no Raft state, e.g. it is being started for the first
// time; all members that specify an initial membership must have the same
// initial membership; alternatively, "peer" can be used to specify an entry
// peer to join the domain through, if the local server has no Raft state
"initialMembership": {
"nuoadmin-0": { "transport": "localhost:48005", "version": "0:10000" }
},
// used by engine processes to communicate with the admin and by SQL clients
// to obtain a SQL connection
"agentPort": "48004",
// used by admin processes to communicate with each other
"adminPort": "48005",
// port range for engine processes connected to this admin process
"portRange": "48006",
"adminBindAddress": "",
"otherServices": ["ping", "rest", "cron"],
"cron.command": ["${NUODB_HOME}/etc/nuoca/bin/start_insights.sh"],
"cron.initialDelay": "10000",
"cron.delay": "300000",
"restyml": "nuoadmin.rest.yml",
"logConfig": "nuoadmin.logback.xml",
"domainEntryServer.changeMembershipTimeout": "5",
"domainEntryClient.peerWaitTimeout": "60",
// the amount of time to wait for all initialized archives to be started when
// assigning storage group leaders for a database
"leaderAssignmentTimeout": "10000",
// the amount of time to wait for an engine process to connect to the admin
// after being started
"pendingProcessTimeout": "5000",
// the amount of time to wait for an engine process to reconnect to the admin
// after the admin is restarted
"pendingReconnectTimeout": "10000",
"loadBalancerExpiration": "60000",
"reaperPeriod": "5000",
// set to -1 to disable liveness checking of database processes; a better
// solution for evicting unresponsive processes is the --ping-timeout
// database option; if enabled, this should be set to a value larger than 10
// seconds, which is the period at which a process sends <Status/> messages
// to its connected admin server; if an admin does not receive a message from
// a process for more than the specified time, then a message is sent to it;
// if the process does not respond to this message within "reaperPeriod"
// milliseconds, it is evicted from the database
"processLivenessCheckSec": "-1",
// for large clusters (e.g. >=20 admin servers) or high-latency environments,
// consider increasing "broadcastLatencyMicros"; "broadcastLatencyMicros" is
// an upper bound for the amount of time it would take the raft leader to
// send a heartbeat to all followers; the heartbeat period and election
// timeout range are both calculated as multiples of "broadcastLatencyMicros"
// with the election timeout being an order of magnitude larger than the
// heartbeat period; these values can also be specified directly using the
// commented out properties below
"broadcastLatencyMicros": "1000",
// "raft.heartbeatPeriodMillis": "500",
// "raft.minElectionTimeoutMillis": "2000",
// "raft.maxElectionTimeoutMillis": "4000",
// for large clusters, consider increasing "thrift.threads.max"; this should
// be set to at least 4x the number of servers
"thrift.threads.max": "50",
// to disable TLS, set "ssl" to "false", which allows us to omit all keystore
// and truststore properties below
"ssl": "true",
"keystore": "nuoadmin.jks",
"keystore-type": "JKS",
"keystore-password": "changeIt",
"truststore": "nuoadmin-truststore.jks",
"truststore-type": "JKS",
"truststore-password": "changeIt"
}

Setting nuoadmin.conf properties

Using a text editor, define admin process settings using nuoadmin.conf properties. nuoadmin.conf properties are described in the following table:

Property

Description

Comments

initialMembership A map of servers and details for admin processes that are allowed to create/bootstrap the domain. Each key is a server ID, with values such as that seen in Example 1 below. If one of the keys matches the value of ThisServerId, then this server can create/bootstrap the domain. If not, then it attempts to enter the domain indicated via a server ID in the initial membership. Alternatively, the peer property may be used to specify an entry peer. All members must have the same initial membership, unless peer is used to specify the entry peer.

The associated transport property is a host name or IP address and port number following the format ipaddr:port.

For more information, see Notes on Setting the peer and initialMembership Properties.

peer

An endpoint (host:port) through which new admin servers can join the membership. This can be the address of a specific admin server in the membership, or a load-balancer that is routing traffic to existing admin servers.

For more information, see Notes on Setting the peer and initialMembership Properties.

ThisServerId

A unique identifier for this admin service/process, for example server0.

Used by TEs to communicate with the admin processes, and used by SQL clients to obtain a SQL connection.

agentPort The port used by SQL clients to receive a connection from an admin process. It is used by database engine processes to communicate with an admin process. Default value is 48004.
adminPort The port which admin processes use to communicate with each other. Default value is 48005.
portRange The port range or the first port that engines should listen on. For each additional engine per host, a unique port number is assigned within the range. For example, a range can be specified using, 48006, 48010. Alternatively, 48006 can be specified as a starting port. Default value is 48006 (indicating that the port range starts at 48006).
altAddr The address which the admin process stores in the raft membership, and the address that all other admin processes use to communicate with it. If it is not specified, an admin process stores its hostname in the durable membership (which other admin processes can then use to communicate with it).

When set, the value for altAddr also displays for the host machine value when running NuoDB Command's show domain command.

Note: If a host cannot be reached using the altAddr value, the server ID on that host shows as "Disconnected"

ssl Used to enable or disable SSL (TLS) domain connection and membership authentication. Default value is true. To disable TLS, set this property to false.
domainEntryClient.peerWaitTimeout The maximum amount of time to wait, for a request for a peer to join an admin domain to complete. Default is 60000 (60 seconds).
leaderAssignmentTimeout The maximum amount of time to wait for all initialized archives to be started when assigning storage group leaders for a database. Default is 10000 (10 seconds).
pendingProcessTimeout The maximum amount of time to wait for a TE to connect to the admin process after the TE is started. Default is 5000 (5 seconds).
pendingReconnectTimeout The maximum amount of time to wait for a TE to reconnect to the admin process after the admin process is restarted. Default is 10000 (10 seconds).
broadcastLatencyMicros The maximum period of time (microseconds) it would take the raft leader to send a heartbeat to all followers; the heartbeat period and election timeout range are both calculated as multiples of the broadcastLatencyMicros value, with the election timeout being an order of magnitude larger than the heartbeat period.

When working with large clusters (20 or more admin servers/processes), or high-latency environments, consider increasing the value set for this property.

The value set for broadcastLatencyMicros is an approximation of the time it would take for an admin process to communicate with all of its peers in the admin server membership. It is used to heuristically derive parameters for the Raft algorithm, specifically the raft.heartbeatPeriodMillis, raft.minElectionTimeoutMillis, and raft.maxElectionTimeoutMillis properties, which can also be specified directly. For example, if there are ten admin processes spread evenly across two regions, and the intra-region server-to-server latency is 500 microseconds and the inter-region server-to-server latency is 1500 microseconds, a reasonable setting for broadcastLatencyMicros would be 9500 (4 * 500 + 5 * 1500).

(raft.heartbeatPeriodMillis is the period at which the leader sends heartbeat messages to followers to assert its leadership status. raft.minElectionTimeoutMillis and raft.maxElectionTimeoutMillis specify the range of the randomly-generated timeout values used by followers to start an election in the absence of a leader.)

Notes on Setting the peer and initialMembership Properties

Physical and VMware deployments

So there is no restriction to the endpoints that a new server can join through (those configured using the initialMembership property), the peer property can be set to a load-balancer in front of the admin servers currently running. However, when setting peer, a disadvantage is that there is a race condition on who the initial member is, which could lead to different servers bootstrapping different domains. This is solved by setting both peer and initialMembership. If the domain has not been bootstrapped yet, then it can only be bootstrapped by a majority of the servers displayed in initialMembership. If a server is not displayed in initialMembership, then it must join through the peer endpoint.

OpenShift Deployments

In containerized environments, the docker entrypoint script to start the admin process injects values into nuoadmin.conf based on environment variables. The NUODB_DOMAIN_ENTRYPOINT variable becomes the peer and the NUODB_BOOTSTRAP_SERVERID variable becomes the lone member in the initial membership.

For more information, see About Admin Processes and Peering.

Example 1

The following nuoadmin.conf code sample provides settings for an admin service defined on host1 (a host server) that is also the initial member:

{
"initialMembership": {
"server0" : { "transport" : "host1:48005", "version" : "0:10000" }
},
"ThisServerId" : "server0",
"agentPort" : "48004",
"agentBindAddress" : "",
"adminPort" : "48005",
"portRange" : "48006",
"adminBindAddress" : "",
"altAddr" : "Host1",
...

Note: As this is the first host, the value of ThisServerId is equal to the value for initialMembership. The default value is server0; if this is to be used later on for multi-host setup, you must replace the value set for localhost with a host name that is connectable externally.

Note: Once the domain has been bootstrapped (service started), you can't change the value for initialMembership. In the event that you need to change this value, you must first remove the raftlog file (located in /var/opt/nuodb) and restart NuoDB Admin. This may be required in a scenario where you began with a single host configuration (using localhost), and later decided to change it to reflect a real host name.

Example 2

The use of the peer property enables late binding of the initial membership.

Note: peer and initialMembership are only used if there is no durable Raft state. If there is durable Raft state, the membership from the recovered Raft state is used.

The peer property specifies an address (host and port) to query for the initial membership to use when entering the domain. Unlike the addresses in the initial membership, there is no requirement that the address specified by peer actually belongs to a server in the membership, which means that it could actually be the address of a load balancer (for example Nginx or an Elastic Load Balancers) that is brokering the existing set of admin servers.

{
"peer": "Host1",
"ThisServerId" : "server0",
"agentPort" : "48004",
"agentBindAddress" : "",
"adminPort" : "48005",
"portRange" : "48006",
"adminBindAddress" : "",
"altAddr" : "Host1",
...