Host Properties (nuoadmin.conf)

To configure a host, set the required values for properties in nuoadmin.conf. You can find nuoadmin.conf in the following locations:

Installation type Location

RPM

/etc/nuodb

TAR

$NUODB_HOME/etc

Windows MSI

%NUODB_HOME%/etc

Setting nuoadmin.conf Properties

Using a text editor, define Admin Process (AP) settings using nuoadmin.conf properties. Some of the nuoadmin.conf properties are described in the below table. All properties are described in the configuration file shipped with the product or in the sample configuration file located at $NUODB_HOME/etc/nuoadmin.conf.sample.

Property Description Comments

initialMembership

A map of servers and details for APs that are allowed to create/bootstrap the domain. Each key is a server ID, with values such as that seen in Example 1 below. If one of the keys matches the value of ThisServerId, then this server can create/bootstrap the domain. If not, then it attempts to enter the domain indicated via a server ID in the initial membership. Alternatively, the peer property may be used to specify an entry peer. All members must have the same initial membership, unless peer is used to specify the entry peer.

The associated transport property is a host name or IP address and port number following the format <address>:<port>.

The initialMembership is immutable during the domain lifecycle. Changing it requires destroying the domain and bootstrapping a new one.

peer

An endpoint (host:port) through which new admin servers can join the membership. This can be the address of a specific admin server in the membership, or a load-balancer that is routing traffic to existing admin servers.

ThisServerId

A unique identifier for this AP that is permanently assigned, for example server0.

altAddr

The address which the AP stores in the Raft membership, and the address that all other APs use to communicate with it. If it is not specified, an AP stores its hostname in the durable membership (which other APs can then use to communicate with it).

When set, the value for altAddr also displays for the host machine value when running the nuocmd show domain command.

If an AP cannot be reached using the altAddr value, it shows as Disconnected when invoking nuocmd show domain.

agentPort

The port used by SQL clients to obtain a connection from an AP and also used by database processes to communicate with an AP.

Default is 48004.

adminPort

The port that APs use to communicate with each other.

Default is 48005.

rest.port

The port used by the REST API service. If rest is not specified in otherServices, then this property is ignored.

Default is 8888.

logging.defaultLogLevel

Default log level for the root logger, which is inherited by all other loggers that do not have log levels defined using logging.logLevels. Accepted values are all, trace, debug, info, warn, error, and off.

Default is info.

logging.logLevels

Log levels for not-root loggers, specified as a mapping of logger name, or prefix of logger name, to log level. For example, {"com.nuodb": "trace"} raises the verbosity of all classes/loggers in the NuoDB code-base to trace, while {"ProcessManager": "trace"} only raises the verbosity of the ProcessManager class/logger. NuoDB classes/loggers can be specified using either the fully-qualified class name (e.g. com.nuodb.host.ProcessManager) or the simple name (e.g. ProcessManager), while all non-NuoDB classes/loggers must be specified using the fully-qualified class name (e.g. org.eclipse.jetty.server.Server). Alternatively, log levels can be specified as a string rather than a map by using comma-delimited values of the form <logger name>:<level>, e.g. com.nuodb:debug,ProcessManager:trace.

Default is {} (no overrides for log levels).

logging.enableConsoleAppender

Used to enable or disable logging to standard output. This must be set to true for APs started in containerized deployments, such as Docker and Kubernetes environments, so that logging appears when running docker logs <AP_container_name> or kubectl logs <AP_pod_name>.

Default is false.

logging.consoleLogLevels

Log levels for standard output logging. This is specified in the same way logging.logLevels, but without affecting the verbosity of logging to $NUODB_LOGDIR/nuoadmin.log. This is useful whenever the user wants less logging to appear when running docker logs <AP_container_name> or kubectl logs <AP_pod_name>, and can only be used to lower the verbosity of logging with respect to logging.logLevels. If the user wants to raise the verbosity of logging, then the change should be made to logging.logLevels, which also affects the verbosity of logging to standard output.

Default is {} (no overrides for log levels).

portRange

The port range that database processes managed by the AP should listen on. A database process on a host will scan the port range until it finds a port that it can listen on. Can be an bounded range such as 48006,48010, or an unbounded range such as 48006, which only specifies the starting port.

Default value is 48006 (indicating that the port range starts at 48006).

ssl

Used to enable or disable SSL (TLS) domain connection and membership authentication.

Default value is true. To disable TLS, set this property to false.

authorizedDomains

If set to a non-empty value, the AP enforces hostname matching during the certificate verification process to restrict access to clients with non-root certificates. This process allows a public CA to be used as a trusted root without giving access to all clients with certificates signed by the same CA.

authorizedDomains is a comma-delimited list of hostnames that is compared to the Subject Alternative Name (SAN) extension and Common Name (CN) field during the client certificate verification process. The client is granted access if any of the hostnames have authorizedDomain as a suffix. Clients presenting trusted root certificates do not have to satisfy the restriction imposed by the authorizedDomains property.

Default value is the empty string (""), which indicates that no restriction should be enforced.

domainEntryClient.peerWaitTimeout

The maximum amount of time (in seconds) for a new AP to wait to join the domain.

Default is 60 (60 seconds).

leaderAssignmentTimeout

The maximum amount of time (in milliseconds) to wait for all initialized archives to be started when assigning storage group leaders for a database.

Default is 60000 (60 seconds).

pendingProcessTimeout

The maximum amount of time (in milliseconds) to wait for a database process to connect to the AP after it is started.

Default is 5000 (5 seconds).

pendingReconnectTimeout

The maximum amount of time (in milliseconds) to wait for a database process to reconnect to the AP after the AP is restarted.

Default is 10000 (10 seconds).

evictUnknownProcesses

Whether to force unknown database processes attempting to reconnect to the AP to shutdown. Note that this implies that the process either does not exist at all in the current domain (e.g. the domain was reprovisioned from scratch but the process was left running from the previous domain), or was explicitly removed from the domain state (e.g. using nuocmd shutdown process …​ --evict).

Default is false.

broadcastLatencyMicros

The maximum period of time (in microseconds) it would take the Raft leader to send a heartbeat to all followers; the heartbeat period and election timeout range are both calculated as multiples of the broadcastLatencyMicros value, with the election timeout being an order of magnitude larger than the heartbeat period.

When working with large clusters (20 or more APs), or high-latency environments, consider increasing the value set for this property.

The value set for broadcastLatencyMicros is an approximation of the time it would take for an AP to communicate with all of its peers in the admin server membership. It is used to heuristically derive parameters for the Raft algorithm, specifically the raft.heartbeatPeriodMillis, raft.minElectionTimeoutMillis, and raft.maxElectionTimeoutMillis properties, which can also be specified directly. For example, if there are ten APs spread evenly across two regions, and the intra-region server-to-server latency is 500 microseconds and the inter-region server-to-server latency is 1500 microseconds, a reasonable setting for broadcastLatencyMicros would be 9500 (4 * 500 + 5 * 1500).

(raft.heartbeatPeriodMillis is the period at which the leader sends heartbeat messages to followers to assert its leadership status. raft.minElectionTimeoutMillis and raft.maxElectionTimeoutMillis specify the range of the randomly-generated timeout values used by followers to start an election in the absence of a leader.)

tombstoneGarbageCollectionPeriodSec

Garbage collection of exited database processes (tombstones) prevents the domain state from growing too large. Garbage collection is performed periodically every tombstoneGarbageCollectionPeriodSec seconds.

Default is 30 (30 seconds).

tombstoneGarbageCollectionThreshold

Defines the number of tombstones at which garbage collection is triggered.

Default is 5000.

tombstonesToRetain

Defines the minimum number of tombstones to retain and should be lower than tombstoneGarbageCollectionThreshold.

Default is 4000.

incarnationsToRetain

Defines the number of database incarnations to retain when garbage collecting tombstones for processes that were gracefully shutdown. Processes that were gracefully shutdown more than incarnationsToRetain major incarnations in the past are removed first, and if this does not succeed in reducing the number of tombstones in the domain state below tombstoneGarbageCollectionThreshold, a second round of garbage collection is performed in which the oldest process tombstones are removed regardless of whether they were shutdown gracefully (tombstone age is defined as the difference between the current major incarnation for the database and the major incarnation in which the process exited).

Default is 10.

generatedKeyAlgorithm

If the Admin is TLS-enabled and has a certificate with signing rights (i.e. the certificate has the attribute BasicConstraint:CA=true), then it will generate a new key-pair and certificate for every database process that it starts, which it signs with its own key. generatedKeyAlgorithm controls the asymmetric encryption algorithm for the key-pair and can be either RSA or EC.

Default is RSA.

generatedKeyStrength

The strength of the generated key, which corresponds to a particular size in bits for each algorithm. The key strengths are WEAK, MEDIUM, STRONG, and VERY_STRONG. See the shipped nuoadmin.conf.sample for the corresponding key sizes for each algorithm type.

Default is MEDIUM.

licenseFilePath

The NuoDB license file location. A license file at the supplied path will be loaded if no other license has been installed using nuocmd set license. The path is relative to $NUODB_CFGDIR unless an absolute path is supplied.

Default is $NUODB_CFGDIR/nuodb.lic.

Notes on Setting the peer and initialMembership Properties

Physical and VMware deployments

So there is no restriction to the endpoints that a new server can join through, the peer property can be set to a load-balancer in front of the admin servers currently running. However, when setting peer, a disadvantage is that there is a race condition on who the initial member is, which could lead to different servers bootstrapping different domains. This is solved by setting both peer and initialMembership. If the domain has not been bootstrapped yet, then it can only be bootstrapped by a majority of the servers displayed in initialMembership. If a server is not displayed in initialMembership, then it must join through the peer endpoint.

For NuoDB Admin step by step configuration guidelines, see Configuring NuoDB Admin.

Containerized Deployments

In containerized environments, the container entrypoint script to start the AP injects values into nuoadmin.conf based on environment variables. The NUODB_DOMAIN_ENTRYPOINT variable becomes the peer and the NUODB_BOOTSTRAP_SERVERID variable becomes the lone member in the initial membership.

For more information, see Admin Process (AP).

Example 1

The following nuoadmin.conf code sample provides settings for an admin service defined on host server0 that is also the initial member:

    ...
    "initialMembership": {
        "server0" : { "transport" : "server0:48005", "version" : "0:10000" }
    },
    "ThisServerId": "server0",
    "altAddr": "server0",
    "adminPort": "48005",
    ...
As this is the first host, the value of ThisServerId appears in the initialMembership. The altAddr property is the address that is advertised by a particular AP. The default value of altAddr is $(hostname), which expands to the hostname of the server. The ThisServerId property is a user-defined unique identifier for an AP, and is typically chosen to match the altAddr, assuming that all hosts have stable and unique hostnames.
Once the domain has been bootstrapped (service started), you can’t change the value for initialMembership. In the event that you need to change this value, you must shutdown the nuoadmin service and remove the raftlog file for all APs in the domain (located in /var/opt/nuodb by default). This would result in the loss of any configuration data in the domain, including all database and archive metadata (the archive data itself is unaffected) and SQL load balancer configuration.

Example 2

The use of the peer property enables late binding of the initial membership.

peer and initialMembership are only used if there is no durable Raft state. If there is durable Raft state, the membership from the recovered Raft state is used.

The peer property specifies an address (host and port) to query for the initial membership to use when entering the domain. Unlike the addresses in the initial membership, there is no requirement that the address specified by peer actually belongs to a server in the membership, which means that it could actually be the address of a load balancer (for example Nginx or an Elastic Load Balancer) that is brokering the existing set of admin servers.

    ...
    "peer": "server0",
    "ThisServerId" : "server0",
    "altAddr" : "server0",
    ...

Expansion of Files and Variables

It is possible to specify configuration properties in nuoadmin.conf that are not bound until the AP is started.

The syntax ${VAR} can be used evaluate a JVM property or an environment variable. For example, the following entry can be used to specify the password used to protect an AP’s keystore:

    ...
    "keystore-password": "${NUODB_KEYSTORE_PASSWORD}",
    ...

This will evaluate to the JVM property NUODB_KEYSTORE_PASSWORD or the environment variable NUODB_KEYSTORE_PASSWORD. JVM properties take precedence over environment variables.

It is also possible to expand the contents of a file as a configuration property, using the syntax $(</path/to/file). For example, the following entry can be used to specify the password used to protect an AP’s keystore:

    ...
    "keystore-password": "$(</etc/nuodb/keys/keystore-password)",
    ...

This will expand the contents of /etc/nuodb/keys/keystore-password, removing any leading or trailing whitespace.

Variable and file expansion can be performed for the value of a key-value entry, but not the key. In other words, these expressions can only appear on right-hand side of a key-value entry.