A preview of the new NuoDB Automation Console feature is available in NuoDB 2.0.3. The Automation Console is a streamlined web interface for launching and monitoring databases and database resources in the NuoDB Domain. The automated process leverages predefined templates to start and scale a databases on one or more hosts. Templates can be used to automatically configure single host databases, minimally redundant ones, multi-host databases and geo-distributed deployments.
Full instructions for using the Automation Console, as well as template definitions can be found below.
Users can leverage the Automation Console to start a database on a single host, a cluster, or in a geo-distributed setup, elastically scale up and manage the database.
All of these actions above can be completed on the fly and at the touch of a button through the use of database templates.
Provisioning Hosts for Automation
The NuoDB instance running the nuodb_system database is automatically provisioned for automation by the bootstrap flag of the run-auto-console script described below. Depending on what you are trying to accomplish, additional provisioning may be required. Provisioning usually takes the form of editing the default.properties, nuodb.config and webapp.properties files. The table below presents basic property settings and considerations for provisioning hosts for automation. For a complete list, see default.properties, webapp.properties, nuodb.config and the nuodb-rest-api.yml file.
Name of the domain to which this host belongs.
Must match the domain.admin property in webapp.properties
The administrative password used by agents to setup and maintain the domain securely. In production environments, the default domain and domainPassword should be changed.
Must match the domain.password property in webapp.properties
|Peer brokers to one another (for failover and redundancy) and agents to brokers. In multi-region databases, agents should peer to brokers in their own region.|
|For geo-distributed or multi-region database. Specifies a region name for the host. All hosts in a region must share the same region name.|
|Enables a policy where SQL clients are connected to TEs in this Broker's region.|
|Specifies that this host has automation enabled and should attempt to access the nuodb_system admin database.|
If enableAutomationBootstrap is enabled, this template is used to enforce the NuoDB "nuodb_system" admin database.
Options are: Single Host, Minimally Redundant, Multi Host and Geo-distributed.
When "enableAutomation" is set, prune metrics by age. Default is 4 hours: 4h
Supported units are d=day, h=hour, m=minute.
|Must match the domain property in default.properties|
|Must match the domainPassword property in default.properties|
|domain.admin||Must match the domain property in default.properties|
|domain.password||Must match the domainPassword property in default.properties|
Configuration parameters in the nuodb.config file are passed to every SM and TE started on that host. All hosts in a domain with automationEnabled must pass identical start all TEs and SMs with the same parameters, archive and journal directories excepted.
Required for all commit protocols other than local commit.
Journaling is disabled by default.
Commits the transaction after an ACK message is received from any SM
|Commits the transaction after an ACK message is received from one SM in the same region.|
|Commits the transaction after an ACK message is received from one SM each in <n> regions.|
Launch the Automation Console and Bootstrap the nuodb_system Database
Database automation is managed by the nuodb_system administration database. The nuodb_system database can be enforced as according to any database template. The template used is configured in with the
automationTemplate = flag in the default.properties file.
The nuodb_system database is created when the run-auto-console script is run with the
--bootstrap flag. The
--bootstrap flag checks if the broker on this host is provisioned to enable and bootstrap automation. If it has, then the Automation Console is started. If it hasn't, the broker is stopped, automation properties are updated to enable it, and broker is and restarted for the changes to take effect. This creates the "nuodb_system" as a managed database.
The nuodb_system database is not reported as a in the Automation Console, but it can be viewed (and managed) from the NuoDB Console.
--bootstrap flag will be ignored if the nuodb_system database already exists in the domain, and the script will start the Automation Console.
Launch the NuoDB Automation Console and bootstrap the nuodb_system database:
The run-auto-console script is located in the /bin directory of the NuoDB install directory. If this path is not already set, see Environment Path for instructions do to so.
If you are launching the Automation Console from a NuoDB instance that is not hosting the nuodb_system database, skip to step 2.
- Point a browser tab to http://localhost:8888.
The NuoDB Automation Console login screen appears.
Login using the default credentials: domain/bird.
To change the Automation Console default credentials, edit the username and password in /opt/nuodb/etc/nuodb-rest-api.yml.
You can also update nuodb-rest-api.yml with NuoDB domain credentials.
The nuodb-rest-api.yml file contains parameters for both NuoDB and HTTP options.
The nuodb_system database provides administration functions for the automated database features within NuoDB. As noted above, this database is created when the NuoDB Automation Console script is executed the first time it is run with the
--bootstrap flag. If you set up automation in a multi-region, multi-host scenario, and if you ran two brokers at the same time with
enableAutomaticBootstrap and they are not peered, both will provision their own nuodb_system database in isolation of each other, which will cause multi-region, multi-host automation to fail.
The nuodb_system database, can be on any Host whether there is an Agent or Broker running on it. The Agent/Broker writes metrics and enables database description enforcement.
All metrics captured by the management layer for each Transaction Engine, Storage Manager and Host machine OS in the domain are captured by the nuodb_system database. Currently, only commits (as TPS) are reported through the UI. In future releases all metrics will become available through the Automation Console, and will add language to express enforcement based on statistics/metrics, e.g., some threshold is reached.
To get an idea of what metrics are being captured for TEs and SMs and what will become available within the Automation Console, run:
For operating systems, nuodb_system stores CPU utilization (user, system, idle, wait), memory used/free, swap activity, number of processes/threads, network activity etc. Host-level metrics are valuable for developing rule-based enforcements.
Additionally, access to additional TE/SM and OS statistics will become available not only to the Automation Console, but also thru a REST API and will integrate with 3rd party monitoring and graphing software like StatsD, Graphite, JMX and others. Statistics that are now collected and stored in the the nuodb_system administration database will be pushed to monitoring software in a future release.
Upgrading Automation from 2.0
During a multi-SM start-up (including cold restarts) the Brokers and Storage Managers in the domain check every host's archive directory to ensure that the UUID matches, and then determines which SM to start first. In NuoDB 2.0.1 and later, the UUID is stored in the /etc/nuodb/cfg/Description file. However, NuoDB 2.0 does not store the archive directory UUID there. Before upgrading it is necessary to first determine the UUID of the Storage Manager's archive, and then to add the "DatabaseUUID" attribute to the Broker configuration. This process precludes doing a rolling upgrade.
To upgrade Automation from 2.0:
The following instructions assume NuoDB 2.0 is running on all machines.
A new installation will overwrite the default.properties, webapp.properties, nuodb.config and nuodb-type.config files. If those have been edited, backup each configuration file on all Hosts and then after installing the new binaries (Steps 6-7 below) replace the configuration files with these backups.
Pick one of the Broker Hosts in the domain and locate and record the DatabaseUUID for use later:
which will return something like:
From the Automation Console, Stop the database.
Shutdown all Broker/Agents in the domain. See Starting and Stopping NuoDB Processes.
On the Broker Host selected in step 1, add the DatabaseUUID to the Description file, and then restart the Broker on that machine (see Starting and Stopping NuoDB Processes).
- Install the NuoDB 2.0.1 or later binaries on the the Broker Host with the updated DatabaseUUID Description.
Upgrade the remaining hosts in the database topology, and run the "run-auto-console" script to restart automation.
A template is a blueprint or a set of rules that describe the database to be created. When you create a database that is managed from a template, NuoDB instantiates a database description from one of the template chosen in the Create Database dialog. Four database templates are available:
Starts one Storage Manager and one Transaction Engine on a host that you specify. These processes will be fixed to that host and will not be moved if that host goes offline.
This template requires a minimum of two hosts, but may use up to four. The Minimally Redundant template starts two Storage Managers and two Transactions Engines. Each host will have at most one SM and one TE.
Starts up to two Storage Managers and as many Transaction Engines as possible. Each host will have at most one SM and one TE, so this template can be used with any number of hosts in a NuoDB Domain. A process (a TE or SM) will be restarted on a new host if one goes offline and another is available.
Starts one Storage Manager and as many Transaction Engines as possible in every region. Each host will have at most one SM and one TE. The Geo-distributed template can be used with any number of hosts. A process will restart on a new host if one goes offline and another is available.
Tools for managing databases in the Automation Console are accessed through the Database link on the left navigation bar.
The database page lists databases running and displays a live graph of the transactions per second. The database management toolbar (see the screenshot below) provides a set of tools with with to manage the database:
- Delete—permanently deletes all data of the database from NuoDB. This is a non-recoverable event.
- Edit—opens the Edit Database dialog, enabling the database template and host to change.
- Stop—brings down the database; button toggles to Start
- Start—restarts a stopped database; button toggles to Stop. See also Safe Database Restart below.
- Quience—puts the database in a standby mode, which makes it accessible; button toggles to Unquience.
- Unquience—returns the quiescent database to an active state; button toggles to Quience.
Safe Database Restart
Databases with multiple NuoDB Storage Managers cannot be restarted through the Automation Console. The reason is to prevent a situation whereby the wrong SM is started, thus leading to a loss of data. Databases with multiple SMs must be manually restarted, or with the file created by the database capture command (see NuoDB Manager-Restart).
Viewing Process Details
The processes in a NuodDB Database are its Transaction Engines and Storage Managers.
To access and view the details of database processes:
- Click the database/template icon under the Databases Transactions per Second graph.
The Database Processes screen appears.
- To view the details of any particular process, click anywhere in the process' ID table.
The Process Details page opens.
- The Process Details page is divided into four sections:
- UID + PID number
- Memory Manager
- Record Manager
Scale Out Automation Example
This example will walk through the process of scaling out a Single Host database to a Multi-Host one, and finally to a Geo-distributed one. The domain topology in this example has three regions with eight hosts in each for a total of 24 hosts. The example utilizes data and transactions generated from the Storefront Demo web application that ships with NuoDB. The following steps are for illustrative purposes only.
- From the NuoDB Automation Console, click Launch DB Instance button.
The Create Database dialog appears.
- Choose Single Host from the Templates drop-down list, fill-in the form and click Submit.
The Blackbirds database is provisioned and launches on a single host.
- Click Databases in the left navigation bar.
The database metrics screen appears.
- Next we'll scale the database out to all the hosts in a specific region.
Click the Actions drop-down and select Edit.
In the Edit Database dialog, choose the Multi Host template, choose the Region where the database will run, and click Submit.
The Blackbirds database automatically scales to eight hosts with a Transaction Engine started on every host in the region and a total of two Storage Managers for a total of 10 processes.
- Finally, we'll scale the database out to all hosts across all geo-distributed regions.
Click the Actions drop-down and select Edit.
In the Edit Database dialog, choose the Geo-distributed template and click Submit.
The database automatically scales to all hosts in all regions. In this case, two regions containing 16 additional hosts are added to the database. A Transaction Engine is started on each of these and two additional Storage Managers, one for each additional region, are added to the 10 processes running from the database's previous incarnation as a Multi Host database (previous step).