Skip to end of metadata
Go to start of metadata

 

Overview

A preview of the new NuoDB Automation Console feature is available in NuoDB 2.0.3. The Automation Console is a streamlined web interface for launching and monitoring databases and database resources in the NuoDB Domain. The automated process leverages predefined templates to start and scale a databases on one or more hosts. Templates can be used to automatically configure single host databases, minimally redundant ones, multi-host databases and geo-distributed deployments.

The Automation Console is a feature preview and should not be deployed in production environments.

 

Full instructions for using the Automation Console, as well as template definitions can be found below.

Users can leverage the Automation Console to start a database on a single host, a cluster, or in a geo-distributed setup, elastically scale up and manage the database.

All of these actions above can be completed on the fly and at the touch of a button through the use of database templates.

Provisioning Hosts for Automation

The NuoDB instance running the nuodb_system database is automatically provisioned for automation by the bootstrap flag of the run-auto-console script described below. Depending on what you are trying to accomplish, additional provisioning may be required. Provisioning usually takes the form of editing the default.properties, nuodb.config and webapp.properties files. The table below presents basic property settings and considerations for provisioning hosts for automation. For a complete list, see default.propertieswebapp.propertiesnuodb.config and the nuodb-rest-api.yml file.

 

Tables sections: default.properties | rest-api.yml | nuodb.config | webapp.properties

PropertyDescription
DEFAULT.PROPERTIES
broker=True on all hosts that will run connection brokers, false on all others.
domain=

Name of the domain to which this host belongs.

Must match the domain.admin property in webapp.properties

domainPassword=

The administrative password used by agents to setup and maintain the domain securely. In production environments, the default domain and domainPassword should be changed.

Must match the domain.password property in webapp.properties

peer=Peer brokers to one another (for failover and redundancy) and agents to brokers. In multi-region databases, agents should peer to brokers in their own region.
region=For geo-distributed or multi-region database. Specifies a region name for the host. All hosts in a region must share the same region name.
balancer=RegionBalancerEnables a policy where SQL clients are connected to TEs in this Broker's region.
enableAutomation=trueSpecifies that this host has automation enabled and should attempt to access the nuodb_system admin database.
automationTemplate=

If enableAutomationBootstrap is enabled, this template is used to enforce the NuoDB "nuodb_system" admin database.

Options are: Single Host, Minimally Redundant, Multi Host and Geo-distributed.

metricsPurgeAge =

When "enableAutomation" is set, prune metrics by age. Default is 4 hours: 4h

Supported units are d=day, h=hour, m=minute.

  
 
NUODB-REST-API.YML
usernameMust match the domain property in default.properties
passwordMust match the domainPassword property in default.properties
WEBAPP.PROPERTIES

Note

webapp.properties governs the NuoDB Console, which contains the NuoDB Admin Console and SQL Explorer. It becomes unusable if its domain and domain password properties diverge from those set in default.properties. The NuoDB Console provides the ability to explore the nuodb_system admin database through a GUI interface. The nuodb_system database is not visible in the NuoDB Automation Console GUI, but is accessible through the REST API an NuoSQL CLI.

domain.adminMust match the domain property in default.properties
domain.passwordMust match the domainPassword property in default.properties
NUODB.CONFIG

Configuration parameters in the nuodb.config file are passed to every SM and TE started on that host. All hosts in a domain with automationEnabled must pass identical start all TEs and SMs with the same parameters, archive and journal directories excepted.

Note

Journaling is disabled by default, and Commit Local is the default commit protocol. If these settings are not changed in the nuodb.config file, all TEs and SMs started by automation will be started with these default properties, viz., local commit and journaling disable.

journal enable

Required for all commit protocols other than local commit.

Journaling is disabled by default.

commit remote

Commits the transaction after an ACK message is received from any SM

commit regionCommits the transaction after an ACK message is received from one SM in the same region.
commit region <n>Commits the transaction after an ACK message is received from one SM each in <n> regions.

 

Launch the Automation Console and Bootstrap the nuodb_system Database

Database automation is managed by the nuodb_system administration database. The nuodb_system database can be enforced as according to any database template. The template used is configured in with the automationTemplate = flag in the default.properties file. 

The nuodb_system database is created when the run-auto-console script is run with the --bootstrap flag. The --bootstrap flag checks if the broker on this host is provisioned to enable and bootstrap automation. If it has, then the Automation Console is started. If it hasn't, the broker is stopped, automation properties are updated to enable it, and broker is and restarted for the changes to take effect. This creates  the "nuodb_system" as a managed database.

It is important to not bootstrap multiple Brokers that have not been peered yet. This will produce multiple isolated nuodb_system databases in the same domain. When those brokers are subsequently peered to one another, automation will malfunction due to the multiple nuodb_system (automation management) databases existing in the same domain. For automation to function properly, there can only be one nuodb_system database in a domain.


The nuodb_system database is not reported as a in the Automation Console, but it can be viewed (and managed) from the NuoDB Console.   

The --bootstrap flag will be ignored if the nuodb_system database already exists in the domain, and the script will start the Automation Console.

In NuoDB 2.0 nuodb_system database in enforced as a Single Host database description. A Single Host database template has only 1 Transaction Engine and 1 Storage Manager.

With NuoDB 2.0.2 and later, the nuodb_system database may use any database template available in the Automation Console, and is specified with the automationTemplate flag in the default.properties file.

 

  1. Launch the NuoDB Automation Console and bootstrap the nuodb_system database:
    The run-auto-console script is located in the /bin directory of the NuoDB install directory. If this path is not already set, see Environment Path for instructions do to so.
    If you are launching the Automation Console from a NuoDB instance that is not hosting the nuodb_system database, skip to step 2. 

  2. Point a browser tab to http://localhost:8888.
    The NuoDB Automation Console login screen appears.


  3. Login using the default credentials: domain/bird.

    To change the Automation Console default credentials, edit the username and password in /opt/nuodb/etc/nuodb-rest-api.yml.
    You can also update nuodb-rest-api.yml with NuoDB domain credentials.
    The nuodb-rest-api.yml file contains parameters for both NuoDB and HTTP options. 

     

Administration Database

The nuodb_system database provides administration functions for the automated database features within NuoDB. As noted above, this database is created when the NuoDB Automation Console script is executed the first time it is run with the --bootstrap flag. If you set up automation in a multi-region, multi-host scenario, and if you ran two brokers at the same time with enableAutomation and enableAutomaticBootstrap and they are not peered, both will provision their own nuodb_system database in isolation of each other, which will cause multi-region, multi-host automation to fail.

Therefore, in multi-region setups, it is strongly recommended to install your first region with one Broker where enableAutomation and enableAutomationBootstrap are set to true (done in that Host's default.properties file or by running the run-auto-console.sh script on that machine; and then on all other Brokers/Agents in the domain, configure enableAutomation=true and enableAutomationBootstrap=false.

The nuodb_system database, can be on any Host whether there is an Agent or Broker running on it. The Agent/Broker writes metrics and  enables database description enforcement.

Metrics

All metrics captured by the management layer for each Transaction Engine, Storage Manager and Host machine OS in the domain are captured by the nuodb_system database. Currently, only commits (as TPS) are reported through the UI. In future releases all metrics will become available through the Automation Console, and will add language to express enforcement based on statistics/metrics, e.g., some threshold is reached.

To get an idea of what metrics are being captured for TEs and SMs and what will become available within the Automation Console, run:

For operating systems, nuodb_system stores CPU utilization (user, system, idle, wait), memory used/free, swap activity, number of processes/threads, network activity etc. Host-level metrics are valuable for developing rule-based enforcements. 

Additionally, access to additional TE/SM and OS statistics will become available not only to the Automation Console, but also thru a REST API and will integrate with 3rd party monitoring and graphing software like StatsD, Graphite, JMX and others. Statistics that are now collected and stored in the the nuodb_system administration database will be pushed to monitoring software in a future release.

 

Upgrading Automation from 2.0

During a multi-SM start-up (including cold restarts) the Brokers and Storage Managers in the domain check every host's archive directory to ensure that the UUID matches, and then determines which SM to start first. In NuoDB 2.0.1 and later, the UUID is stored in the /etc/nuodb/cfg/Description file. However, NuoDB 2.0 does not store the archive directory UUID there. Before upgrading it is necessary to first determine the UUID of the Storage Manager's archive, and then to add the "DatabaseUUID" attribute to the Broker configuration. This process precludes doing a rolling upgrade.

The following upgrade procedure is only required if you need to preserve data in any of the databases managed by automation, including nuodb_system. If don't need to preserve any of this data, then simply:

  • Shut everything down
  • Delete the old archives: rm -rf /var/opt/nuodb/production-archives/<name of database>_*
  • Delete the database Description file: /etc/nuodb/cfg/Description
  • Install the new binaries

To upgrade Automation from 2.0:

The following instructions assume NuoDB 2.0 is running on all machines.

  1. A new installation will overwrite the default.properties, webapp.properties, nuodb.config and nuodb-type.config files. If those have been edited, backup each configuration file on all Hosts and then after installing the new binaries (Steps 6-7 below) replace the configuration files with these backups.

  2. Pick one of the Broker Hosts in the domain and locate and record the DatabaseUUID for use later:

    which will return something like:

  3. From the Automation Console, Stop the database.

  4. Shutdown all Broker/Agents in the domain. See Starting and Stopping NuoDB Processes.

  5. On the Broker Host selected in step 1, add the DatabaseUUID to the Description file, and then restart the Broker on that machine (see Starting and Stopping NuoDB Processes).

  6. Install the NuoDB 2.0.1 or later binaries on the the Broker Host with the updated DatabaseUUID Description.
  7. Upgrade the remaining hosts in the database topology, and run the "run-auto-console" script to restart automation.

 

Templates

A template is a blueprint or a set of rules that describe the database to be created. When you create a database that is managed from a template, NuoDB instantiates a database description from one of the template chosen in the Create Database dialog. Four database templates are available:

Single Host

Starts one Storage Manager and one Transaction Engine on a host that you specify. These processes will be fixed to that host and will not be moved if that host goes offline.

Minimally Redundant

This template requires a minimum of two hosts, but may use up to four. The Minimally Redundant template starts two Storage Managers and two Transactions Engines. Each host will have at most one SM and one TE.

Multi Host

Starts up to two Storage Managers and as many Transaction Engines as possible. Each host will have at most one SM and one TE, so this template can be used with any number of hosts in a NuoDB Domain. A process (a TE or SM) will be restarted on a new host if one goes offline and another is available.

Geo-distributed

Starts one Storage Manager and as many Transaction Engines as possible in every region. Each host will have at most one SM and one TE. The Geo-distributed template can be used with any number of hosts. A process will restart on a new host if one goes offline and another is available.

 

Future releases will contain the ability to customize existing and create new templates. In future releases templates will be able to express rules based on "tags", such as "operating system", "host" etc., which would make it possible to write rules based on an Amazon EC2 tag key/value.

Managing Databases

Tools for managing databases in the Automation Console are accessed through the Database link on the left navigation bar.

Only databases created via the Automation Console can be managed in the through the Automation Console toolset.

 

The database page lists databases running and displays a live graph of the transactions per second. The database management toolbar (see the screenshot below) provides a set of tools with with to manage the database:

  • Delete—permanently deletes all data of the database from NuoDB. This is a non-recoverable event.
  • Edit—opens the Edit Database dialog, enabling the database template and host to change.
  • Stop—brings down the database; button toggles to Start
  • Start—restarts a stopped database; button toggles to Stop. See also Safe Database Restart below.
  • Quience—puts the database in a standby mode, which makes it accessible; button toggles to Unquience.
  • Unquience—returns the quiescent database to an active state; button toggles to Quience.

 

Safe Database Restart

Databases with multiple NuoDB Storage Managers cannot be restarted through the Automation Console. The reason is to prevent a situation whereby the wrong SM is started, thus leading to a loss of data. Databases with multiple SMs must be manually restarted, or with the file created by the database capture command (see NuoDB Manager-Restart).

Viewing Process Details

The processes in a NuodDB Database are its Transaction Engines and Storage Managers.

To access and view the details of database processes:

  1. Click the database/template icon under the Databases Transactions per Second graph.

    The Database Processes screen appears.
     
  2.  To view the details of any particular process, click anywhere in the process' ID table.


    The Process Details page opens.
  3. The Process Details page is divided into four sections:
    • UID + PID number 
    • Memory Manager
    • Record Manager
    • Configuration

Scale Out Automation Example

This example will walk through the process of scaling out a Single Host database to a Multi-Host one, and finally to a Geo-distributed one. The domain topology in this example has three regions with eight hosts in each for a total of 24 hosts. The example utilizes data and transactions generated from the Storefront Demo web application that ships with NuoDB. The following steps are for illustrative purposes only.

  1. From the NuoDB Automation Console, click Launch DB Instance button.

    The Create Database dialog appears.
  2. Choose Single Host from the Templates drop-down list, fill-in the form and click Submit.


    The Blackbirds database is provisioned and launches on a single host.
  3. Click Databases in the left navigation bar.
    The database metrics screen appears.

     

    Notice that although the database is on a single host with two processes running (in this case a single TE and SM each), three regions are showing. The database is not running on three regions. Rather, the number of regions is an aggregate of what is available in the NuoDB domain.

  4. Next we'll scale the database out to all the hosts in a specific region.
    Click the Actions drop-down and select Edit.
    In the Edit Database dialog, choose the Multi Host template, choose the Region where the database will run, and click Submit.


    The Blackbirds database automatically scales to eight hosts with a Transaction Engine started on every host in the region and a total of two Storage Managers for a total of 10 processes.


  5. Finally, we'll scale the database out to all hosts across all geo-distributed regions.
    Click the Actions drop-down and select Edit.
    In the Edit Database dialog, choose the Geo-distributed template and click Submit.


    The database automatically scales to all hosts in all regions. In this case, two regions containing 16 additional hosts are added to the database. A Transaction Engine is started on each of these and two additional Storage Managers, one for each additional region, are added to the 10 processes running from the database's previous incarnation as a Multi Host database (previous step).