Setting Up Automatic Restart of Processes

Databases are governed by process requirements as defined by the database template. A database must have a minimum number of transaction engine (TE) and storage manager (SM) processes. If more hosts are available, then the enforcerThe enforcer is a process that runs in each broker in the domain. The enforcer compares the state of the domain against the requirements for the database(s) or against the template(s) used to create the database(s). On only the leader broker, the enforcer starts additional processes as needed. will start more processes until the maximum number of required processes has been reached. Enforcement happens periodically, continually comparing the requirements of the templates against the actual information about database processes and hosts in the domain.

Automation enforcement has a mechanism to gracefully handle failure of the processes it tries to start. There are times when best practice will be to automatically restart the failed process and there will be other times when some intervention is required before restarting the failed process. For example, automation enforcement may cause a TE to be started and it fails with an assertion at which point enforcement will cause a TE to be started again that may just fail with the same assertion.

NuoDB automation retry and back-off is a mechanism to gracefully handle these types of situations. There are ways to govern how many times and how often automation enforcement will retry starting a new database process given an event that causes it to shutdown.