Removing an Unreachable Admin Process Host from the Domain

Suppose that a NuoDB Admin process becomes unreachable and cannot be restarted. This can happen, for example, if the admin process host terminates. This situation prevents you from starting any processes for any databases that had processes running on that host. Consequently, if an admin process is unreachable and cannot be restarted then you must remove the host of the unreachable admin process from the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each NuoDB Admin process in the domain by means of a Raft log..

The steps for doing this vary according to whether or not the domain has an admin process quorum when it cannot reach one of the admin processes. If the domain does not have an admin process quorum, see Re-establishing Admin Process Quorum. If the domain has an admin process quorum, the steps for removing the unreachable admin process host from the durable domain configuration are as follows:

  1. Obtain the stable ID of the unreachable admin process host by invoking NuoDB Manager's show domain serverConfig command.
  2. Invoke NuoDB Manager's agent deprovision command. This command removes the unreachable admin process from the durable domain configuration. For more information, see Restarting Brokers and Agent Deprovision. For example:

    nuodb [myDomain] > agent deprovision stableId uuid:913cefd3-13dd-484d-ab2d-ee9b92627f54
  3. Invoke the remove host command to instruct transaction engines to hang up any sockets (close any connections) with transaction engines that serve the specified database on the unreachable admin process host.

    This step is necessary only when you are not using the --ping-timeout database option. See Engine-level failure detection. Setting that option hangs up sockets with non-responsive processes automatically.

    Perform this step for each database that had processes running on the unreachable admin process host. For example:

    nuodb [myDomain] > remove host address ip-172-31-5-193:48004 database mydb
  4. Invoke the domainstate removehostprocesses command. This command removes from the durable domain configuration any processes that are or have been running on the unreachable admin process host. Specify the stable ID of the unreachable admin process host. For example:

    nuodb [myDomain] > domainstate removehostprocesses id 
       uuid:1bd3e439-7c48-4e69-8361-0dcf6cfe87c1
  5. Invoke the remove host archiveLocations command. This command updates the durable domain configuration. For the databases that were running on the unreachable admin process host, this command removes their archive locations on the unreachable admin process host. This is required to be able to restart the database. A cold restart of a managed database must be able to reach all archive locations recorded in the durable domain configuration. Specify the stable ID of the unreachable admin process host. For example:

    nuodb [myDomain] > remove host archiveLocations stableId 
       uuid:1bd3e439-7c48-4e69-8361-0dcf6cfe87c1
  6. Invoke the NuoDB Manager agent unreachable command to inform the connected admin processes to ignore the unreachable admin process . For example:

    nuodb [myDomain] > agent unreachable host nameOfLostBrokerHost:1234