Removing an Unreachable Broker's Host from the Domain

Suppose that a broker becomes unreachable and cannot be restarted. This can happen, for example, if the broker host terminates. This situation prevents you from starting any processes for any databases that had processes running on that host. Consequently, if a broker is unreachable and cannot be restarted then you must remove the host of the unreachable broker from the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each broker in the domain by means of a Raft log..

The steps for doing this vary according to whether or not the domain has a broker quorum when it cannot reach one of the brokers. If the domain does not have a broker quorum, see Re-establishing Broker Quorum. If the domain has a broker quorum, the steps for removing the unreachable broker's host from the durable domain configuration are as follows:

  1. Obtain the stable ID of the unreachable broker's host by invoking the NuoDB Manager show domain serverConfig command.
  2. Invoke the NuoDB Manager agent deprovision command. This command removes the unreachable broker from the durable domain configuration. For more information, see Restarting Brokers and Agent Deprovision. For example:

    nuodb [myDomain] > agent deprovision stableId uuid:913cefd3-13dd-484d-ab2d-ee9b92627f54
  3. Invoke the remove host command to instruct transaction engines to hang up any sockets (close any connections) with transaction engines that serve the specified database on the unreachable broker's host.

    This step is necessary only when you are not using the --ping-timeout database option. See Setting the ping-timeout Database Option. Setting that option hangs up sockets with non-responsive processes automatically.

    Perform this step for each database that had processes running on the unreachable broker's host. For example:

    nuodb [myDomain] > remove host address ip-172-31-5-193:48004 database mydb
  4. Invoke the domainstate removehostprocesses command. This command removes from the durable domain configuration any processes that are or have been running on the unreachable broker's host. Specify the stable ID of the unreachable broker's host. For example:

    nuodb [myDomain] > domainstate removehostprocesses id 
       uuid:1bd3e439-7c48-4e69-8361-0dcf6cfe87c1
  5. Invoke the remove host archiveLocations command. This command updates the durable domain configuration. For the databases that were running on the unreachable broker's host, this command removes their archive locations on the unreachable broker's host. This is required to be able to restart the database. A cold restart of a managed database must be able to reach all archive locations recorded in the durable domain configuration. Specify the stable ID of the unreachable broker's host. For example:

    nuodb [myDomain] > remove host archiveLocations stableId 
       uuid:1bd3e439-7c48-4e69-8361-0dcf6cfe87c1
  6. Invoke the NuoDB Manager agent unreachable command to inform the connected brokers to ignore the unreachable broker. For example:

    nuodb [myDomain] > agent unreachable host nameOfLostBrokerHost:1234