Handling Unreachable Processes

An unreachable broker or process continues to be recorded in the durable domain configuration. When the NuoDB Manager show domain summary command indicates that a broker is unreachable then any processes that communicate with the unavailable broker are also shown as being unreachable. Although they are unreachable, they might still be running even though they have no broker to communicate with. It is safe for these processes to continue running but it is not safe to start another process for the same database. You can start a database process only when the output of the show domain summary command indicates that all existing processes that serve that database are running. If you try to start a database process and all existing processes that serve that database are not running then NuoDB prevents creation of the new process to avoid running a database in split-brain mode.

As soon as you determine that there are database processes that have no broker to communicate with you should try to restart the broker on that host. If you can restart the broker then it automatically starts communicating with the formerly unreachable database processes. Invoking show domain summary should now show that those processes are running.

If you cannot restart the broker then you should not kill the database processes only because the local broker is down. The processes can continue to run and when a broker becomes available on the host then that broker will connect with those processes. However, if the processes are not running then you do need to remove the unreachable processes from the durable domain configuration. You do this by removing their host from the domain. See Removing an Unreachable Broker's Host from the Domain.