Deprovisioning Brokers

Deprovisioning a broker means that you remove it from the durable domain configurationThe durable domain configuration provides domain configuration information that is stored consistently on each broker in the domain by means of a Raft log.. As with any change to the durable domain configuration, a quorumIn a domain, a quorum is the minimum number of brokers that must acknowledge a change to the durable domain configuration for that change to be made. Usually, this is a simple majority. Many operations in the management tier require acknowledgement from a majority of brokers in the domain in order to ensure safety (never returning an incorrect result) in case of failure. of brokers is required to deprovision a broker. This ensures safety in case of network partition or failure.

You use the NuoDB Manager command line tool to deprovision a broker. To deprovision a broker by using its ServerInfo ID or the broker stableId, see the NuoDB Manager agent deprovision command. Alternatively, you can use the NuoDB Manager shutdown host command and specify options that shut down the broker on that host and also remove that host from the domain.

After a graceful shut down (kill -TERM or service/launchctl stop), the broker's default behavior is to continue to be in the durable domain configuration. The default behavior is determined by the removeSelfOnShutdown host property, which is set to false by default. For production systems, the recommendation is to keep the removeSelfOnShutdown property set to false. That is, you want to avoid parallel shut down of all or most brokers, which can happen when removeSelfOnShutdown is true.

The advantage of turning this property on is that it allows graceful expansion/contraction of the domain. The intended use case for setting removeSelfOnShutdown to true is the ability to deprovision one region, even if, for example, you run with only two regions and an equal number of brokers per region. This property would enable gracefully shutting down one broker at a time in the region to be removed. With three regions, you can shut down all brokers in one region at the same time because the remaining two regions will provide the quorum. For more details, see About Broker Quorum.

When removeSelfOnShutdown is set to true you should avoid stopping all brokers in the domain at the same time because of the following: