Logging to Standard Syslog Files

NuoDB brokers are configured, by default, to log messages to java.util.logging.ConsoleHandler. To log NuoDB messages to syslog files, you must implement your own Java utility logging (JUL) configuration file. Doing this overrides the default ConsoleHandler and the broker will log messages to only the handlers configured in the file named by the java.util.logging.config.file property. If you want NuoDB to continue to log to a console and also log to a syslog file then you must configure a console handler as well as a syslog handler.

If you implement a syslog handler without also implementing a console handler, then the information in the broker's log file (for example, agent.log on Linux) will be incomplete. The NuoDB Manager diagnose command will retrieve only the agent.log file and will not retrieve any broker log messages in the system log (files). Consequently, if is very important that you continue to use a console handler as well as a syslog handler.

To log NuoDB messages to a syslog file, the general steps are as follows. Details are in the sections after this list:

  1. Define the syslog handler in its own file or by updating the NuoDB host's default.properties file.
  2. Set the java.util.logging.config.file property to point to the file that defines your syslog handler.
  3. If you want transaction engines and storage managers to also send messages to syslog, set the remote-syslog and remote-syslog-server database options.
  4. Restarting Brokers.
  5. Restart any transaction engines and storage managers that you want to send messages to syslog.

If you want all hosts in your domain to log to syslog then you need to repeat these steps on each host.

Defining a Syslog Handler

To define a syslog handler, do one of the following:

Following is an example of creating a separate file that defines both a console handler and a syslog handler.

# JUL configuration file defining a ConsoleHandler and a SyslogHandler
# along with properties specific to each handler.
#
handlers= java.util.logging.ConsoleHandler, com.agafua.syslog.SyslogHandler

# Default logging level 
.level= INFO

# Console handler configuration
# SimpleFormatter is the default formatter but it is highly recommended
# that you use NuoLogFormatter for each Handler except SyslogHandler.
#
java.util.logging.ConsoleHandler.formatter = com.nuodb.impl.util.NuoLogFormatter

# Syslog handler
#
# The following settings are optional. Uncomment and change as needed.
#
# UDP is the default transport. TCP is also supported.
# 514 is the default port.
# USER is the default facility.
# mycentral-logger.mydomain.com is the default remote syslog hostname
# SimpleFormatter is the default formatter but it is highly recommended
# that you use NuoSyslogFormatter.
#
#com.agafua.syslog.SyslogHandler.transport = udp
#com.agafua.syslog.SyslogHandler.facility = user
#com.agafua.syslog.SyslogHandler.port = 514
#com.agafua.syslog.SyslogHandler.hostname = mycentral-logger.mydomain.com
com.agafua.syslog.SyslogHandler.formatter = com.nuodb.agent.NuoSyslogFormatter

Pointing to a Syslog Handler

After you define the syslog handler (either in its own configuration file or in the default.properties file), you must set the java.util.logging.config.file property to point to it. You do this in the same way that you would for any other JUL handler. For example, suppose you added the syslog handler properties to your $NUODB_HOME/etc/default.properties file. Add the following property setting to your /etc/nuodb/jvm-options file:

NUODB_AGENT_JAVA_OPTS="-Djava.util.logging.config.file=$NUODB_HOME/etc/default.properties"

See Java JVM Options (jvm-options).

Remember that changes to the logging configuration take effect only after you restart the broker.

Sending TE and SM Messages to a Syslog

Your syslog handler logs only broker messages to the syslog. To send TE and SM messages to a syslog, set the remote-syslog and remote-syslog-server database options. You can do this in either of the following ways:

Distinguishing Log Message Sources

If you choose to log broker, TE and SM messages to a central syslog, you can use message headers to distinguish the source of each message. A message coming from a broker uses the following format for its header:

timestamp source-hostname app[pid]: log-level log-message

A message that comes from a transaction engine or a storage manager uses the following format for its header:

timestamp source-hostname app[pid]: (database-name) {TE | SM} log-message

For example:

Mar  9 08:53:06 ip-172-31-14-171/52.24.143.83 nuoagent[5804]: INFO EventManager.notifyNodeIdSet\ 
   (serv-socket6-thread-2) Node setId=1 [Node SM db=[demosyslog1] pid=5841 id=-1 req=SMs (local)]\
Mar  9 08:53:06 ip-172-31-14-171/52.24.143.83 nuoagent[5804]: INFO EventManager.notifyNodeStateChange\ 
   (serv-socket6-thread-2) Node state changed to ACTIVE [Node SM db=[demosyslog1] pid=5841 id=1 \
   req=SMs (local)]\
Mar  9 08:53:06 ip-172-31-14-171/52.24.143.83 nuoagent[5804]: INFO EventManager.notifyNodeStateChange\
   (serv-socket6-thread-2) Node state changed to RUNNING [Node SM db=[demosyslog1] pid=5841 id=1\
   req=SMs (local)]\
Mar  9 08:53:06 ip-172-31-14-171/52.24.143.83l nuoagent[5804]: INFO Enforcer.enforceRemaining \  
   (asynch-dao5-thread-1) Starting ProcessSpec[demosyslog1, TE, req=TEs, \
   peer=[LocalPeer ip-172-31-14-171/52.24.143.83:48004 (broker)]]\
Mar  9 08:53:11 ip-172-31-14-171/52.24.143.83 nuodb[5856]: (demosyslog1) TE 
   Starting Transaction Engine 2.5.2.dev-9999-f24cc3edc3: database=demosyslog1,
   protocol=1.458752.f24cc3edc3 (2.5.2)
Mar  9 08:53:11 ip-172-31-14-171/52.24.143.83 nuodb[5856]: (demosyslog1) TE 
   Node::initStorageGroups. LocalNode. NodeId: 2147483647, Type: Transaction, Storage Groups: (1,Running,0),
Mar  9 08:53:11 ip-172-31-14-171/52.24.143.83 nuodb[5856]: (demosyslog1) TE 
   getnameinfo(10.8.100.2) => error 8: nodename nor servname provided, or not known in 0.001 seconds
Mar  9 08:53:11 ip-172-31-14-171/52.24.143.83 nuodb[5856]: (demosyslog1) TE 
   Failed to resolve network address: getnameinfo(10.8.100.2) error 8: nodename nor servname provided, or not known
Mar  9 08:53:11 ip-172-31-14-171/52.24.143.83 nuoagent[5804]: INFO EventManager.notifyNodeEvent\
    (serv-socket6-thread-3) Node joined: [Node TE db=[demosyslog1] pid=5856 id=-1 req=TEs (local)]\