Metrics Published by Database Processes

The table below lists all metrics that a NuoDB database publishes. Each metric is produced by each database process (transaction engines, storage managers, and snapshot storage managers). Each metric value is for only the database process that generates it.

These metrics are available to subscribers, which can be any NuoDB management client or the NuoDB Manager command line tool. See Obtaining Metrics Using NuoDB Manager Monitors.

All metrics are produced every 10 seconds. The exact interval is provided by the Milliseconds metric.

If a metric value has not changed since the last time metrics were provided then that metric is omitted from the metric values provided to subscribers. Consequently, metrics that are not relevant for a process type are provided the first time that process provides metrics and then those metrics are no longer provided by that process. For eample, consider the ArchiveQueue metric, which is relevant for only storage managers or snapshot storage managers. The first time that a transaction engine generates metrics, it provides a 0 for the ArchiveQueue metric. For transaction engines, the value of this metric never changes so after the first time a transaction engine provides metrics, it no longer generates a value for the ArchiveQueue metric.

While some of the metrics in this table are internal only, you might need to provide this information if you are working with NuoDB customer support.

See also: Obtaining Operation Metrics and Obtaining Metrics for the Domain, Hosts and Processes.

Metric Name

Description Relevant For
ActiveTime Milliseconds of active thread time since the last time metrics were provided. This value divided by the elapsed time equals the average number of active threads.  

ActualVersion

Internal NuoDB version number. It never changes for the life of the process.  

AdminReceived

Number of administrative service requests since the last time metrics were provided.  

AdminSent

Number of administrative service responses since the last time metrics were provided. This includes log and metrics messages.  
ArchiveBandwidthThrottleTime Milliseconds spent in archive I/O throttle since the last time metrics were provided. I/O throttle limits disk bandwidth consumption for writes, to leave some bandwidth for reads.  

ArchiveBufferedBytes

While saving atomThe internal object structure representing all data in a NuoDB database. Atoms are self-coordinating objects that represent specific types of information (such as data, indexes or schemas).s, journaling can fall behind. On the second attempt to save an atom, a snapshot of the context is saved, the atom marked as dirty, and the atom is placed back on the archive queue. ArchiveBufferedBytes is the number of bytes saved in the snapshots since the last time metrics were provided.  

ArchiveDirectory

Archive directory. SM or SSM only. This value never changes for the life of the process. SM/SSM only
ArchiveDirectoryTime Milliseconds spent in archive directory operations since the last time metrics were provided. SM/SSM only
ArchiveFsyncTime Milliseconds spent finishing archive writes since the last time metrics were provided. SM/SSM only

ArchiveQueue

At the moment that metrics are provided, this is the number of atoms queued and waiting to be written to disk by the SM or SSM. SM/SSM only
ArchiveReadTime Milliseconds spent in archive read since the last time metrics were provided. SM/SSM only
ArchiveSyncThrottleTime Milliseconds spent in archive synchronization throttle since the last time metrics were provided. Archive synchronization throttle slows the process down to decrease interference with other work. SM/SSM only
ArchiveWriteTime Milliseconds spent in archive write since the last time metrics were provided. SM/SSM only
AtomProcessingThreadBacklog Milliseconds spent waiting to allocate an atom processing thread since the last time metrics were provided.  

BroadcastTime

Milliseconds spent broadcasting messages to other database processes since the last time metrics were provided.  

BytesBuffered

Number of bytes buffered to be sent over sockets since the last time metrics were provided.  

BytesReceived

Number of bytes received by the sockets since the last time metrics were provided.  

BytesSent

Number of bytes sent over the sockets since the last time metrics were provided.  

ClientCncts

Number of SQL client socket connections at the moment this metric is provided.  

ClientReceived

Number of messages received from the SQL client since the last time metrics were provided.  

ClientSent

Number of responses sent to the SQL client since the last time metrics were provided.  
ClientThreadBacklog Milliseconds spent waiting to allocate a client thread since the last time metrics were provided.  

Commits

Number of transaction commits since the last time metrics were provided. TEs Only
CreatePlatformRecordsTime Milliseconds spent waiting for the chairman to create a Records atom since the last time metrics were provided.  

CurrentActiveTransactions

Number of active transactions at the moment this metric is provided.  

CurrentCommittedTransactions

Number of committed transactions since the last time metrics were provided.  

CurrentPurgedTransactions

Number of transactions that have been placed on the purgatory list since the last time metrics were provided. A transaction being deleted is placed on the purgatory list to give other threads a chance to remove references to that transaction.  

Deletes

Number of records deleted since the last time metrics were provided. TEs Only
DependentCommitWaits To ensure that there is always a consistent view of the database, NuoDB makes committed transactions visible to transactions on a particular transaction engine in the correct order. At the moment this metric is provided, this is the number of commits that have been received from other transaction engines and that cannot yet be visible to transactions on this TE. These commits cannot be visible yet because they depend on commits that have not yet arrived and that must be visible first.

A consistently high value for the DependentWaitCommits metric might indicate that network latency is causing a delay before a transaction committed on another TE becomes visible to transactions on this TE.
TEs Only

DiskWritten

Number of bytes written to a volume, by the SM or SSM when storing atoms, since the last time metrics were provided.  

EffectiveVersion

Internal NuoDB version number. This value can change as the result of a rolling upgrade.  

HeapActive

At the moment this metric is provided, this is the total number of bytes in active pages of the heap. This number is equal to the HeapAllocated metric plus space lost due to fragmentation.

HeapActive includes pages that are swapped out. HeapActive does not include pages entirely devoted to allocator metadata nor empty pages that have not yet been returned to the operating system.
 

HeapAllocated

Current used memory in the heap, in bytes, at the moment this metric is provided.  

HeapMapped

Current virtual memory, in bytes, allocated to the heap at the moment this metric is provided. Includes empty pages but not empty chunks. Includes pages that are swapped out.  

Hostname

Name of the process host. This value never changes for the life of the process.  
HTTPProcessingThreadBacklog Milliseconds spent waiting to allocate an HTTP storage thread since the last time metrics were provided.  
IdleTime Milliseconds that the process had nothing to do since the last time metrics were provided  

IdManagerBlockingStallCount

Number of times a thread had to wait for object IDs from the chairman since the last time metrics were provided.  

IdManagerNonBlockingStallCount

Number of times a non-blocking request for an object ID had to wait for IDs from the chairman since the last time metrics were provided.  

Inserts

Number of records inserted since the last time metrics were provided. TEs Only
JournalBandwidthThrottleTime Milliseconds spent in journal I/O throttle since the last time metrics were provided. Journal I/O throttle leaves some disk bandwidth for the archive  
JournalDirectoryTime Milliseconds spent in journal directory operations since the last time metrics were provided. SM/SSM only
JournalFsyncTime Milliseconds spent finishing journal writes since the last time metrics were provided. SM/SSM only

JournalQueue

At the moment the metric is provided, this is the number of messages in the queue to be added to the journal file. SM/SSM only
JournalWriteTime Milliseconds spent in journal write since the last time metrics were provided. SM/SSM only

JrnlBytes

Number of bytes written to the journal file since the last time metrics were provided. SM/SSM only

JrnlWrites

Number of writes to the journal file since the last time metrics were provided. SM/SSM only

KernelMilliseconds

CPU time in milliseconds spent processing in the system kernel since the last time metrics were provided.  
LoadObjectTime Milliseconds spent loading objects from archive since the last time metrics were provided. SM/SSM only
LocalCommitOrderTime Time spent ensuring that commit operations on this transaction engine are made visible to other transactions on this TE in the correct order. This is a number of milliseconds since the last time metrics were provided.

Time counted by LocalCommitOrderTime is expended in the commit operation. After transactions on the same TE that are supposed to be visible first have committed, this commit operation returns to its client.
TEs Only

LogMsgs

Number of monitor log messages sent since the last time metrics were provided.  

Memory

At the moment the metric is provided, this is the current physical memory, in bytes, that is being used by this process. This amount does not include shared libraries nor pages that are swapped out.  
MemoryThrottleTime Total cumulative milliseconds spent in heap memory throttle, which slows processing to decrease memory consumption.  

MessagesReceived

Number of socket messages received from other database processes since the last time metrics were provided.  

MessagesSent

Number of socket messages sent by this process since the last time metrics were provided.  

Milliseconds

Elapsed time since last snapshot of metrics was taken. Typically, this is about 10 seconds.  

Naps

Time in milliseconds that volume I/O threads were forced to sleep since the last time metrics were provided. Naps allow other threads to process.

Replaced by ArchiveBandwidthThrotttleTime in NuoDB 2.4
 

NodeApplyPingTime

Time in milliseconds spent applying ping messages from another database process since the last time metrics were provided.  

NodeId

The node ID of this process. This value never changes for the life of the process.  

NodePingTime

Time in milliseconds spent pinging another database process since the last time metrics were provided.  

NodePostMethodTime

Time spent processing messages from other database processes since the last time metrics were provided.  

NodeSocketBufferWriteTime

Time in milliseconds spent doing inter-process socket writes since the last time metrics were provided.  

NodeState

Internal state of this database process at the moment that the metric is provided.  

NodeType

Indicates whether this process is a transaction engine, storage manager or snapshot storage manager. This value never changes for the life of the process.  

NonChairSplitTime

Time in milliseconds to process index split on non-chairman database processes since the last time metrics were provided.  

NumberCores

Number of processor cores. This value never changes for the life of the process.  

NumSplits

Number of index splits since the last time metrics were provided.  
ObjectFootprint The estimated, average amount of memory (in bytes) that the process allocated from the heap for each object.  

Objects

At the moment the metric is provided, this is the number of atom objects: (ObjectsCreated + ObjectsImported + Object(ObjectsCreated + ObjectsImported + (ObjectsLoaded - ObjectsReloaded) - ObjectsDropped  

ObjectsBounced

Number of requests for an atom that are bounced by a database process for various reasons. One example of a bounce is when a TE does not have a requested atom. This is the number of bounced requests since the last time metrics were provided.  

ObjectsCreated

Number of atom objects created since the last time metrics were provided.  

ObjectsDeleted

Number of atom objects deleted since the last time metrics were provided.  

ObjectsDropped

Number of aged atom objects dropped since the last time metrics were provided.  
ObjectsDroppedPurged Number of purged Atom objects dropped. SM/SSM only

ObjectsExported

Number of atom objects exported since the last time metrics were provided.  

ObjectsImported

Number of atom objects imported since the last time metrics were provided.  

ObjectsLoaded

Number of atom objects loaded since the last time metrics were provided. SMs and SSMs only. SM/SSM only

ObjectsPurged

Number of aged atom objects purged during garbage collection since the last time metrics were provided. SMs and SSMs only. GC has marked the object as not being part of a transaction but it is currently available in at least one TE. SM/SSM only

ObjectsRedirected

Number of atom object requests redirected to another database process since the last time metrics were provided.

Obsolete from NuoDB 2.4.

 

ObjectsRequested

Number of atom object requests since the last time metrics were provided.  

ObjectsSaved

Number of atom objects written to a volume by the SM or SSM since the last time metrics were provided. SM/SSM only
P-cpuUsage Percentage of CPU capacity that this process used since the last time metrics were provided. NuoDB calculates this by normalizing the percentage reported by the ps command to the total CPU capacity available. Note that this is not the same as the PercentCpuTime metric in that P-cpuUsage will never be above 100%.

To see this metric, you must set the NuoDB Manager monitorDomainNew property to true. The default is false.
 

PacketsReceived

Number of socket reads since the last time metrics were provided.  

PacketsSent

Number of socket writes since the last time metrics were provided.  

PageFaults

Number of system page faults since the last time metrics were provided.  
PendingEventsCommitTime Milliseconds spent waiting for commit event processing since the last time metrics were provided.  

PendingInsertWaitTime

Time in milliseconds spent waiting on a pending insert since the last time metrics were provided.  

PendingUpdateStallCount

Number of times this process waited on a pending update since the last time metrics were provided.  

PendingUpdateWaitTime

Time in milliseconds this database process waited on a pending update since the last time metrics were provided.  

PercentCpuTime

Percentage of both system kernel and system user time since the last time metrics were provided. This is PercentUserTime+PercentSystemTime.  

PercentSystemTime

Percentage of system kernel time since the last time metrics were provided. This is KernelMilliseconds/Milliseconds.  

PercentUserTime

Percentage of system user time since the last time metrics were provided. This is UserMilliseconds/Milliseconds.  

PlatformCatalogStallCount

Number of times this process had to stall waiting for a block of atom fingerprints from another database process while syncing.  
PlatformIndexCheckAcknowledgedTime Time in milliseconds spent checking if an index Atom object is acknowledged, i.e. it is complete and all other nodes acknowledge directly to the Atom's node. The purpose of this timer is to measure any stall time that can happen waiting for the operation to complete. Unique index updates require index atoms to be acknowledged to perform uniqueness checks.

PlatformObjectCheckCompleteTime

Time in milliseconds since the last time metrics were provided that this process waited for an atom object to become complete. That is, all other database processes acknowledge its existence.  

PlatformObjectCheckOpenTime

Time in milliseconds since the last time metrics were provided that this process waited for an atom object to be opened/loaded. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.  

PlatformObjectCheckPopulatedTime

Time in milliseconds since the last time metrics were provided that this process spent waiting for an atom to be fully populated. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.  

ProcessId

The process ID for this process. This value never changes for the life of the process.  
PruneAtomsThrottleTime Total cumulative milliseconds when atom pruning was throttled to give the SM time to catch up writing dirty atoms to the archive.  
PurgedObjects Number of purged objects currently in memory. SM/SSM only. ObjectsPurged - ObjectsReloaded - ObjectsDroppedPurged

RefactorTXQueueTime

Time in milliseconds since the last time metrics were provided that this process spent refactoring the transaction queue.  

RemoteCommitTime

Time in milliseconds since the last time metrics were provided that this process spent waiting for commit acknowledgement from SM or SSM. Relevant to remote commit. Relevant to remote commit.

Rollbacks

Number of transaction rollbacks since the last time metrics were provided.  

SendQueueSize

Number of buffered, unsorted messages in message sequencer to be sent to other database processes. This is the number at the moment the metric is provided.  

SendSortedQueueSize

Number of buffered, sorted messages in message sequencer to be sent to other database processes. This is the number at the moment the metric is provided.  

ServerReceived

Number of socket reads for the platform service since the last time metrics were provided.  

ServerSent

Number of socket writes for the platform service since the last time metrics were provided.  

SnapshotAlbumClosed

Number of snapshot albumA snapshot storage manager uses albums to efficiently manage snapshots. An album refers to a set of snapshots taken during some period of time. Also, an album provides a place to store additional state required to use snapshots.s that were closed since the last time metrics were provided. SSM only. SSM Only

SnapshotAlbumClosedForGC

This is the number of snapshot albums that were closed to allow expunging of old record versions (as opposed to being closed because they were either old or large). SSM only. These albums were closed since the last time metrics were provided. SSM Only

SnapshotAlbumSize

Since the last time metrics were provided, this is the average number of snapshots in a closed snapshot album. SSM Only

SnapshotAlbumTime

Since the last time metrics were provided, this is the average length of time in seconds that a snapshot album was open. SSM Only

SocketBufferBytes

At the moment that metrics are provided, this is the number of bytes buffered for inter-process send operations.  

SqlListenerIdleTime

Time in milliseconds since the last time metrics were provided that this TE was waiting on a request from an application client. TE only

SqlListenerSqlProcTime

Time spent processing SQL messages, in milliseconds, since the last time metrics were provided. Processing time starts from the point when a request is received from an application client, includes all the time needed to process the message, and ends when a response is sent back to the client.  

SqlListenerStallTime

Time in milliseconds since the last time metrics were provided that this process is waiting for something to happen, including time blocked, objects to load, network waits and locks. This time is included in the SqlListenerSqlProcTime metric.  

SqlListenerThrottleTime

Time in milliseconds waiting for memory to be freed since the last time metrics were provided.  

SqlMsgs

Number of socket writes for the platform service since the last time metrics were provided.  

StallPointWaitTime

Time in milliseconds that this process is waiting in a stall point.  

Stalls

Number of thread stalls since the last time metrics were provided.  
SyncPointWaitTime Milliseconds spent waiting for a resource locked by another thread since the last time metrics were provided.  

TransactionBlockedTime

Since the last time metrics were provided, aggregated time in milliseconds that uncommitted transactions are waiting for another transaction to commit.

A transaction might be waiting because another transaction locked a record or because another transaction commit is supposed to be visible to this transaction.

This metric also includes time spent waiting for metadata changes to commit.
 

Updates

Number of records updated since the last time metrics were provided. TEs Only

UserMilliSeconds

CPU time in milliseconds spent processing user code since the last time metrics were provided.  

WaitForDatabaseStallCount

Number of times the transaction engine had to wait for a database atom since the last time metrics were provided.

Obsolete from NuoDB 2.4.
 
WaitForSplitTime Milliseconds spent by an index insert waiting for split since the last time metrics were provided.  
WriteLoadLevel This metric indicates the value at the time the metric is provided.

When a storage manager provides this metric, it indicates the number of arriving objects (atoms) per second that exceed the storage manager's estimated, maximum, sustainable capacity for saving the modified atoms to the archive.

When a transaction engine provides this metric, it indicates the highest WriteLoadLevel provided by any of the database's storage managers since the last time metrics were provided.
 
WriteThrottleSetting At the time metrics are provided, this is the number of microseconds that the transaction engine delays each database write operation. This delay throttles the rate at which the transaction engine sends atom changes to the database's storage managers.

Throttling occurs only when the rate of atom changes exceeds the rate at which the database's storage managers can save the modified atoms in the archive.
TEs Only
WriteThrottleTime Since the last time metrics were provided, this is the time in milliseconds that the transaction engine delayed database write operations.

Transaction engines throttle the rate at which they send atom changes to the database's storage managers when the rate of atom changes exceeds the rate at which the storage managers can save the modified atoms in the archive.
TEs Only