Metrics Published by Database Processes

Each database process (Transaction Engine (TE) or Storage Manager (SM)) maintains a set of metrics providing insight into how the process is performing. Each process’s metrics cover only that process.

The nuocmd get stats subcommand is used to obtain metrics from either a single database process or all the processes in the database. It will retrieve and display metrics as concatenated JSON, every 10 seconds, until the command is canceled. Multiple commands may be retrieving metrics from the same database process at the same time, if desired.

The first response will contain a value for every metric. Subsequent responses will only contain metrics whose values have changed after the previous response. Metrics which are not relevant for a process (for example, the ArchiveQueue metric is not relevant to TEs) will appear in the initial response but not subsequent responses, because their values do not change.

The table below uses these important terms:

  • elapsed time refers to the exact interval covered by a response and is reflected by the Milliseconds metric.

  • active thread refers to a thread when it is either doing work or has work to do, but is waiting for an operation to complete.

  • active thread time refers to the total time accumulated by all threads that are active during the elapsed time. The active thread time may exceed the elapsed time, because multiple threads in the process can be active simultaneously.

  • idle thread refers to a thread that is not active (has no work to do).

  • drop atom refers to evicting and removing cached atom from memory.

  • delete atom refers to permanently removing atom from disk.

This table describes the available metrics. While some of the metrics in this table are internal only, you might need to provide this information when liaising with NuoDB Customer Support.

Metric Name Description Relevant For

ActiveTime

Milliseconds of active thread time since the last time metrics were provided. This value divided by the elapsed time equals the average number of active threads.

ActualVersion

Internal NuoDB version number. It never changes for the life of the process.

AdminReceived

Number of administrative service requests since the last time metrics were provided.

AdminSent

Number of administrative service responses since the last time metrics were provided. This includes log and metrics messages.

ArchiveAtomsCompacted

Number of atoms moved by Log-Structured Archive (LSA) compaction.

SM only

ArchiveBandwidthThrottleTime

Milliseconds spent in archive I/O throttle since the last time metrics were provided. I/O throttle limits disk bandwidth consumption for writes, to leave some bandwidth for reads.

SM only

ArchiveBufferedBytes

While saving atoms, journaling can fall behind. On the second attempt to save an atom, a snapshot of the context is saved, the atom marked as dirty, and the atom is placed back on the archive queue. ArchiveBufferedBytes is the number of bytes saved in the snapshots since the last time metrics were provided.

SM only

ArchiveBytesCompacted

Bytes moved by LSA compaction.

SM only

ArchiveCatalogsPruned

Number of LSA catalogs pruned from memory

SM only

ArchiveDirectory

Archive directory. This value never changes for the life of the process.

SM only

ArchiveDirectoryTime

Milliseconds spent in archive directory operations since the last time metrics were provided.

SM only

ArchiveFsyncTime

Milliseconds spent finishing archive writes since the last time metrics were provided.

SM only

ArchiveMemory

Bytes of memory allocated for managing archive storage in LSA

SM only

ArchiveQueue

At the moment that metrics are provided, this is the number of atoms queued and waiting to be written to disk by the SM.

SM only

ArchiverActiveTime

Milliseconds when the archiver thread(s) are doing work, i.e. not idle.

SM only

ArchiveReadQueueLow

Displays the number of queued atom read requests in the low priority queue.

SM only

ArchiveReadQueueMedium

Displays the number of queued atom read requests in the medium priority queue.

SM only

ArchiveReadQueueHigh

Displays the number of queued atom read requests in the high priority queue.

SM only

ArchiveReadTime

Milliseconds spent in archive read since the last time metrics were provided.

SM only

ArchiveSize

Total bytes of disk space used by LSA

SM only

ArchiveSpaceWasted

Bytes of disk space wasted by old versions of atoms in LSA

SM only

ArchiveSyncThrottleTime

Milliseconds spent in archive synchronization throttle since the last time metrics were provided. Archive synchronization throttle slows the process down to decrease interference with other work.

SM only

ArchiveThrottleTime

The duration, in milliseconds, for which the archive writes were throttled to allow LSA compaction to catch up

SM only

ArchiveWaitForJournalTime

Milliseconds that the archiver thread(s) slept until the journal caught up.

SM only

ArchiveWriteTime

Milliseconds spent in archive write since the last time metrics were provided.

SM only

AsyncSMDelayTime

Estimated time delay between the active database and the Asynchronous Storage Manager’s (ASM) consistent state which would be restored by handoff (in microseconds). It does not include network delay between active and passive data centers.

ASM only

AsyncSMDelayedTransactions

Number of committed transactions received by the ASM that are not yet part of the consistent state which would be restored by handoff.

ASM only

AtomProcessingThreadBacklog

Milliseconds spent waiting to allocate an atom processing thread since the last time metrics were provided.

BackupDirectoryTime

Milliseconds spent in backup (hot copy) directory operations.

BackupFsyncTime

Milliseconds spent finishing backup (hot copy) writes

BackupReadTime

Milliseconds spent in backup (hot copy) read.

BackupWriteTime

Milliseconds spent in backup (hot copy) write.

BroadcastTime

Milliseconds spent broadcasting messages to other database processes since the last time metrics were provided.

BtreeObjectsChanged

The number of multi index B+ tree objects changed during Merge Tree Indexing compaction.

BytesBuffered

Number of bytes buffered to be sent over sockets since the last time metrics were provided.

BytesReceived

Number of bytes received by the sockets since the last time metrics were provided.

BytesSent

Number of bytes sent over the sockets since the last time metrics were provided.

CleanObjectsBytes

Estimated memory usage by clean atoms and fingerprint blocks on an SM, or by all atoms and fingerprint blocks on a TE.

ClientCncts

Number of SQL client socket connections at the moment this metric is provided.

ClientReceived

Number of messages received from the SQL client since the last time metrics were provided.

ClientSent

Number of responses sent to the SQL client since the last time metrics were provided.

ClientThreadBacklog

Milliseconds spent waiting to allocate a client thread since the last time metrics were provided.

Commits

Number of transaction commits since the last time metrics were provided.

TE only

CreatePlatformRecordsTime

Milliseconds spent waiting for the chairman to create a Records atom since the last time metrics were provided.

CurrentActiveTransactions

Number of active transactions at the moment this metric is provided.

CurrentCommittedTransactions

Number of committed transactions at this time.

CurrentPurgedTransactions

Number of transactions that have been placed on the purgatory list at this time. A transaction being deleted is placed on the purgatory list to give other threads a chance to remove references to that transaction.

Deletes

Number of records deleted since the last time metrics were provided.

TE only

DependentCommitWaits

To ensure that there is always a consistent view of the database, NuoDB makes committed transactions visible to transactions on a particular Transaction Engine in the correct order. At the moment this metric is provided, this is the number of commits that have been received from other TEs and that cannot yet be visible to transactions on this TE. These commits cannot be visible yet because they depend on commits that have not yet arrived and that must be visible first.
A consistently high value for the DependentWaitCommits metric might indicate that network latency is causing a delay before a transaction committed on another TE becomes visible to transactions on this TE.

TE only

DirtyObjectsBytes

Estimated memory usage by dirty atoms and fingerprint blocks.

SM only

DirtyObjectsSkipped

Count atoms considered by archiver but not saved.

SM only

DiskReadGeneral

Reports the number of bytes read from disk for all database operations other than hot copy.

DiskReadHotCopy

Reports the number of bytes read from disk as part of hot copy operations.

DiskWritten

Number of bytes written to a volume, by the SM when storing atoms since the last time metrics were provided.

SM

EffectiveVersion

Internal NuoDB version number. This value can change as the result of a rolling upgrade.

FingerprintBlocksSaved

Number of fingerprint blocks written to a volume by the SM since the last time metrics were provided.

SM only

FingerprintBlocksSkipped

Count fingerprint blocks considered by archiver but not saved.

SM only

HeapActive

At the moment this metric is provided, this is the total number of bytes in active pages of the heap. This number is equal to the HeapAllocated metric plus space lost due to fragmentation.
HeapActive includes pages that are swapped out. HeapActive does not include pages entirely devoted to allocator metadata nor empty pages that have not yet been returned to the operating system.

HeapAllocated

Current used memory in the heap, in bytes, at the moment this metric is provided.

HeapMapped

Current virtual memory, in bytes, allocated to the heap at the moment this metric is provided. Includes empty pages but not empty chunks. Includes pages that are swapped out.

Hostname

Name of the process host. This value never changes for the life of the process.

HTTPProcessingThreadBacklog

Milliseconds spent waiting to allocate an HTTP storage thread since the last time metrics were provided.

IdleTime

Milliseconds that the process had nothing to do since the last time metrics were provided

IdManagerBlockingStallCount

Number of times a thread had to wait for object IDs from the chairman since the last time metrics were provided.

IdManagerNonBlockingStallCount

Number of times a non-blocking request for an object ID had to wait for IDs from the chairman since the last time metrics were provided.

IncomingMessagesBytes

Estimated memory usage by incoming messages, including messages being applied, pending messages, and messages in the reconciliation queue.

Inserts

Number of records inserted since the last time metrics were provided.

TE only

JournalBandwidthThrottleTime

Milliseconds spent in journal I/O throttle since the last time metrics were provided. Journal I/O throttle leaves some disk bandwidth for the archive

JournalCheckpoints

Number of journal checkpoints declared since the last time metrics were provided.

SM only

JournalDirectoryTime

Milliseconds spent in journal directory operations since the last time metrics were provided.

SM only

JournalFsyncTime

Milliseconds spent finishing journal writes since the last time metrics were provided.

SM only

JournalQueue

At the moment the metric is provided, this is the number of messages in the queue to be added to the journal file.

SM only

JournalReadTime

Milliseconds spent in journal read

JournalWriteTime

Milliseconds spent in journal write since the last time metrics were provided.

SM only

JrnlBytes

Number of bytes written to the journal file since the last time metrics were provided.

SM only

JrnlWrites

Number of writes to the journal file since the last time metrics were provided.

SM only

KernelMilliseconds

CPU time in milliseconds spent processing in the system kernel since the last time metrics were provided.

LoaderActiveTime

Milliseconds when the loader thread is doing work, i.e. not idle.

SM only

LoadObjectTime

Milliseconds spent loading objects from archive since the last time metrics were provided.

SM only

LocalCommitOrderTime

Time spent ensuring that commit operations on this TE are made visible to other transactions on this TE in the correct order. This is a number of milliseconds since the last time metrics were provided.
Time counted by LocalCommitOrderTime is expended in the commit operation. After transactions on the same TE that are supposed to be visible first have committed, this commit operation returns to its client.

TE only

LogMsgs

Number of monitor log messages sent since the last time metrics were provided.

Memory

At the moment the metric is provided, this is the current physical memory, in bytes, that is being used by this process. This amount does not include shared libraries nor pages that are swapped out.

MemoryThrottleTime

Total cumulative milliseconds spent in heap memory throttle, which slows processing to decrease memory consumption.

MessagesReceived

Number of socket messages received from other database processes since the last time metrics were provided.

MessagesSent

Number of socket messages sent by this process since the last time metrics were provided.

Milliseconds

Elapsed time since last snapshot of metrics was taken. Typically, this is about 10 seconds.

MiscellaneousDirectoryTime

Milliseconds spent in Miscellaneous directory operations

MiscellaneousReadTime

Milliseconds spent in Miscellaneous read.

MiscellaneousWriteTime

Milliseconds spent in Miscellaneous write.

MutationsArchived atom_type

Number of mutations (that is, messages that dirtied an atom) saved to archive, broken down by type (MasterCatalog, TransactionManager etc).

NextCheckpointDirty

Number of dirty objects and fingerprints in next checkpoint.

NextCheckpointUnreapableMessages

Number of messages that dirtied objects in next checkpoint.

NextCheckpointWaitingForJournal

Number of dirty objects waiting for journal to catch up.

NodeApplyPingTime

Milliseconds spent applying ping messages from another database process since the last time metrics were provided.

NodeId

The node ID of this process. This value never changes for the life of the process.

NodePingTime

Milliseconds spent pinging another database process since the last time metrics were provided.

NodePostMethodTime

Time spent processing messages from other database processes since the last time metrics were provided.

NodeSocketBufferWriteTime

Milliseconds spent doing inter-process socket writes since the last time metrics were provided.

NodeState

Internal state of this database process at the moment that the metric is provided.

NodeType

Indicates whether this process is a TE or a SM. This value never changes for the life of the process.

NonChairSplitTime

Time in milliseconds to process index split on non-chairman database processes since the last time metrics were provided.

NumberCores

Number of processor cores. This value never changes for the life of the process.

NumSplits

Number of index splits since the last time metrics were provided.

ObjectFootprint

The estimated, average amount of memory (in bytes) that the process allocated from the heap for each object.

Objects

The number of atoms in memory at the instance the metric is provided.

ObjectsBounced

Number of requests for an atom that are bounced by a database process for various reasons. One example of a bounce is when a TE does not have a requested atom. This is the number of bounced requests since the last time metrics were provided.

ObjectsCreated

Number of atoms created since the last time metrics were provided.

ObjectsDeleted

Number of atoms deleted (permanently removed from disk) since the last time metrics were provided.

ObjectsDropped

Number of times an atom was dropped (evicted and removed from cache memory) since the last time metrics were provided.

ObjectsDroppedPurged

Number of purged atoms dropped.

SM only

ObjectsExported

Number of atoms sent to another database process since the last time metrics were provided. Atoms are exported in response to atom requests (ObjectsRequested) and when newly-created atoms are pushed (ObjectsCreated). See also ObjectsImported.

ObjectsImported

Number of atoms received from other database processes since the last time metrics were provided. An atom may be imported because it was requested (ObjectsRequested), or because another database process created and pushed a new atom (ObjectsCreated). See also ObjectsExported.

ObjectsLoaded

Number of atoms loaded from disk since the last time metrics were provided.

SM only

ObjectsPurged

Number of aged atoms purged during garbage collection since the last time metrics were provided.
GC has marked the object as not being part of a transaction but it is currently available in at least one TE.

SM only

ObjectsReloaded

Number of atoms purged from memory, then loaded again from the archive.

If this (purged object reloading) happens frequently, it may indicate that SM memory is too small for the workload in the TEs.

SM only

ObjectsRequested

Number of atom requests since the last time metrics were provided.

ObjectsSaved

Number of atoms written to the archive by the SM since the last time metrics were provided.

SM only

ObjectsSavedEarly

Number of atoms written to the archive because of memory pressure or to allow archive synchronization to finish, since the last time metrics were provided.

SM only

OutstandingObjectRequests

Number of object requests sent and not yet answered.

PacketsReceived

Number of socket reads since the last time metrics were provided.

PacketsSent

Number of socket writes since the last time metrics were provided.

PageFaults

Number of major page faults since the process started. Major page faults are memory read operations that require swapping in pages from disk into the process’s virtual memory. This metric is obtained from the operating system.

PageFaultsMinor

Number of minor page faults since the process started. Minor page faults are memory read operations that require linking in-memory pages into the process’s virtual memory. This metric is obtained from the operating system.

PendingEventsCommitTime

Milliseconds spent waiting for commit event processing since the last time metrics were provided.

PendingInsertWaitTime

Milliseconds spent waiting on a pending insert since the last time metrics were provided.

PendingUpdateStallCount

Number of times this process waited on a pending update since the last time metrics were provided.

PendingUpdateWaitTime

Milliseconds this database process waited on a pending update since the last time metrics were provided.

PercentArchiveRead

Percentage of active thread time spent reading atoms from the archive on disk.

PercentArchiveWrite

Percentage of active thread time spent writing atoms to the archive on disk.

PercentCommit

Percentage of active thread time waiting for a transaction to be made durable on disk.

PercentCpuTime

Percentage of both system kernel and system user time since the last time metrics were provided. This is PercentUserTime + PercentSystemTime.

PercentFetch

Percentage of active thread time waiting to fetch an atom from another node or from disk.

PercentIdle

Percentage of elapsed time that the process had no work to do (all threads idle).

PercentJournalWrite

Percentage of active thread time spent writing messages to the journal for crash recovery (serialization plus disk busy).

PercentLock

Percentage of active thread time that a client was waiting for another client to finish a transaction that locked an SQL record.

PercentNetworkSend

Percentage of active thread time spent sending messages into the network.

PercentRunning

Percentage of active thread time that was spent running code, in user or kernel mode.

PercentStall

Percentage of active thread time that one thread was waiting for another thread to release exclusive access to an object.

PercentSystemTime

Percentage of system kernel time since the last time metrics were provided. This is KernelMilliseconds/Milliseconds.

PercentThrottle

Percentage of active thread time that processing was paused due to overconsumption of a resource such as memory or I/O bandwidth; sum of all throttles.

PercentUserTime

Percentage of system user time since the last time metrics were provided. This is UserMilliseconds/Milliseconds.

PercentWastedDiskSpace

Percentage of archive disk space wasted by old versions of atoms in LSA.

SM only

PlatformCatalogStallCount

Number of times this process had to stall waiting for a block of atom fingerprints from another database process while syncing.

PlatformObjectCheckAcknowledgedTime

Milliseconds spent checking if an atom is acknowledged, that is, it is complete and all other nodes acknowledge directly to the atom’s node. The purpose of this timer is to measure any stall time that can happen waiting for the operation to complete.

PlatformObjectCheckCompleteTime

Milliseconds since the last time metrics were provided that this process waited for an atom to become complete. That is, all other database processes acknowledge its existence.

PlatformObjectCheckOpenTime

Milliseconds since the last time metrics were provided that this process waited for an atom to be opened/loaded. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.

PlatformObjectCheckPopulatedTime

Milliseconds since the last time metrics were provided that this process spent waiting for an atom to be fully populated. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.

PreviousCheckpointDirty

Number of dirty objects and fingerprints in previous checkpoint.

PreviousCheckpointUnreapableMessages

Number of messages that dirtied objects in previous checkpoint.

PreviousCheckpointWaitingForJournal

Number of dirty objects waiting for journal to catch up.

ProcessId

The process ID for this process. This value never changes for the life of the process.

PruneAtomsThrottleTime

Total cumulative milliseconds when atom pruning was throttled to give the SM time to catch up writing dirty atoms to the archive.

PurgedObjects

Number of purged objects currently in memory.

SM only

ReconciliationQueueSize

Number of messages saved in case of failover. This is the number at the moment the metric is provided.

RefactorTXQueueTime

Milliseconds since the last time metrics were provided that this process spent refactoring the transaction queue.

RemoteCommitTime

Milliseconds since the last time metrics were provided that this process spent waiting for commit acknowledgment from SM.

TE only

ReplicationMessagesReceived

Number of replication messages received by the TE or SM.

Rollbacks

Number of transaction rollbacks since the last time metrics were provided.

SendQueueSize

Number of outgoing messages in message sequencer to be sent to other database processes. This is the number at the moment the metric is provided.

ServerReceived

Number of socket reads for the platform service since the last time metrics were provided.

ServerSent

Number of socket writes for the platform service since the last time metrics were provided.

SocketBufferBytes

At the moment that metrics are provided, this is the number of bytes buffered for inter-process send operations.

SpillFileDirectoryTime

Milliseconds spent in spill file directory operations

SpillFileReadTime

Milliseconds spent in spill file read.

SpillFileWriteTime

Milliseconds spent in spill file read.

SqlListenerIdleStatementTime

Milliseconds that this TE was waiting on a request from an application client with an open statement since the last time metrics were provided.

TE only

SqlListenerIdleTime

Milliseconds that this TE was waiting on a request from an application client since the last time metrics were provided.

TE only

SqlListenerIdleTransactionTime

Milliseconds that this TE was waiting on a request from an application client with an open transaction since the last time metrics were provided.

TE only

SqlListenerSqlProcTime

Milliseconds spent processing SQL messages since the last time metrics were provided. Processing time starts from the point when a request is received from an application client, includes all the time needed to process the message, and ends when a response is sent back to the client.

SqlListenerStallTime

Milliseconds since the last time metrics were provided that this process is waiting for something to happen, including time blocked, objects to load, network waits and locks. This time is included in the SqlListenerSqlProcTime metric.

SqlListenerThrottleTime

Time in milliseconds waiting for memory to be freed since the last time metrics were provided.

SqlMsgs

Number of socket writes for the platform service since the last time metrics were provided.

StallPointWaitTime

Milliseconds that this process is waiting in a stall point.

Stalls

Number of thread stalls since the last time metrics were provided.

SyncAtomByteCopied

Size of atoms copied to archive by sync.

SyncAtomsCopied

Atoms copied by archive synchronization.

SyncAtomsUpToDate

Up-to-date atoms in catalogs whose sync has been started.

SyncCatalogsStarted

Number of catalogs whose sync has been started.

SyncCurrentStorageGroup

Storage Group ID of the current storage group being synced, 0 otherwise.

SyncCurrentTableMigration

Object ID of the table currently being migrated, 0 otherwise.

SyncPointWaitTime

Milliseconds spent waiting for a resource locked by another thread since the last time metrics were provided.

SyncTotalAtoms

Total number of atoms in catalogs whose sync has been started.

SM only

SyncTotalCatalogs

The total catalogs to be synced.

SM only

SyncTotalFingerprintBlocks

Total number of fingerprint blocks to be synced.

SM only

Time

Time at end of monitor period.

TransactionBlockedTime

Since the last time metrics were provided, aggregated time in milliseconds that uncommitted transactions are waiting for another transaction to commit.
A transaction might be waiting because another transaction locked a record or because another transaction commit is supposed to be visible to this transaction.
This metric also includes time spent waiting for metadata changes to commit.

TransactionDependencyRemoteMin

The minimum time, in microseconds, required to resolve transaction dependency.

TE only

TransactionDependencyRemoteMax

The maximum time, in microseconds, required to resolve transaction dependency.

TE only

TransactionDependencyRemoteMean

The average time, in microseconds, required to resolve transaction dependency.

TE only

Updates

Number of records updated since the last time metrics were provided.

TE only

UserMilliseconds

CPU time in milliseconds spent processing user code since the last time metrics were provided.

WaitForSplitTime

Milliseconds spent by an index insert waiting for split since the last time metrics were provided.

WriteLoadLevel

This metric indicates the value at the time the metric is provided.
When an SM provides this metric, it indicates the number of arriving objects (atoms) per second that exceed the SM’s estimated, maximum, sustainable capacity for saving the modified atoms to the archive.
When a TE provides this metric, it indicates the highest WriteLoadLevel provided by any of the database’s SMs since the last time metrics were provided.

WriteThrottleTime

The time in milliseconds that the TE delayed database write operations since the last time metrics were provided.
TEs throttle the rate at which they send atom changes to the database’s SMs when the rate of atom changes exceeds the rate at which the SMs can save the modified atoms in the archive.

TE only