Metrics Published by Database Processes

The table below lists all metrics that a NuoDB database publishes. Each metric is produced by a database process, that is, a Transaction Engine (TE) or a Storage Manager (SM). Each metric value is relevant only for the database process that generates it.

These metrics can be obtained using the nuocmd get stats subcommand.

All metrics are produced every 10 seconds. The exact interval is provided by the Milliseconds metric.

If a metric value has not changed since the last time metrics were provided then that metric is omitted from the metric values provided to subscribers. Consequently, metrics that are not relevant for a process type are provided the first time that process provides metrics and then those metrics are no longer provided by that process. For example, consider the ArchiveQueue metric, which is relevant only for SMs. The first time that a TE generates metrics, it provides a 0 for the ArchiveQueue metric. For TEs, the value of this metric never changes so after the first time a TE provides metrics, it no longer generates a value for the ArchiveQueue metric.

While some of the metrics in this table are internal only, you might need to provide this information when liaising with NuoDB Customer Support.

Metric Name Description Relevant For

ActiveTime

Milliseconds of active thread time since the last time metrics were provided. This value divided by the elapsed time equals the average number of active threads.

ActualVersion

Internal NuoDB version number. It never changes for the life of the process.

AdminReceived

Number of administrative service requests since the last time metrics were provided.

AdminSent

Number of administrative service responses since the last time metrics were provided. This includes log and metrics messages.

ArchiveBandwidthThrottleTime

Milliseconds spent in archive I/O throttle since the last time metrics were provided. I/O throttle limits disk bandwidth consumption for writes, to leave some bandwidth for reads.

ArchiveBufferedBytes

While saving atoms, journaling can fall behind. On the second attempt to save an atom, a snapshot of the context is saved, the atom marked as dirty, and the atom is placed back on the archive queue. ArchiveBufferedBytes is the number of bytes saved in the snapshots since the last time metrics were provided.

ArchiveDirectory

Archive directory. This value never changes for the life of the process.

SM only

ArchiveDirectoryTime

Milliseconds spent in archive directory operations since the last time metrics were provided.

SM only

ArchiveFsyncTime

Milliseconds spent finishing archive writes since the last time metrics were provided.

SM only

ArchiveQueue

At the moment that metrics are provided, this is the number of atoms queued and waiting to be written to disk by the SM.

SM only

ArchiverActiveTime

Milliseconds when the archiver thread(s) are doing work, i.e. not idle.

SM only

ArchiveReadQueueLow

Displays the number of queued atom read requests in the low priority queue.

SM only

ArchiveReadQueueMedium

Displays the number of queued atom read requests in the medium priority queue.

SM only

ArchiveReadQueueHigh

Displays the number of queued atom read requests in the high priority queue.

SM only

ArchiveReadTime

Milliseconds spent in archive read since the last time metrics were provided.

SM only

ArchiveSyncThrottleTime

Milliseconds spent in archive synchronization throttle since the last time metrics were provided. Archive synchronization throttle slows the process down to decrease interference with other work.

SM only

ArchiveWaitForJournalTime

Milliseconds that the archiver thread(s) slept until the journal caught up.

SM only

ArchiveWriteTime

Milliseconds spent in archive write since the last time metrics were provided.

SM only

AtomProcessingThreadBacklog

Milliseconds spent waiting to allocate an atom processing thread since the last time metrics were provided.

BackupDirectoryTime

Milliseconds spent in backup (hot copy) directory operations.

BackupFsyncTime

Milliseconds spent finishing backup (hot copy) writes

BackupReadTime

Milliseconds spent in backup (hot copy) read.

BackupWriteTime

Milliseconds spent in backup (hot copy) write.

BroadcastTime

Milliseconds spent broadcasting messages to other database processes since the last time metrics were provided.

BytesBuffered

Number of bytes buffered to be sent over sockets since the last time metrics were provided.

BytesReceived

Number of bytes received by the sockets since the last time metrics were provided.

BytesSent

Number of bytes sent over the sockets since the last time metrics were provided.

ClientCncts

Number of SQL client socket connections at the moment this metric is provided.

ClientReceived

Number of messages received from the SQL client since the last time metrics were provided.

ClientSent

Number of responses sent to the SQL client since the last time metrics were provided.

ClientThreadBacklog

Milliseconds spent waiting to allocate a client thread since the last time metrics were provided.

Commits

Number of transaction commits since the last time metrics were provided.

TEs Only

CreatePlatformRecordsTime

Milliseconds spent waiting for the chairman to create a Records atom since the last time metrics were provided.

CurrentActiveTransactions

Number of active transactions at the moment this metric is provided.

CurrentCommittedTransactions

Number of committed transactions at this time.

CurrentPurgedTransactions

Number of transactions that have been placed on the purgatory list at this time. A transaction being deleted is placed on the purgatory list to give other threads a chance to remove references to that transaction.

Deletes

Number of records deleted since the last time metrics were provided.

TEs Only

DependentCommitWaits

To ensure that there is always a consistent view of the database, NuoDB makes committed transactions visible to transactions on a particular transaction engine in the correct order. At the moment this metric is provided, this is the number of commits that have been received from other transaction engines and that cannot yet be visible to transactions on this TE. These commits cannot be visible yet because they depend on commits that have not yet arrived and that must be visible first.

A consistently high value for the DependentWaitCommits metric might indicate that network latency is causing a delay before a transaction committed on another TE becomes visible to transactions on this TE.

TEs Only

DirtyObjectsSkipped

Count atoms considered by archiver but not saved.

SM only

DiskReadGeneral

Reports the number of bytes read from disk for all database operations other than hot copy.

DiskReadHotCopy

Reports the number of bytes read from disk as part of hot copy operations.

DiskWritten

Number of bytes written to a volume, by the SM when storing atoms, since the last time metrics were provided.

EffectiveVersion

Internal NuoDB version number. This value can change as the result of a rolling upgrade.

FingerprintBlocksSaved

Number of fingerprint blocks written to a volume by the SM since the last time metrics were provided.

SM only

FingerprintBlocksSkipped

Count fingerprint blocks considered by archiver but not saved.

SM only

HeapActive

At the moment this metric is provided, this is the total number of bytes in active pages of the heap. This number is equal to the HeapAllocated metric plus space lost due to fragmentation.

HeapActive includes pages that are swapped out. HeapActive does not include pages entirely devoted to allocator metadata nor empty pages that have not yet been returned to the operating system.

HeapAllocated

Current used memory in the heap, in bytes, at the moment this metric is provided.

HeapMapped

Current virtual memory, in bytes, allocated to the heap at the moment this metric is provided. Includes empty pages but not empty chunks. Includes pages that are swapped out.

Hostname

Name of the process host. This value never changes for the life of the process.

HTTPProcessingThreadBacklog

Milliseconds spent waiting to allocate an HTTP storage thread since the last time metrics were provided.

IdleTime

Milliseconds that the process had nothing to do since the last time metrics were provided

IdManagerBlockingStallCount

Number of times a thread had to wait for object IDs from the chairman since the last time metrics were provided.

IdManagerNonBlockingStallCount

Number of times a non-blocking request for an object ID had to wait for IDs from the chairman since the last time metrics were provided.

Inserts

Number of records inserted since the last time metrics were provided.

TEs Only

JournalBandwidthThrottleTime

Milliseconds spent in journal I/O throttle since the last time metrics were provided. Journal I/O throttle leaves some disk bandwidth for the archive

JournalCheckpoints

Number of journal checkpoints declared since the last time metrics were provided.

SM only

JournalDirectoryTime

Milliseconds spent in journal directory operations since the last time metrics were provided.

SM only

JournalFsyncTime

Milliseconds spent finishing journal writes since the last time metrics were provided.

SM only

JournalQueue

At the moment the metric is provided, this is the number of messages in the queue to be added to the journal file.

SM only

JournalReadTime

Milliseconds spent in journal read

JournalWriteTime

Milliseconds spent in journal write since the last time metrics were provided.

SM only

JrnlBytes

Number of bytes written to the journal file since the last time metrics were provided.

SM only

JrnlWrites

Number of writes to the journal file since the last time metrics were provided.

SM only

KernelMilliseconds

CPU time in milliseconds spent processing in the system kernel since the last time metrics were provided.

LoaderActiveTime

Milliseconds when the loader thread is doing work, i.e. not idle.

SM only

LoadObjectTime

Milliseconds spent loading objects from archive since the last time metrics were provided.

SM only

LocalCommitOrderTime

Time spent ensuring that commit operations on this transaction engine are made visible to other transactions on this TE in the correct order. This is a number of milliseconds since the last time metrics were provided.

Time counted by LocalCommitOrderTime is expended in the commit operation. After transactions on the same TE that are supposed to be visible first have committed, this commit operation returns to its client.

TEs Only

LogMsgs

Number of monitor log messages sent since the last time metrics were provided.

Memory

At the moment the metric is provided, this is the current physical memory, in bytes, that is being used by this process. This amount does not include shared libraries nor pages that are swapped out.

MemoryThrottleTime

Total cumulative milliseconds spent in heap memory throttle, which slows processing to decrease memory consumption.

MessagesReceived

Number of socket messages received from other database processes since the last time metrics were provided.

MessagesSent

Number of socket messages sent by this process since the last time metrics were provided.

Milliseconds

Elapsed time since last snapshot of metrics was taken. Typically, this is about 10 seconds.

MiscellaneousDirectoryTime

Milliseconds spent in Miscellaneous directory operations

MiscellaneousReadTime

Milliseconds spent in Miscellaneous read.

MiscellaneousWriteTime

Milliseconds spent in Miscellaneous write.

MutationsArchived atom_type

Number of mutations (that is, messages that dirtied an atom) saved to archive, broken down by type (MasterCatalog, TransactionManager etc).

Naps

Milliseconds that volume I/O threads were forced to sleep since the last time metrics were provided. Naps allow other threads to process.

Replaced by ArchiveBandwidthThrotttleTime in NuoDB 2.4

NodeApplyPingTime

Milliseconds spent applying ping messages from another database process since the last time metrics were provided.

NodeId

The node ID of this process. This value never changes for the life of the process.

NodePingTime

Milliseconds spent pinging another database process since the last time metrics were provided.

NodePostMethodTime

Time spent processing messages from other database processes since the last time metrics were provided.

NodeSocketBufferWriteTime

Milliseconds spent doing inter-process socket writes since the last time metrics were provided.

NodeState

Internal state of this database process at the moment that the metric is provided.

NodeType

Indicates whether this process is a transaction engine, storage manager. This value never changes for the life of the process.

NonChairSplitTime

Time in milliseconds to process index split on non-chairman database processes since the last time metrics were provided.

NumberCores

Number of processor cores. This value never changes for the life of the process.

NumSplits

Number of index splits since the last time metrics were provided.

ObjectFootprint

The estimated, average amount of memory (in bytes) that the process allocated from the heap for each object.

Objects

At the moment the metric is provided, this is the number of atoms.

ObjectsBounced

Number of requests for an atom that are bounced by a database process for various reasons. One example of a bounce is when a TE does not have a requested atom. This is the number of bounced requests since the last time metrics were provided.

ObjectsCreated

Number of atoms created since the last time metrics were provided.

ObjectsDeleted

Number of atoms deleted since the last time metrics were provided.

ObjectsDropped

Number of aged atoms dropped since the last time metrics were provided.

ObjectsDroppedPurged

Number of purged atoms dropped.

SM only

ObjectsExported

Number of atoms exported since the last time metrics were provided.

ObjectsImported

Number of atoms imported since the last time metrics were provided.

ObjectsLoaded

Number of atoms loaded since the last time metrics were provided.

SM only

ObjectsPurged

Number of aged atoms purged during garbage collection since the last time metrics were provided.
GC has marked the object as not being part of a transaction but it is currently available in at least one TE.

SM only

ObjectsRedirected

Number of atom object requests redirected to another database process since the last time metrics were provided.
Obsolete from NuoDB 2.4.

ObjectsReloaded

Number of atoms purged from memory, then loaded again from the archive.

If this (purged object reloading) happens frequently, it may indicate that SM memory is too small for the workload in the TEs.

SM

ObjectsRequested

Number of atom object requests since the last time metrics were provided.

ObjectsSaved

Number of atoms written to a volume by the SM since the last time metrics were provided.

SM only

P-cpuUsage

Percentage of CPU capacity that this process used since the last time metrics were provided. NuoDB calculates this by normalizing the percentage reported by the ps command to the total CPU capacity available. Note that this is not the same as the PercentCpuTime metric in that P-cpuUsage will never be above 100%.

PacketsReceived

Number of socket reads since the last time metrics were provided.

PacketsSent

Number of socket writes since the last time metrics were provided.

PageFaults

Number of system page faults since the last time metrics were provided.

PendingEventsCommitTime

Milliseconds spent waiting for commit event processing since the last time metrics were provided.

PendingInsertWaitTime

Milliseconds spent waiting on a pending insert since the last time metrics were provided.

PendingUpdateStallCount

Number of times this process waited on a pending update since the last time metrics were provided.

PendingUpdateWaitTime

Milliseconds this database process waited on a pending update since the last time metrics were provided.

PercentCpuTime

Percentage of both system kernel and system user time since the last time metrics were provided. This is PercentUserTime+PercentSystemTime.

PercentSystemTime

Percentage of system kernel time since the last time metrics were provided. This is KernelMilliseconds/Milliseconds.

PercentUserTime

Percentage of system user time since the last time metrics were provided. This is UserMilliseconds/Milliseconds.

PlatformCatalogStallCount

Number of times this process had to stall waiting for a block of atom fingerprints from another database process while syncing.

PlatformIndexCheckAcknowledgedTime

Milliseconds spent checking if an index Atom object is acknowledged, that is, it is complete and all other nodes acknowledge directly to the Atom’s node. The purpose of this timer is to measure any stall time that can happen waiting for the operation to complete.

PlatformObjectCheckCompleteTime

Milliseconds since the last time metrics were provided that this process waited for an atom object to become complete. That is, all other database processes acknowledge its existence.

PlatformObjectCheckOpenTime

Milliseconds since the last time metrics were provided that this process waited for an atom object to be opened/loaded. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.

PlatformObjectCheckPopulatedTime

Milliseconds since the last time metrics were provided that this process spent waiting for an atom to be fully populated. The purpose of this metric is to measure any stall time that can happen while waiting for the operation to complete.

ProcessId

The process ID for this process. This value never changes for the life of the process.

PruneAtomsThrottleTime

Total cumulative milliseconds when atom pruning was throttled to give the SM time to catch up writing dirty atoms to the archive.

PurgedObjects

Number of purged objects currently in memory.

SM only.

ReconciliationQueueSize

Number of messages saved in case of failover. This is the number at the moment the metric is provided.

RefactorTXQueueTime

Milliseconds since the last time metrics were provided that this process spent refactoring the transaction queue.

RemoteCommitTime

Milliseconds since the last time metrics were provided that this process spent waiting for commit acknowledgment from SM.

TE

Rollbacks

Number of transaction rollbacks since the last time metrics were provided.

SendQueueSize

Number of outgoing messages in message sequencer to be sent to other database processes. This is the number at the moment the metric is provided.

ServerReceived

Number of socket reads for the platform service since the last time metrics were provided.

ServerSent

Number of socket writes for the platform service since the last time metrics were provided.

SocketBufferBytes

At the moment that metrics are provided, this is the number of bytes buffered for inter-process send operations.

SpillFileDirectoryTime

Milliseconds spent in spill file directory operations

SpillFileReadTime

Milliseconds spent in spill file read.

SpillFileWriteTime

Milliseconds spent in spill file read.

SqlListenerIdleStatementTime

Milliseconds that this TE was waiting on a request from an application client with an open statement since the last time metrics were provided.

TE only

SqlListenerIdleTime

Milliseconds that this TE was waiting on a request from an application client since the last time metrics were provided.

TE only

SqlListenerIdleTransactionTime

Milliseconds that this TE was waiting on a request from an application client with an open transaction since the last time metrics were provided.

TE only

SqlListenerSqlProcTime

Milliseconds spent processing SQL messages, since the last time metrics were provided. Processing time starts from the point when a request is received from an application client, includes all the time needed to process the message, and ends when a response is sent back to the client.

SqlListenerStallTime

Milliseconds since the last time metrics were provided that this process is waiting for something to happen, including time blocked, objects to load, network waits and locks. This time is included in the SqlListenerSqlProcTime metric.

SqlListenerThrottleTime

Time in milliseconds waiting for memory to be freed since the last time metrics were provided.

SqlMsgs

Number of socket writes for the platform service since the last time metrics were provided.

StallPointWaitTime

Milliseconds that this process is waiting in a stall point.

Stalls

Number of thread stalls since the last time metrics were provided.

SyncAtomByteCopied

Size of atoms copied to archive by sync.

SyncAtomsCopied

Atoms copied by archive sync.

SyncAtomsUpToDate

Up-to-date atoms in catalogs whose sync has been started.

SyncCatalogsStarted

Number of catalogs whose sync has been started.

SyncCurrentStorageGroup

Storage Group ID of the current storage group being synced, 0 otherwise.

SyncCurrentTableMigration

Object ID of the table currently being migrated, 0 otherwise.

SyncPointWaitTime

Milliseconds spent waiting for a resource locked by another thread since the last time metrics were provided.

SyncTotalAtoms

Total number of atoms in catalogs whose sync has been started.

SM only

SyncTotalCatalogs

The total catalogs to be synced.

SM only

SyncTotalFingerprintBlocks

Total number of fingerprint blocks to be synced.

SM only

Time

Time at end of monitor period.

TransactionBlockedTime

Since the last time metrics were provided, aggregated time in milliseconds that uncommitted transactions are waiting for another transaction to commit.

A transaction might be waiting because another transaction locked a record or because another transaction commit is supposed to be visible to this transaction.

This metric also includes time spent waiting for metadata changes to commit.

TransactionDependencyRemoteMin

The minimum time, in microseconds, required to resolve transaction dependency.

TEs Only

TransactionDependencyRemoteMax

The maximum time, in microseconds, required to resolve transaction dependency.

TEs Only

TransactionDependencyRemoteMean

The average time, in microseconds, required to resolve transaction dependency.

TEs Only

Updates

Number of records updated since the last time metrics were provided.

TEs Only

UserMilliSeconds

CPU time in milliseconds spent processing user code since the last time metrics were provided.

WaitForDatabaseStallCount

Number of times the transaction engine had to wait for a database atom since the last time metrics were provided.

Obsolete from NuoDB 2.4.

WaitForSplitTime

Milliseconds spent by an index insert waiting for split since the last time metrics were provided.

WriteLoadLevel

This metric indicates the value at the time the metric is provided.

When a storage manager provides this metric, it indicates the number of arriving objects (atoms) per second that exceed the storage manager’s estimated, maximum, sustainable capacity for saving the modified atoms to the archive.

When a transaction engine provides this metric, it indicates the highest WriteLoadLevel provided by any of the database’s storage managers since the last time metrics were provided.

WriteThrottleSetting

At the time metrics are provided, this is the number of microseconds that the transaction engine delays each database write operation. This delay throttles the rate at which the transaction engine sends atom changes to the database’s storage managers.

Throttling occurs only when the rate of atom changes exceeds the rate at which the database’s storage managers can save the modified atoms in the archive.

TEs Only

WriteThrottleTime

Since the last time metrics were provided, this is the time in milliseconds that the transaction engine delayed database write operations.

Transaction engines throttle the rate at which they send atom changes to the database’s storage managers when the rate of atom changes exceeds the rate at which the storage managers can save the modified atoms in the archive.

TEs Only