Troubleshooting with nuocmd

This section documents the use of nuocmd to retrieve diagnostic information and how to generate a support package file. When a NuoDB process terminates unexpectedly, the core file gets written to a core subdirectory in the NuoDB log directory. To generate a core file, see get diagnose-info and get core-file. Please send all the core files to NuoDB Support for analysis. For assistance, contact NuoDB Support.

Collecting Diagnostics

The following nuocmd commands are used to collect information:

For more information on nuocmd and other command line tools, see Command Line Tools.

To enable the auto-completion and verify the commands before the execution, type bash in the terminal command window and press the ⏎ or Enter key.

get diagnose-info

The nuocmd get diagnose-info collects admin logs, admin configuration, system binaries, and cores into a single support package (.zip) file.

Example
nuocmd get diagnose-info --output-dir /tmp/nuodiag --include-cores

get server-logs

The nuocmd get server-logs creates a zip file containing the Admin Service log files in the specified location.

Example
nuocmd get server-logs --output /tmp

get log-messages

The nuocmd get log-messages streams log messages for the specified process, database, or domain.

Example
nuocmd get log-messages --log-options msgs
The --log-options parameter is required. For a complete list of log parameters, see Logging Options.

get core-file

The nuocmd get core-file downloads the core file for a running database process.

Example
nuocmd get core-file --start-id 1 --output-dir /tmp/
To get the value for the --start-id parameter, run the nuocmd show domain command and then note the start ID value displayed for the SM or TE process for which you want to generate a core file. If this parameter is not specified, permission to write to the directory where you executed the command is required.

get database-connectivity

The nuocmd get database-connectivity provides connectivity information for the specified database in JSON format.

Example
nuocmd get database-connectivity --db-name test
{
  "0": {
    "1": {
      "lastAckDeltaInMilliSeconds": 746,
      "lastMsgDeltaInMilliSeconds": 755
    },
    "2": {
      "lastAckDeltaInMilliSeconds": 460,
      "lastMsgDeltaInMilliSeconds": 755
    },
    "3": {
      "lastAckDeltaInMilliSeconds": 860,
      "lastMsgDeltaInMilliSeconds": 755
    }
  },
  "1": {
    "0": {
    "lastAckDeltaInMilliSeconds": 802,
    "lastMsgDeltaInMilliSeconds": 794
    },
    "2": {
      "lastAckDeltaInMilliSeconds": 508,
      "lastMsgDeltaInMilliSeconds": 794
    },
    "3": {
      "lastAckDeltaInMilliSeconds": 907,
      "lastMsgDeltaInMilliSeconds": 794
    }
  },
...

show database-connectivity

The nuocmd show database-connectivity displays a connectivity graph of the specified database.

Example
nuocmd show database-connectivity --db-name test --with-node-ids
     1    2    3    4
1             27s
2             27s
3    ?    ?    ?    ?
4             32s

Legend:
X: node at this row does not consider node at this column a peer
?: node at this row could not be queried for connectivity information
!: node at this row does not have expected metadata for node at this column
[0-9]+[hms]: time since node at this row last heard from node at this column

Workload Capture

A workload capture records a sequence of SQL commands and parameters sent by a database client application during a specific period of time. Workload captures are useful for reproducing and resolving performance issues quickly. They are created only under the guidance of the NuoDB Support team when it is determined that a workload capture will expedite support ticket resolution.

To reproduce and analyze performance issues, the following are often required:

  • A copy of the database before the workload starts.

  • A workload capture to record the SQL commands being executed.

Create a Copy of the Database

Create a copy of the database using an online backup method, such as Hot Copy Backup or Hot Snap Backup.

To ensure that the workload capture is consistent with the copy of the database, pause the transactions before taking the backup using nuocmd pause transactions. After starting the workload capture, resume transactions using nuocmd resume transactions. For more information, see nuocmd Reference.

Capture the Workload

To capture a workload file, use nuocmd show log-messages.

For more information, see nuocmd Reference.

Steps for Creating a Database Backup and Workload Capture

The nuocmd pause transactions operation will impact application performance and will appear as an outage to active users. Pausing transactions is recommended only for non-production environments, unless an agreement is reached beforehand with users that a short outage is acceptable.

Assumption: The client application is already running.

  1. Pause transactions on all TEs.

    nuocmd pause transactions --db-name <db-name> --timeout <timeout>

    nuocmd pause transactions runs synchronously. It waits for all the active transactions on TEs to finish and blocks new transactions from starting. The --timeout option specifies the duration for which transactions on TEs remain paused. Transactions resume when the timeout expires or when nuocmd resume transactions is executed. If a longer pause is required, run the command with a longer timeout value. The default timeout value is 30s. For more information, see nuocmd pause transactions.

  2. Take a Hot Snap online backup of the database.

    For more information, see Hot Snap Backup.

  3. Start the workload capture.

     nuocmd show log-messages --db-name <db-name> --log-options sql-workload,sql-params > nuodb-sql-workload.log

    In this example, the SQL workload is captured in the file, nuodb-sql-workload.log.

  4. Resume transactions on all TEs.

    nuocmd resume transactions --db-name <db-name>

    The command resumes all the transactions on TEs. For more information, see nuocmd resume transactions.

  5. After the client application workload has ended or enough application SQL has been captured, stop the workload capture.