Log Management ​

CMS Log Management ​

CMS is the cluster management module of oGRAC. It communicates directly with oGRAC database processes, storage-side devices, and CMS processes on other nodes, and is mainly responsible for cluster status query, node addition and removal, heartbeat detection, disk detection, and so on.

From the process perspective, CMS can be divided into CMS SERVER and CMS TOOLS. The logs generated by these two types of processes are stored in different log files. For details, see the "Viewing Logs" section.

Viewing Logs ​

Viewing CMS Configuration ​

Multiple configuration parameters in the CMS configuration file manage the path, level, size, and number of backup files of CMS logs. Therefore, the CMS configuration file path must be clarified: ${CMS_HOME}/cfg/cms.ini

Here, CMS_HOME is specified by the installation script and is an environment variable that can be viewed under the installation user. Execute the echo $CMS_HOME command to view it. The default value is /opt/ograc/cms.

CMS SERVER Log Path ​

The process started by the cms server -start command is called CMS SERVER, which resides on the server. The logs of this process are as follows:

NOTE

1. CMS_LOG is a configuration parameter in the CMS configuration file. In a script‑based default installation, the default value is /opt/ograc/log/cms.

2. The black box log mainly prints the process call stack and register variables when the CMS SERVER process accesses an invalid address space. It has the highest importance and is enabled by default.

3. The run log mainly prints key information (INFO), warning information (WARN), and error information (ERROR) during CMS SERVER running, and is enabled by default.

4. The debug log prints more content and is relatively less important, basically not affecting the running of CMS SERVER. Other aspects are similar to the run log. In a script‑based installation, it is disabled by default.

5. Heartbeat detection log. In a script‑based installation, it is disabled by default.

6. Whether a corresponding log file exists in the log path depends on the log level. For details, see the "Configuring Log Levels" section.

CMS TOOLS Log Path ​

Except for the command that starts the CMS SERVER, the CMS processes started by other CMS-supported commands such as cms res -start/-stop/-add/-del, cms -stat, and cms -iostat are all temporary. Such processes exit after the command finishes execution. They are collectively referred to as CMS TOOLS processes, and have independent run log files.

Log Format ​

The log format is: "time_zone time_information|module_information|session_information|thread_information|log_level>core_log_content"

NOTE

1. Time zone, in the format "UTC+XX:00", indicates the time zone on which the time information is based. For example, "UTC+08:00" indicates China Standard Time, which is 8 hours ahead of Coordinated Universal Time (UTC).

2. Time information, in the format "yyyy-mm-dd hh24:mi:ss.ff3", with milliseconds accurate to 3 digits.

3. Module information, uniformly CMS.

4. Session information, identifying the session ID, which defaults to 00000 under CMS.

5. Thread information, identifying which thread ID printed the log.

6. Log level, including INFO, WARN, and ERROR.

7. Core log content, in two formats. If the current system throws an error code, the format is "OG-XXXX:log information,error code information [file name:line number]". Otherwise, the format is "log information [file name:line number]".

Example: UTC+08:00 2025-11-11 17:26:57.675|CMS|00000|393363|INFO>message count in send que:0 [cms_mes.c:504]

Configuring Log Levels ​

_LOG_LEVEL ​

The CMS log level is controlled by the _LOG_LEVEL parameter in the CMS configuration file. The log level determines which log types are output and which log files are generated. The control method is to convert the decimal value of the parameter to binary, where each bit enables a specific log level — 1 means enabled, and 0 means disabled.

For example, when _LOG_LEVEL = 1, the binary representation has only bit 1 set to 1, indicating that only RUN ERROR logs are enabled. When _LOG_LEVEL = 3, the binary representation has bits 1 and 2 set to 1, meaning that both RUN ERROR and WARN logs are enabled.

NOTE

1. If the _LOG_LEVEL value is greater than 0, the black box log is enabled.

2. In a script-based default installation, the CMS _LOG_LEVEL is set to 7, which indicates that only the run log and the black box log are enabled, and only the corresponding log files are generated.

3. If bit 7 of the _LOG_LEVEL binary representation has a mask of 1, the CMS heartbeat detection log is also enabled.

4. To enable all CMS logs, _LOG_LEVEL can be set to 119, 127, 255, and so on.

Configuring Log Size ​

_LOG_MAX_FILE_SIZE ​

The size of a single log file is limited and configured by the _LOG_MAX_FILE_SIZE parameter of CMS, in bytes, with a default size of 10 MB. When the size limit is reached, the system may either remove older log files or archive them to newly created files.

Configuring the Number of Log Files ​

_LOG_BACKUP_FILE_COUNT ​

When _LOG_BACKUP_FILE_COUNT is 0, log file backup is disabled. If a log file exceeds the size limit, it is deleted. When _LOG_BACKUP_FILE_COUNT is greater than 0, log backup is enabled. Once a log file exceeds the size limit, it is renamed to old_log_filename_timestamp.log. The value of _LOG_BACKUP_FILE_COUNT indicates the maximum number of historical log backup files that can be retained. When this limit is exceeded, the oldest backup file is deleted to keep the total count within the limit. The default value is 10.

Configuration Method ​

1. Open the CMS configuration file cms.ini and modify the corresponding parameter values.

2. Restart CMS SERVER or ignore it. CMS TOOLS does not need to be restarted because it is a temporary process.

cms server -stop
cms server -start &

Database Management Logs ​

Database log management is a core component of the database system. During database running, a large number of run, debug, audit, operation, and black box logs are generated to serve daily database O&M. When a database exception or fault occurs, these logs can be used for problem analysis and locating, database recovery, and other operations.

Log Classification ​

Log-RUN ​

Outputs RUN information during database running. Log path: /opt/ograc/log/ograc/run/ogracd.rlog

Log-DEBUG ​

Outputs DEBUG information during database running. Log path: /opt/ograc/log/ograc/debug/ogracd.dlog

Log-Audit ​

Outputs audit information during database running. Log path: /opt/ograc/log/ograc/audit/ogracd.aud

Log-Operation ​

Records information about user operations on the database. Log path: /opt/ograc/log/ograc/oper/ogsql.olog

Log-Black Box ​

Records exception information related to core occurrences in the database process. This log is enabled by default. To disable it, set the value of the _LOG_LEVEL parameter to 0. Log path: /opt/ograc/log/ograc/blackbox/ogracd.blog

Log-LONGSQL ​

Records SQL information whose execution duration exceeds the threshold in the database to the ogracd.lsql log file. Log path: /opt/ograc/log/ograc/longsql/ogracd.lsql

Configuring Logs ​

oGRAC supports modifying the number of backup files, log size, log level, log permissions, and log attributes for some logs, and supports log deletion and recovery. The modified log parameters take effect after the database restarts.

Basic Log Parameters ​

LOG_HOME: root directory of logs. A change to this parameter changes the upper-level directory of logs such as RUN, DEBUG, audit, operation, black box, and LONGSQL. Default value: /opt/ograc/log/ograc

_LOG_LEVEL: logging level. The scope of this parameter does not include audit logs. The parameter value is the sum of the identifiers for each sub‑log type, and users interact with it in decimal. Whether each sub‑log is enabled is indicated by binary bits (0 for off, 1 for on). Different bit positions represent different log types, from low to high: RUN ERROR, RUN WARNING, RUN INFORMATION, DEBUG ERROR, DEBUG WARNING, DEBUG INFORMATION, LONGSQL LOG, OPER LOG, and so on. When _LOG_LEVEL is 0, it indicates that all logs are disabled. The default value is 7, which means all RUN logs are enabled.

_LOG_MAX_FILE_SIZE: maximum size for a log file. This parameter applies to log types such as log-RUN, log-DEBUG, log-operation, and log-LONGSQL. If a single log file exceeds this parameter value, the log file is backed up, and the backup log name is: ogracd_yyyymmddhhmissfff.rlog. Value range: [1M,4G], in bytes. The default value is 10M.

_LOG_BACKUP_FILE_COUNT: maximum number of backup log files. This parameter applies to log types such as log-RUN, log-DEBUG, log-operation, and log-LONGSQL. If the number of backup log files exceeds this parameter value, the earliest backup log file is deleted to ensure that the total number is within this limit. Value range: [0,128]. The default value is 10.

_LOG_FILE_PERMISSIONS: The write permission of the log owner and group members on backup log files is automatically removed. The constraint scope of this parameter does not include log files that have already taken effect. Value range: [600,777]. The default value is 640.

_LOG_PATH_PERMISSIONS: This parameter can set the permissions of the current layer and upper-layer directories of log-RUN, log-DEBUG, and log-audit. When a log is backed up or a new log file is created, the directory permissions are changed to the value of this parameter. Value range: [700,777]. The default value is 750.

_BLACKBOX_STACK_DEPTH: depth of the call stack that is allowed to be printed in the log when a database process crashes. Value range: [2,40]. The default value is 30.

Configuring Log Parameters ​

_LOG_LEVEL ​

The _LOG_LEVEL parameter is mainly used to control the recording of RUN logs and DEBUG logs. It can control multiple sub-log types simultaneously. Different log types are controlled by Boolean values at different binary bit positions. The parameter value corresponding to each log type can be found in the following table.

Method for modifying the parameter: Connect to the database, generally through an interactive SQL tool, and use the ALTER SYSTEM SET statement to modify it. The modification takes effect immediately, and the effective scope of this operation is the current node. For example, to set the logging level to RUN WARNING and DEBUG INFORMATION, set _LOG_LEVEL to 66 (000001000010). The specific command is: ALTER SYSTEM SET _LOG_LEVEL = 66

Method for querying the parameter: SHOW PARAMETER

Note: After a user configures _LOG_LEVEL to a certain value, the system converts it to binary, pads high-order bits with 0, and finds the corresponding log type according to the corresponding bit, where 0 indicates disabled and 1 indicates enabled. For example, if the parameter value is configured as 7, it is converted to the binary value 000000000111, which indicates that RUN ERROR, RUN WARNING, and RUN INFORMATION logs are recorded.

Association: A change to the _LOG_LEVEL parameter will directly synchronize changes to the values of _LOG_LEVEL_MODE and LONGSQL_LOG_MODE.

_LOG_LEVEL_MODE ​

The _LOG_LEVEL_MODE parameter is mainly used to display and modify the log level represented by _LOG_LEVEL. Its function is to display and modify the current log level in a more user-friendly way. The corresponding relationship can be found in the following table.

Note: The _LOG_LEVEL_MODE parameter is not saved to a file or memory. It is only used as a mapping parameter of _LOG_LEVEL to intuitively display the log level. This parameter does not consider whether reserved bits are enabled; it only considers whether the existing kernel log bits are enabled. USER_DEFINE applies when the value of _LOG_LEVEL does not fall into any of the standard categories: FATAL, DEBUG, WARN, ERROR, or RUN.

Association: Changes to the _LOG_LEVEL_MODE parameter will directly synchronize changes to the value of _LOG_LEVEL.

LONGSQL_LOG_MODE ​

The LONGSQL_LOG_MODE parameter is used to display and modify whether LONGSQL_LOG of the _LOG_LEVEL parameter is enabled. For the corresponding relationship, see the following table.

Note: The LONGSQL_LOG_MODE parameter is not saved to a file or memory. It is used only as a mapping parameter of _LOG_LEVEL to intuitively display whether LONGSQL_LOG is enabled. This parameter does not consider whether reserved bits are enabled; it only reflects whether LONGSQL_LOG is enabled.

Association: Changes to the LONGSQL_LOG_MODE parameter will directly synchronize changes to the value of _LOG_LEVEL.

AUDIT_LEVEL ​

AUDIT_LEVEL is used to control whether audit logs are recorded. For the parameter value corresponding to each log type, see the following table.

Functional syntax: The LOAD, DUMP, EXP, and IMP functional syntaxes are composed of multiple SQL statements. To ensure that they are recorded, AUDIT_LEVEL must be configured to a level that includes the relevant SQL types. For other functional syntaxes, to ensure that they are recorded, AUDIT_LEVEL must be configured to a level that includes DCL.

Note: AUDIT_LEVEL set to 0 indicates that the audit log is disabled. When AUDIT_LEVEL is greater than 0, convert it to binary and take the last four bits. From the most significant bit to the least significant bit, the corresponding audit log types are PL, DML, DCL, and DDL, where 0 indicates disabled and 1 indicates enabled.

Method for modifying the parameter: Connect to the database, generally through an interactive SQL tool, and use the ALTER SYSTEM SET statement to make the modification. The modification takes effect immediately, and the effective scope of this operation is the current node. For example, to set the audit log type to operation logs such as DDL, DCL, DML, and PL, set AUDIT_LEVEL to 15 (00001111). The specific command is: ALTER SYSTEM SET AUDIT_LEVEL = 15

AUDIT_TRAIL_MODE ​

AUDIT_TRAIL_MODE is used to configure the mode of audit log recording.

Parameter value range: NONE, SYSLOG, DB, FILE, and ALL.

NONE: Audit logs are not recorded. SYSLOG: Audit logs are recorded to syslog. DB: Audit logs are recorded to system tables. FILE: Audit logs are recorded to files. ALL: Audit logs are recorded to syslog, system tables, and files.

Default value: FILE

Note: When AUDIT_TRAIL_MODE is set to ALL or DB, it is recommended to enable auto-extension for the system tablespace.

Querying Logs ​

During database running, some operations may have caused errors without affecting database running. However, the data in the database may have become inconsistent. It is recommended to check the database run logs monthly to detect potential risks in a timely manner.

Log Path ​

Log Classification ​

Log-RUN ​

When a cluster fault occurs, you can use the system log to promptly locate the cause of the fault and formulate a cluster recovery plan accordingly.

Database log format: for example, UTC+08:00 2025-11-21 09:15:17.059|COMMON|00357|2089731|INFO>[DB] Finish to truncate table LTT_ANALYZE_JOB3, ret:0 [knl_interface.c:8330]

• Event occurrence time

• Module

• Session ID

• Thread ID

• Log level

• Log content

Log-DEBUG ​

Compared with the run log, the debug log is more from an internal perspective, recording the specific steps and states of program execution so that developers can locate issues accurately.

DEBUG log format: for example, UTC+08:00 2025-01-04 17:19:29.857|SERVER|00048|1795|INFO>Succeed to transform rule:og_transf_subquery_rewrite [ogsql_transform.c:249]

• Event occurrence time

• Module

• Session ID

• Thread ID

• Log level

• Log content

Log-Audit ​

Depending on the audit level configured by the user, different audit content is recorded in the log file. After the audit function is enabled, a large amount of audit log content is continuously generated, which occupies a large amount of disk space. It is recommended to configure the relevant audit log maintenance policy based on the available disk space. In the audit log system table, the maximum length of a single audit log record is 7000 bytes, and content exceeding the limit is truncated. The audit log adds one record at the prepare stage and one at the execute stage respectively. The prepare record indicates whether the SQL parsing is correct, and the execute record indicates whether the execution result is correct. In the audit log file, the maximum length of a single audit log record is MIN(64M, _AGENT_STACK_SIZE), and content exceeding the limit is truncated.

Audit log format: for example, UTC+08:00 2025-11-21 11:17:38.159 LENGTH: "135" SESSIONID:[3] "415" STMTID:[0] "" USER:[3] "SYS" HOST:[9] "127.0.0.1" ACTION:[10] "DISCONNECT" RETURNCODE:[8] "OG-00000" SQLTEXT:[0] ""

• Event occurrence time

• Audit content length

• Audit content

  Keyword: [keyword length]"content". SESSIONID: session ID. STMTID: statement ID. USER: user name. HOST: host address. ACTION: event type, including operations such as LOGIN, FREE_STMT, PREPARE, and EXECUTE. RETURNCODE: return code. OG-00000 indicates success, and other values are error codes. SQLTEXT: SQL statement. For operations such as PL, DDL, and DCL, the SQL statement is recorded only in the PREPARE phase and not in the EXECUTE phase. You can use SESSIONID and STMTID to associate with the PREPARE phase to query the SQL statement. For DML operations, the SQL statement is recorded in both the PREPARE and EXECUTE phases.

Maintaining Audit Logs ​

Scenario: After the audit function is enabled, a large number of audit logs are generated continuously, occupying a large amount of disk space. Users can configure the maintenance policy for audit logs based on disk capacity.

Procedure:
Step 1: Log in to the database as the dba user.

Step 2: Configure automatic deletion of audit logs. The default threshold for a single audit file is 10M, with a value range of [1M, 4G], controlled by the _AUDIT_MAX_FILE_SIZE parameter. The number of audit files is controlled by _AUDIT_BACKUP_FILE_COUNT, with a default value of 10 and a value range of [0, 128]. When the number of audit files exceeds the maximum value, the system deletes the earliest audit file. Modification commands: ALTER SYSTEM SET _AUDIT_MAX_FILE_SIZE=20M; ALTER SYSTEM SET _AUDIT_BACKUP_FILE_COUNT=20;

Query commands: SELECT NAME,RUNTIME_VALUE FROM SYS.DV_PARAMETERS WHERE NAME='_AUDIT_MAX_FILE_SIZE'; SELECT NAME,RUNTIME_VALUE FROM SYS.DV_PARAMETERS WHERE NAME='_AUDIT_BACKUP_FILE_COUNT';

Log-Operation ​

An operation log refers to the log generated when a user logs in to the database through ogsql and performs operations. If a database fault occurs, the operation log file can be used to analyze what operations the user performed on the database, reproduce the fault scenario, and resolve the fault.

Operation log format: for example, 2025-11-21 19:09:00.640|ogsql|SELECT * FROM DV_DRC_RES_RATIO WHERE DRC_RESOURCE='GLOBAL_TXN'

• Event occurrence time

• ogsql tool

• Operation command

Log-Black Box ​

When a database process encounters an exception that leads to termination, the black box log records basic exception information.

Black box log format: (1) Regular exception information: event occurrence, exception type, exception process, exception name, exception thread, process that sent the exception signal, OS user that sent the exception signal, exception address, platform information, version information. (2) Exception call stack. (3) Register information. (4) Process maps information at the time of core dump. (5) Thread stack information at the time of core dump. (6) DMS trace information of active DMS event threads at the time of core dump, printed under shared storage. (7) SQL session content. (8) Executed SQL and bind parameter information. (9) Information about important kernel threads and each worker thread.

Log-LONGSQL ​

When a statement's execution time exceeds the threshold, the slow query log records it. The threshold is controlled by the LONGSQL_TIMEOUT parameter, and the default value is 10s. It is disabled by default and is recommended to be enabled during tuning.

Slow query log format: For a DN node, for example: 2025-11-21 19:09:00|USER_EXECUTE|3|127.0.0.1|5679201|"NULL"|65788999|67990178890|"SELECT NAME,GOALS FROM USER.CLASS"

Note: 19:09:00: current time. USER_EXECUTE: the stage the program is currently in. 3: node ID of the DN. 127.0.0.1: client IP. 5679201: elapsed time of the DN node in this stage (microseconds). NULL: parameters of the SQL statement. 65788999: ID of the SQL statement. 67990178890: hash value of the execution plan. SELECT NAME,GOALS FROM USER.CLASS: the specific SQL statement sent to the DN node.

Cleaning Up Logs ​

During database running, a large number of database run logs are generated, consuming a large amount of disk space. It is recommended to keep only the logs from the most recent month and clean up expired log files.

Database Log Cleanup Mechanism ​

When the file size of a run, debug, audit, operation, black box, slow query, or other log file reaches _LOG_MAX_FILE_SIZE (default value: 10 MB), the log file is backed up and a new log file is generated.

When the number of backup log files in the system reaches _LOG_BACKUP_FILE_COUNT (default value: 10), the system automatically cleans up the earliest backup log file.

Alarm Logs ​

The alarm log is also part of the database run log, but it is stored separately due to its importance.

Scope ​

Warnings are issued for some important operations in the availability and security domains of the database.

1. Availability

Note: The module name is used in the log format. Users can search for alarm logs in a specified domain by module name.

2. Security

Log Path ​

ALARM_LOG_DIR ​

The alarm log path can be configured through the ALARM_LOG_DIR parameter in the database configuration file.

1. When the parameter is configured and valid, the alarm log path is ALARM_LOG_DIR/INSTANCENAME_alarm.log.

2. If the parameter is not configured, the default directory is used. The alarm log path is database_data_directory/log/INSTANCENAME_alarm.log.

Note: INSTANCENAME is the instance name, configured by the database INSTANCE_NAME parameter. When the default script is used for installation, its value defaults to ograc.

Log Level ​

As long as the database parameter _LOG_LEVEL is greater than 0, the alarm log is enabled, and it is enabled by default.

Log Format ​

The format is: "time_information|warn_module_ID|warn_module_name|DN|database_instance_name|{'component-name':'DN','datanode-name':'database_instance_name',alarm_content|alarm_category_number}"

Other Logs ​

DSS Logs ​

DSS Log Types ​

-- RUN Log
Prints DSS RUN-level information when the database is in DSS mode. If a DSS runtime failure occurs and RUN-level logging is enabled, please check dsscmd.rlog and dssinstance.rlog. Log directory: defaults to $DSS_HOME/log/run.

-- DEBUG Log
Prints DSS DEBUG-level information when the database is in DSS mode. If a DSS runtime failure occurs and DEBUG-level logging is enabled, please check dsscmd.dlog and dssinstance.dlog. Log directory: defaults to $DSS_HOME/log/debug.

-- Operation Log
Prints DSS OPER-level information when the database is in DSS mode. If a DSS runtime failure occurs and OPER-level logging is enabled, please check dsscmd.olog. Log directory: defaults to $DSS_HOME/log/oper.

-- Audit Log
Prints DSS operation audit data or metadata modification and query information when the database is in DSS mode. Log directory: defaults to $DSS_HOME/log/audit.

-- Black Box Log
Prints basic exception information when the dssserver process terminates abnormally in DSS mode. The black box log is enabled by default. To disable it, set _LOG_LEVEL=0 and restart dssserver for the change to take effect. Log directory: defaults to $DSS_HOME/log/blackbox.

DSS Log Levels ​

DSS uses the _LOG_LEVEL parameter to control the logging of logs except audit logs. To enable multiple log types, set this parameter to the sum of the corresponding values for each desired log type. The mapping of log types to their corresponding values is listed in Table 1.

Table 1 _LOG_LEVEL values corresponding to log types