Query logging can be enabled by setting the query_log directive in the searchd section of the configuration file.

Queries can also be sent to syslog by setting syslog instead of a file path. In this case, all search queries will be sent to the syslog daemon with LOG_INFO priority, prefixed with [query] instead of a timestamp. Only the plain log format is supported for syslog.

query_log example:

searchd {
...
    query_log = /var/log/query.log
    query_log_format = sphinxql # default
...
}

Logging format

Two query log formats are supported:

• sphinxql (default): Logs in SQL format. It also provides an easy way to replay logged queries.
• plain: Logs full-text queries in a simple text format. Recommended if most of your queries are primarily full-text, or if you don't care about non-full-text components of your queries, such as filtering by attributes, sorting, grouping, etc. Queries logged in the plain format cannot be replayed.

To switch between the formats, you can use the searchd setting query_log_format.

SQL log format

The SQL log format is the default setting. In this mode, Manticore logs all successful and unsuccessful select queries. Requests sent as SQL or via the binary API are logged in the SQL format, but JSON queries are logged as is. This type of logging only works with plain log files and does not support the 'syslog' service for logging.

query_log_format example:

query_log_format = sphinxql # default

The features of the Manticore SQL log format compared to the plain format include:

• Full statement data is logged where possible.
• Errors and warnings are logged.
• The query log can be replayed.
• Additional performance counters (currently, per-agent distributed query times) are logged.
• Each log entry is a valid Manticore SQL/JSON statement that reconstructs the full request, except if the logged request is too large and needs to be shortened for performance reasons.
• JSON requests and additional messages, counters, etc., are logged as comments.

sphinxql log entries example:

/* Sun Apr 28 12:38:02.808 2024 conn 2 (127.0.0.1:53228) real 0.000 wall 0.000 found 0 */ SELECT * FROM test WHERE MATCH('test') OPTION ranker=proximity;
/* Sun Apr 28 12:38:05.585 2024 conn 2 (127.0.0.1:53228) real 0.001 wall 0.001 found 0 */ SELECT * FROM test WHERE MATCH('test') GROUP BY channel_id OPTION ranker=proximity;
/* Sun Apr 28 12:40:57.366 2024 conn 4 (127.0.0.1:53256) real 0.000 wall 0.000 found 0 */  /*{
    "index" : "test",
    "query":
    {
        "match":
        {
            "*" : "test"
        }
    },
    "_source": ["f"],
    "limit": 30
} */

Plain log format

With the plain log format, Manticore logs all successfully executed search queries in a simple text format. Non-full-text parts of queries are not logged. JSON queries are logged as flattened to a single line.

query_log_format example:

query_log_format = plain

The log format is as follows:

[query-date] real-time wall-time [match-mode/filters-count/sort-mode total-matches (offset,limit) @groupby-attr] [table-name] {perf-stats} query

where:

• real-time is the time from the start to the finish of the query.
• wall-time is similar to real-time, but excludes time spent waiting for agents and merging result sets from them.
• perf-stats includes CPU/IO stats when Manticore is started with --cpustats (or it was enabled via SET GLOBAL cpustats=1) and/or --iostats (or it was enabled via SET GLOBAL iostats=1):
• ios is the number of file I/O operations carried out;
• kb is the amount of data in kilobytes read from the table files;
• ms is the time spent on I/O operations.

• cpums is the time in milliseconds spent on CPU processing the query.

• match-mode can have one of the following values:

• "all" for SPH_MATCH_ALL mode;
• "any" for SPH_MATCH_ANY mode;
• "phr" for SPH_MATCH_PHRASE mode;
• "bool" for SPH_MATCH_BOOLEAN mode;
• "ext" for SPH_MATCH_EXTENDED mode;
• "ext2" for SPH_MATCH_EXTENDED2 mode;

• "scan" if the full scan mode was used, either by being specified with SPH_MATCH_FULLSCAN or if the query was empty.

• sort-mode can have one of the following values:

• "rel" for SPH_SORT_RELEVANCE mode;
• "attr-" for SPH_SORT_ATTR_DESC mode;
• "attr+" for SPH_SORT_ATTR_ASC mode;
• "tsegs" for SPH_SORT_TIME_SEGMENTS mode;
• "ext" for SPH_SORT_EXTENDED mode.

Note: the SPH* modes are specific to the sphinx legacy interface. SQL and JSON interfaces will log, in most cases, ext2 as match-mode and ext and rel as sort-mode.

Query log example:

[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5] test
[Fri Jun 29 21:17:58 2021] 0.004 sec [all/0/rel 35254 (0,20)] [lj] [ios=6 kb=111.1 ms=0.5 cpums=0.3] test
[Sun Apr 28 15:09:38.712 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20)] [test] test
[Sun Apr 28 15:09:44.974 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,20) @channel_id] [test] test
[Sun Apr 28 15:24:32.975 2024] 0.000 sec 0.000 sec [ext2/0/ext 0 (0,30)] [test] {     "index" : "test",     "query":     {         "match":         {             "*" : "test"         }     },     "_source": ["f"],     "limit": 30 }

Logging only slow queries

By default, all queries are logged. If you want to log only queries with execution times exceeding a specified limit, the query_log_min_msec directive can be used.

The expected unit of measurement is milliseconds, but time suffix expressions can also be used.

query_log_min_msec example:

searchd {
...
    query_log = /var/log/query.log
    query_log_min_msec  = 1000
    # query_log_min_msec  = 1s
...
}

Log file permission mode

By default, the searchd and query log files are created with permission 600, so only the user under which Manticore is running and root can read the log files. The query_log_mode option allows setting a different permission. This can be helpful for allowing other users to read the log files (for example, monitoring solutions running on non-root users).

query_log_mode example:

searchd {
...
    query_log = /var/log/query.log
    query_log_mode = 666
...
}

By default, Manticore search daemon logs all runtime events in a searchd.log file in the directory where searchd was started. In Linux by default, you can find the log at /var/log/manticore/searchd.log.

The log file path/name can be overridden by setting log in the searchd section of the configuration file.

searchd {
...
    log = /custom/path/to/searchd.log
...
}

• You can also use syslog as the file name. In this case, events will be sent to your server's syslog daemon.
• In some cases, you might want to use /dev/stdout as the file name. In this case, on Linux, Manticore will simply output the events. This can be useful in Docker/Kubernetes environments.

Binary logging serves as a recovery mechanism for Real-Time table data, as well as attribute updates for plain tables that would otherwise only be stored in RAM until a flush occurs. When binary logs are enabled, searchd records each transaction to the binlog file and utilizes it for recovery following an unclean shutdown. During a clean shutdown, RAM chunks are saved to disk, and all binlog files are subsequently unlinked.

Disabling binary logging

By default, binary logging is enabled. On Linux systems, the default location for binlog.* files is /var/lib/manticore/data/.
In RT mode, binary logs are stored in the data_dir folder, unless specified otherwise.

To disable binary logging, set binlog_path to empty:

searchd {
...
    binlog_path = # disable logging
...

Disabling binary logging can lead to better performance for Real-Time tables, but it also puts their data at risk.

You can use the following directive to set a custom path:

searchd {
...
    binlog_path = /var/data
...

Operations

When logging is enabled, each transaction committed to an RT table is written to a log file. After an unclean shutdown, logs are automatically replayed upon startup, recovering any logged changes.

Log size

During normal operation, a new binlog file is opened whenever the binlog_max_log_size limit is reached. Older, closed binlog files are retained until all transactions stored in them (from all tables) are flushed as a disk chunk. Setting the limit to 0 essentially prevents the binlog from being unlinked while searchd is running; however, it will still be unlinked upon a clean shutdown. By default, there is no limit to the log file size.

binlog_max_log_size = 16M

Binary flushing strategies

There are 3 different binlog flushing strategies, controlled by the binlog_flush directive:

• 0 - Flush and sync every second. This provides the best performance, but up to 1 second's worth of committed transactions may be lost in the event of a server crash or an OS/hardware crash.
• 1 - Flush and sync every transaction. This has the worst performance, but guarantees that every committed transaction's data is saved.
• 2 - Flush every transaction and sync every second. This offers good performance, and every committed transaction is guaranteed to be saved in the case of a server crash. However, up to 1 second's worth of committed transactions may be lost in the event of an OS/hardware crash.

The default mode is to flush every transaction and sync every second (mode 2).

searchd {
...
    binlog_flush = 1 # ultimate safety, low speed
...
}

Recovery

During recovery after an unclean shutdown, binlogs are replayed, and all logged transactions since the last good on-disk state are restored. Transactions are checksummed, so in case of binlog file corruption, garbage data will not be replayed; such a broken transaction will be detected and will stop the replay.

Flushing RT RAM chunks

Intensive updates to a small RT table that fully fits into a RAM chunk can result in an ever-growing binlog that can never be unlinked until a clean shutdown. Binlogs essentially serve as append-only deltas against the last known good saved state on disk, and they cannot be unlinked unless the RAM chunk is saved. An ever-growing binlog is not ideal for disk usage and crash recovery time. To address this issue, you can configure searchd to perform periodic RAM chunk flushes using the rt_flush_period directive. With periodic flushes enabled, searchd will maintain a separate thread that checks whether RT table RAM chunks need to be written back to disk. Once this occurs, the respective binlogs can be (and are) safely unlinked.

searchd {
...
    rt_flush_period = 3600 # 1 hour
...
}

The default RT flush period is set to 10 hours.

It's important to note that rt_flush_period only controls the frequency at which checks occur. There are no guarantees that a specific RAM chunk will be saved. For example, it doesn't make sense to regularly re-save a large RAM chunk that only receives a few rows worth of updates. Manticore automatically determines whether to perform the flush using a few heuristics.

When you use the official Manticore docker image, the server log is sent to /dev/stdout which can be viewed from host with:

docker logs manticore

The query log can be diverted to the Docker log by passing the variable QUERY_LOG_TO_STDOUT=true.

The log folder is the same as in the case of the Linux package, set to /var/log/manticore. If desired, it can be mounted to a local path to view or process the logs.

Manticore Search accepts the USR1 signal for reopening server and query log files.

The official DEB and RPM packages install a Logrotate configuration file for all files in the default log folder.

A simple logrotate configuration for log files looks like:

/var/log/manticore/*.log {
       weekly
       rotate 10
       copytruncate
       delaycompress
       compress
       notifempty
       missingok
}

FLUSH LOGS

mysql> FLUSH LOGS;
Query OK, 0 rows affected (0.01 sec)

Additionally, the FLUSH LOGS SQL command is available, which works the same way as the USR1 system signal. It initiates the reopening of searchd log and query log files, allowing you to implement log file rotation. The command is non-blocking (i.e., it returns immediately).