¶ 26

Changelog

Version 6.3.0 (preparing for release)

While 6.3.0 is being prepared for release, use the dev version which includes all the below changes - https://mnt.cr/dev/nightly

Major changes

Minor changes

Breaking changes and deprecations

Bug fixes

Version 6.2.12

Released: August 23rd 2023

Version 6.2.12 continues the 6.2 series and addresses issues discovered after the release of 6.2.0.

Bugfixes

Minor changes

MCL

Version 6.2.0

Released: August 4th 2023

Major changes

Minor changes

⚠️ Breaking changes

Bugfixes

Version 6.0.4

Released: March 15 2023

New features

Bugfixes

Version 6.0.2

Released: Feb 10 2023

Bugfixes

Version 6.0.0

Released: Feb 7 2023

Starting with this release, Manticore Search comes with Manticore Buddy, a sidecar daemon written in PHP that handles high-level functionality that does not require super low latency or high throughput. Manticore Buddy operates behind the scenes, and you may not even realize it is running. Although it is invisible to the end user, it was a significant challenge to make Manticore Buddy easily installable and compatible with the main C++-based daemon. This major change will allow the team to develop a wide range of new high-level features, such as shards orchestration, access control and authentication, and various integrations like mysqldump, DBeaver, Grafana mysql connector. For now it already handles SHOW QUERIES, BACKUP and Auto schema.

This release also includes more than 130 bug fixes and numerous features, many of which can be considered major.

Major Changes

Minor changes

We are not planning to make the old forms obsolete, but to ensure compatibility with the documentation, we recommend changing the names in your application. What will be changed in a future release is the "index" to "table" rename in the output of various SQL and JSON commands.

If you are running a replication cluster, you'll need to run ALTER TABLE <table name> REBUILD SECONDARY on all the nodes or follow this instruction with just change: run the ALTER .. REBUILD SECONDARY instead of the OPTIMIZE.

Bugfixes

Version 5.0.2

Released: May 30th 2022

Bugfixes

Version 5.0.0

Released: May 18th 2022

Major new features

Previously (note the response time): ```bash $ time curl -v -sX POST http://localhost:9318/bulk -H "Content-Type: application/x-ndjson" --data '{"insert": {"index": "user", "doc": {"name":"Prof. Matt Heaney IV","email":"ibergnaum@yahoo.com","description":"Tempora ullam eaque consequatur. Vero aut minima ut et ut omnis officiis vel. Molestiae quis voluptatum sint numquam.","age":15,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Prof. Boyd McKenzie","email":"carlotta11@hotmail.com","description":"Blanditiis maiores odio corporis eaque illum. Aut et rerum iste. Neque et ullam quisquam officia dignissimos quo cumque.","age":84,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Mr. Johann Smith","email":"stiedemann.tristin@ziemann.com","description":"Temporibus amet magnam consequatur omnis consequatur illo fugit. Debitis natus doloremque est tempore deserunt vero. Harum eos corrupti nemo ut.","age":89,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Hector Pouros","email":"hickle.mafalda@hotmail.com","description":" as voluptatem inventore sit. Aliquam fugit perferendis est id aut odio et sapiente.","age":64,"active":1}}}' * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 9318 (#0) > POST /bulk HTTP/1.1 > Host: localhost:9318 > User-Agent: curl/7.47.0 > Accept: */* > Content-Type: application/x-ndjson > Content-Length: 1025 > Expect: 100-continue > * Done waiting for 100-continue * We are completely uploaded and fine < HTTP/1.1 200 OK < Server: 4.2.0 15e927b@211223 release (columnar 1.11.4 327b3d4@211223) < Content-Type: application/json; charset=UTF-8 < Content-Length: 434 < * Connection #0 to host localhost left intact {"items":[{"insert":{"_index":"user","_id":2811798918248005633,"created":true,"result":"created","status":201}},{"insert":{"_index":"user","_id":2811798918248005634,"created":true,"result":"created","status":201}},{"insert":{"_index":"user","_id":2811798918248005635,"created":true,"result":"created","status":201}},{"insert":{"_index":"user","_id":2811798918248005636,"created":true,"result":"created","status":201}}],"errors":false} real 0m1.022s user 0m0.001s sys 0m0.010s ``` Now: ```bash $ time curl -v -sX POST http://localhost:9318/bulk -H "Content-Type: application/x-ndjson" --data '{"insert": {"index": "user", "doc": {"name":"Prof. Matt Heaney IV","email":"ibergnaum@yahoo.com","description":"Tempora ullam eaque consequatur. Vero aut minima ut et ut omnis officiis vel. Molestiae quis voluptatum sint numquam.","age":15,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Prof. Boyd McKenzie","email":"carlotta11@hotmail.com","description":"Blanditiis maiores odio corporis eaque illum. Aut et rerum iste. Neque et ullam quisquam officia dignissimos quo cumque.","age":84,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Mr. Johann Smith","email":"stiedemann.tristin@ziemann.com","description":"Temporibus amet magnam consequatur omnis consequatur illo fugit. Debitis natus doloremque est tempore deserunt vero. Harum eos corrupti nemo ut.","age":89,"active":1}}} {"insert": {"index": "user", "doc": {"name":"Hector Pouros","email":"hickle.mafalda@hotmail.com","description":" as voluptatem inventore sit. Aliquam fugit perferendis est id aut odio et sapiente.","age":64,"active":1}}}' * Trying 127.0.0.1... * Connected to localhost (127.0.0.1) port 9318 (#0) > POST /bulk HTTP/1.1 > Host: localhost:9318 > User-Agent: curl/7.47.0 > Accept: */* > Content-Type: application/x-ndjson > Content-Length: 1025 > Expect: 100-continue > < HTTP/1.1 100 Continue < Server: 4.2.1 63e5749@220405 dev < Content-Type: application/json; charset=UTF-8 < Content-Length: 0 * We are completely uploaded and fine < HTTP/1.1 200 OK < Server: 4.2.1 63e5749@220405 dev < Content-Type: application/json; charset=UTF-8 < Content-Length: 147 < * Connection #0 to host localhost left intact {"items":[{"bulk":{"_index":"user","_id":2811798919590182916,"created":4,"deleted":0,"updated":0,"result":"created","status":201}}],"errors":false} real 0m0.015s user 0m0.005s sys 0m0.004s ```

Minor changes

⚠️ Other minor breaking changes

New packages

The new structure is:
- manticore - deb/rpm meta package which installs all the above as dependencies
- manticore-server-core - searchd and everything to run it alone
- manticore-server - systemd files and other supplementary scripts
- manticore-tools - indexer, indextool and other tools
- manticore-common - default configuration file, default data directory, default stopwords
- manticore-icudata, manticore-dev, manticore-converter didn't change much
- .tgz bundle which includes all the packages

Bugfixes

Version 4.2.0, Dec 23 2021

Major new features

Pseudo sharding on vs off in 4.2.0

4.0.2

4.0.2
It takes **48 seconds to insert 1M PQ rules** and **406 seconds to REPLACE just 40K** in 10K batches. 
root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"

Wed Dec 22 10:24:30 AM CET 2021
Wed Dec 22 10:25:18 AM CET 2021
+----------+
| count(*) |
+----------+
|  1000000 |
+----------+

root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 30000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:26:23 AM CET 2021
Wed Dec 22 10:26:27 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql

real    6m46.195s
user    0m0.035s
sys 0m0.008s
#### 4.2.0
4.2.0
It takes **34 seconds to insert 1M PQ rules** and **23 seconds to REPLACE them** in 10K batches. 
root@perf3 ~ # mysql -P9306 -h0 -e "drop table if exists pq; create table pq (f text, f2 text, j json, s string) type='percolate';"; date; for m in `seq 1 1000`; do (echo -n "insert into pq (id,query,filters,tags) values "; for n in `seq 1 1000`; do echo -n "(0,'@f (cat | ( angry dog ) | (cute mouse)) @f2 def', 'j.json.language=\"en\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; [ $n != 1000 ] && echo -n ","; done; echo ";")|mysql -P9306 -h0; done; date; mysql -P9306 -h0 -e "select count(*) from pq"

Wed Dec 22 10:06:38 AM CET 2021
Wed Dec 22 10:07:12 AM CET 2021
+----------+
| count(*) |
+----------+
|  1000000 |
+----------+

root@perf3 ~ # date; (echo "begin;"; for offset in `seq 0 10000 990000`; do n=0; echo "replace into pq (id,query,filters,tags) values "; for id in `mysql -P9306 -h0 -NB -e "select id from pq limit $offset, 10000 option max_matches=1000000"`; do echo "($id,'@f (tiger | ( angry bear ) | (cute panda)) @f2 def', 'j.json.language=\"de\"', '{\"tag1\":\"tag1\",\"tag2\":\"tag2\"}')"; n=$((n+1)); [ $n != 10000 ] && echo -n ","; done; echo ";"; done; echo "commit;") > /tmp/replace.sql; date
Wed Dec 22 10:12:31 AM CET 2021
Wed Dec 22 10:14:00 AM CET 2021
root@perf3 ~ # time mysql -P9306 -h0 < /tmp/replace.sql

real    0m23.248s
user    0m0.891s
sys 0m0.047s

Minor changes

Breaking changes

Bugfixes

Version 4.0.2, Sep 21 2021

Major new features

- read operations (e.g. SELECTs, replication) are performed with snapshots - operations that just change internal index structure without modifying schema/documents (e.g. merging RAM segments, saving disk chunks, merging disk chunks) are performed with read-only snapshots and replace the existing chunks in the end - UPDATEs and DELETEs are performed against existing chunks, but for the case of merging that may be happening the writes are collected and are then applied against the new chunks - UPDATEs acquire an exclusive lock sequentially for every chunk. Merges acquire a shared lock when entering the stage of collecting attributes from the chunk. So at the same time only one (merge or update) operation has access to attributes of the chunk. - when merging gets to the phase, when it needs attributes it sets a special flag. When UPDATE finishes, it checks the flag, and if it's set, the whole update is stored in a special collection. Finally, when the merge finishes, it applies the updates set to the newborn disk chunk. - ALTER runs via an exclusive lock - replication runs as a usual read operation, but in addition saves the attributes before SST and forbids updates during the SST

Minor changes

3.6.0

3.6.0 
time curl -X POST -d '{"update":{"index":"idx","id":4611686018427387905,"doc":{"mode":0}}}' -H "Content-Type: application/x-ndjson" http://127.0.0.1:6358/json/bulk

real    0m43.783s
user    0m0.008s
sys     0m0.007s
#### 4.0.2
4.0.2 
time curl -X POST -d '{"update":{"index":"idx","id":4611686018427387905,"doc":{"mode":0}}}' -H "Content-Type: application/x-ndjson" http://127.0.0.1:6358/json/bulk

real    0m0.006s
user    0m0.004s
sys     0m0.001s

Breaking changes

Migration from Manticore 3

Bugfixes

Version 3.6.0, May 3rd 2021

Maintenance release before Manticore 4

Major new features

Minor changes

Optimizations

Bugfixes

Breaking changes:

Deprecations

Version 3.5.4, Dec 10 2020

New Features

Minor Changes

Deprecations

Bugfixes

Version 3.5.2, Oct 1 2020

New features

Minor changes

Deprecations:

Docker

The official Docker image is now based on Ubuntu 20.04 LTS

Packaging

Besides the usual manticore package, you can also install Manticore Search by components:

Bugifixes

  1. Commit 2a47 Crash of daemon at grouper at RT index with different chunks
  2. Commit 57a1 Fastpath for empty remote docs
  3. Commit 07dd Expression stack frame detection runtime
  4. Commit 08ae Matching above 32 fields at percolate indexes
  5. Commit 16b9 Replication listen ports range
  6. Commit 5fa6 Show create table on pq
  7. Commit 54d1 HTTPS port behavior
  8. Commit fdbb Mixing docstore rows when replacing
  9. Commit afb5 Switch TFO unavailable message level to 'info'
  10. Commit 59d9 Crash on strcmp invalid use
  11. Commit 04af Adding index to cluster with system (stopwords) files
  12. Commit 5014 Merge indexes with large dictionaries; RT optimize of large disk chunks
  13. Commit a2ad Indextool can dump meta from current version
  14. Commit 69f6 Issue in group order in GROUP N
  15. Commit 24d5 Explicit flush for SphinxSE after handshake
  16. Commit 31c4 Avoid copy of huge descriptions when not necessary
  17. Commit 2959 Negative time in show threads
  18. Commit f0b3 Token filter plugin vs zero position deltas
  19. Commit a49e Change 'FAIL' to 'WARNING' on multiple hits

Version 3.5.0, 22 Jul 2020

Major new features:

In plain mode it's called sql_field_string. Now it's available in RT mode for real-time indexes too. You can use it as shown in the example:

```sql
create table t(f string attribute indexed);
insert into t values(0,'abc','abc');
select * from t where match('abc');
+---------------------+------+
| id | f |
+---------------------+------+
| 2810845392541843463 | abc |
+---------------------+------+
1 row in set (0.01 sec)

mysql> select * from t where f='abc';
+---------------------+------+
| id | f |
+---------------------+------+
| 2810845392541843463 | abc |
+---------------------+------+
1 row in set (0.00 sec)
```

Minor changes

Breaking changes:

Deprecations:

3.4.2:
sql mysql> describe t; +-------+--------+----------------+ | Field | Type | Properties | +-------+--------+----------------+ | id | bigint | | | f | field | indexed stored | +-------+--------+----------------+

3.5.0:
sql mysql> describe t; +-------+--------+----------------+ | Field | Type | Properties | +-------+--------+----------------+ | id | bigint | | | f | text | indexed stored | +-------+--------+----------------+

Packages

Bugfixes:

  1. Issue #351 searchd memory leak
  2. Commit ceab Tiny read out of bounds in snippets
  3. Commit 1c3e Dangerous write into local variable for crash queries
  4. Commit 26e0 Tiny memory leak of sorter in test 226
  5. Commit d2c7 Huge memory leak in test 226
  6. Commit 0dd8 Cluster shows the nodes are in sync, but count(*) shows different numbers
  7. Commit f1c1 Cosmetic: Duplicate and sometimes lost warning messages in the log
  8. Commit f1c1 Cosmetic: (null) index name in log
  9. Commit 359d Cannot retrieve more than 70M results
  10. Commit 19f3 Can't insert PQ rules with no-columns syntax
  11. Commit bf68 Misleading error message when inserting a document to an index in a cluster
  12. Commit 2cf1 /json/replace and json/update return id in exponent form
  13. Issue #324 Update json scalar properties and mva in the same query
  14. Commit d384 hitless_words doesn't work in RT mode
  15. Commit 5813 ALTER RECONFIGURE in rt mode should be disallowed
  16. Commit 5813 rt_mem_limit gets reset to 128M after searchd restart
  17. highlight() sometimes hangs
  18. Commit 7cd8 Failed to use U+code in RT mode
  19. Commit 2b21 Failed to use wildcard at wordforms at RT mode
  20. Commit e9d0 Fixed SHOW CREATE TABLE vs multiple wordform files
  21. Commit fc90 JSON query without "query" crashes searchd
  22. Manticore official docker couldn't index from mysql 8
  23. Commit 23e0 HTTP /json/insert requires id
  24. Commit bd67 SHOW CREATE TABLE doesn't work for PQ
  25. Commit bd67 CREATE TABLE LIKE doesn't work properly for PQ
  26. Commit 5eac End of line in settings in show index status
  27. Commit cb15 Empty title in "highlight" in HTTP JSON response
  28. Issue #318 CREATE TABLE LIKE infix error
  29. Commit 9040 RT crashes under load
  30. cd512c7d Lost crash log on crash at RT disk chunk
  31. Issue #323 Import table fails and closes the connection
  32. Commit 6275 ALTER reconfigure corrupts a PQ index
  33. Commit 9c1d Searchd reload issues after change index type
  34. Commit 71e2 Daemon crashes on import table with missed files
  35. Issue #322 Crash on select using multiple indexes, group by and ranker = none
  36. Commit c3f5 HIGHLIGHT() doesn't higlight in string attributes
  37. Issue #320 FACET fails to sort on string attribute
  38. Commit 4f1a Error in case of missing data dir
  39. Commit 04f4 access_* are not supported in RT mode
  40. Commit 1c06 Bad JSON objects in strings: 1. CALL PQ returns "Bad JSON objects in strings: 1" when the json is greater than some value.
  41. Commit 32f9 RT-mode inconsistency. In some cases I can't drop the index since it's unknown and can't create it since the directory is not empty.
  42. Issue #319 Crash on select
  43. Commit 22a2 max_xmlpipe2_field = 2M returned warning on 2M field
  44. Issue #342 Query conditions execution bug
  45. Commit dd8d Simple 2 terms search finds a document containing only one term
  46. Commit 9091 It was impossible in PQ to match a json with capital letters in keys
  47. Commit 56da Indexer crashes on csv+docstore
  48. Issue #363 using [null] in json attr in centos 7 causes corrupted inserted data
  49. Major Issue #345 Records not being inserted, count() is random, "replace into" returns OK
  50. max_query_time slows down SELECTs too much
  51. Issue #352 Master-agent communication fails on Mac OS
  52. Issue #328 Error when connecting to Manticore with Connector.Net/Mysql 8.0.19
  53. Commit daa7 Fixed escaping of \0 and optimized performance
  54. Commit 9bc5 Fixed count distinct vs json
  55. Commit 4f89 Fixed drop table at other node failed
  56. Commit 952a Fix crashes on tightly running call pq

Version 3.4.2, 10 April 2020

Critical bugfixes

Version 3.4.0, 26 March 2020

Major changes

Minor changes

Features

Improvements

Bugfixes

Version 3.3.0, 4 February 2020

Features

Improvements

Bugfixes

Version 3.2.2, 19 December 2019

Features

Improvements and changes

Bugfixes

Version 3.2.0, 17 October 2019

Features

Improvements and changes

Bugfixes

Version 3.1.2, 22 August 2019

Features and Improvements

Bugfixes

Version 3.1.0, 16 July 2019

Features and Improvements

Removals

Bugfixes

Version 3.0.2, 31 May 2019

Improvements

Removals

Deprecations

Bugfixes

Version 3.0.0, 6 May 2019

Features and improvements

Behaviour changes

Removed directives

Version 2.8.2 GA, 2 April 2019

Features and improvements

Compiling notes

Cmake minimum version is now 3.13. Compiling requires boost and libssl
development libraries.

Bugfixes

Version 2.8.1 GA, 6 March 2019

Features and improvements

Bugfixes

Version 2.8.0 GA, 28 January 2019

Improvements

Bugfixes

Version 2.7.5 GA, 4 December 2018

Improvements

Bugfixes

Version 2.7.4 GA, 1 November 2018

Improvements

Bugfixes

Version 2.7.3 GA, 26 September 2018

Improvements

Bugfixes

Version 2.7.2 GA, 27 August 2018

Improvements

Bugfixes

Version 2.7.1 GA, 4 July 2018

Improvements

Bugfixes

Version 2.7.0 GA, 11 June 2018

Improvements

Bugfixes

Version 2.6.4 GA, 3 May 2018

Features and improvements

Bugfixes

Version 2.6.3 GA, 28 March 2018

Improvements

Bugfixes

Version 2.6.2 GA, 23 February 2018

Improvements

Bugfixes

Version 2.6.1 GA, 26 January 2018

Improvements

Bugfixes

Version 2.6.0, 29 December 2017

Features and improvements

Bugfixes

Upgrade

In this release we've changed internal protocol used by masters and agents to speak with each other. In case you run Manticoresearch in a distributed environment with multiple instances make sure your first upgrade agents, then the masters.

Version 2.5.1, 23 November 2017

Features and improvements

Bugfixes

Version 2.4.1 GA, 16 October 2017

Features and improvements

Compiling

Manticore Search is built using cmake and the minimum gcc version required for compiling is 4.7.2.

Folders and service

Bugfixes

Version 2.3.3, 06 July 2017