manticoresoftware / columnar

Manticore Columnar Library

Manticore Columnar Library

Manticore Columnar Library is a column-oriented storage and secondary indexing library, aiming to provide decent performance with low memory footprint at big data volume. When used in combination with Manticore Search can be beneficial for those looking for:

1. log analytics including rich free text search capabities (which is missing in e.g. Clickhouse - great tool for metrics analytics)
2. faster / low resource consumption log/metrics analytics. Since the library and Manticore Search are both written in C++ with low level optimizations in mind, in many cases the performance / RAM consumption is better than in Lucene / SOLR / Elasticsearch
3. running log / metric analytics in docker / kubernetes. Manticore Search + the library can work with as little as 30 megabytes of RAM which Elasticsearch / Clickhouse can't. It also starts in less than a second or a few seconds in the worst case. Since the overhead is so little you can afford having more nodes of Manticore Search + the library than Elasticsearch. More nodes and quicker start means higher high availability and agility.
4. powerful SQL for logs/metrics analytics and everything else Manticore Search can give you

Getting started

Requirements

1. CPU: with SSE >= 4.2
2. Architecture: arm64 or x86_64
3. OS: Debian-based OS (Debian, Ubuntu, Mint), RHEL-based (RHEL, Centos, Alma, Oracle Linux, Amazon Linux) OS, Windows, MacOS

Installation from yum/apt repositories

Ubuntu, Debian:

wget https://repo.manticoresearch.com/manticore-repo.noarch.deb
sudo dpkg -i manticore-repo.noarch.deb
sudo apt update
sudo apt install manticore-columnar-lib

Centos:

sudo yum install https://repo.manticoresearch.com/manticore-repo.noarch.rpm
sudo yum install manticore-columnar-lib

searchd -v should include columnar x.y.z, e.g.:

root@srv# searchd -v
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)

Basic usage:

1. Read https://manual.manticoresearch.com/Creating_a_table/Data_types#Row-wise-and-columnar-attribute-storages
2. Create plain or real-time table specifying that the columnar storage should be used

Benchmarks

Log analytics - 6x faster than Elasticsearch

https://db-benchmarks.com/test-logs10m/#elasticsearch-tuned-vs-manticore-search-columnar-storage

Log analytics - 1.4x faster than Clickhouse

https://db-benchmarks.com/test-logs10m/#clickhouse-vs-manticore-search-columnar-storage

Medium data - 110M Hackernews comments - 5x faster than Elasticsearch

https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-elasticsearch

Medium data - 110M Hackernews comments - 11x faster than Clickhouse

https://db-benchmarks.com/test-hn/#manticore-search-columnar-storage-vs-clickhouse

Big data - 1.7B NYC taxi rides - 4x faster than Elasticsearch

https://db-benchmarks.com/test-taxi/#manticore-search-vs-elasticsearch

Big data - 1.7B NYC taxi rides - 1.8x faster than Clickhouse

https://db-benchmarks.com/test-taxi/#manticore-search-vs-clickhouse

columnar's Issues

indextool check produce error

indextool --check test produce error

FAILED, checking attribute '$internal_id_HASH'...  
Unsupported header type: 4  

during check of columnar index files for index created as

create table test (publish_date timestamp, internal_id string attribute, tags_id json, tags_name json, entities_id json, article_body_hash text stored, content text indexed) rt_mem_limit='134217728' engine='columnar'"  

seems string attribute that produces HASH check is missed and got reported as error.

Why the create table sql is different from the show create table sql

mysql> create table idx(title text, type int, price float engine='columnar');
Query OK, 0 rows affected (0.01 sec)

mysql> show create table idx;
+-------+------------------------------------------------------------------------------+
| Table | Create Table                                                                 |
+-------+------------------------------------------------------------------------------+
| idx   | CREATE TABLE idx (
type integer,
price float engine='columnar',
title text
) |
+-------+------------------------------------------------------------------------------+
1 row in set (0.00 sec)

I am very puzzled as to why the above two are different. Is there any adjustment in the manticore?

String comparison is not working

In:

mysql> show version;
--------------
show version
--------------

+-----------+--------------------------------+
| Component | Version                        |
+-----------+--------------------------------+
| Daemon    | 6.2.13 dbd94ac5c@24032423 dev  |
| Columnar  | columnar 2.2.5 aa3504b@240304  |
| Secondary | secondary 2.2.5 aa3504b@240304 |
| KNN       | knn 2.2.5 aa3504b@240304       |
| Buddy     | buddy v2.3.1                   |
+-----------+--------------------------------+
5 rows in set (0.01 sec)

select * from t where s < 'b' shows b instead of a:

mysql> drop table if exists t; create table t(s string) engine='columnar'; insert into t values (1,'a'),(2,'b'); select * from t where s < 'b';
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
create table t(s string) engine='columnar'
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
insert into t values (1,'a'),(2,'b')
--------------

Query OK, 2 rows affected (0.00 sec)

--------------
select * from t where s < 'b'
--------------

+------+------+
| id   | s    |
+------+------+
|    2 | b    |
+------+------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

Expected: same as with the row-wise storage:

mysql> drop table if exists t; create table t(s string); insert into t values (1,'a'),(2,'b'); select * from t where s < 'b';
--------------
drop table if exists t
--------------

Query OK, 0 rows affected (0.00 sec)

--------------
create table t(s string)
--------------

Query OK, 0 rows affected (0.01 sec)

--------------
insert into t values (1,'a'),(2,'b')
--------------

Query OK, 2 rows affected (0.00 sec)

--------------
select * from t where s < 'b'
--------------

+------+------+
| id   | s    |
+------+------+
|    1 | a    |
+------+------+
1 row in set (0.00 sec)
--- 1 out of 1 results in 0ms ---

distributed index can not handle knn queries

seems all knn related data is not send to agent that is why agents of the distributed index replies with the error

select id, knn_dist() from distr where knn ( image_vector, 5, (0.286569,-0.031816,0.066684,0.032926) );
ERROR 1064: table distr: agent ip: remote query error: table test: KNN_DIST() is only allowed for KNN() queries

Bigint leads to index corruption

I'm doing an import of a fairly large dataset into Manticore with columnar. The table has several number fields and when these are set as type integer the import finishes without a hitch. However, if i change these to bigint the import crashes after a few minutes. Not instantly, but consistently after about 1M inserts and without any SQL errors but the log says:

malloc(): unsorted double linked list corrupted
Crash!!! Handling signal 6
malloc(): invalid size (unsorted)
Crash!!! Handling signal 11

The bigint's are negative and would overflow 32bit, so using integer to start with was a mistake from my side. I can then not continue the import and the table is corrupted. When I drop the table and recreate it I get the message:

Create schema index err: Error 1064 (42000): error adding table 'TABLENAME': directory is not empty: /var/lib/manticore/TABLENAME

Meaning I have to clear the whole manticore directory and start over.

Add possibility to set EF on every search for vector field

We need to expose the ef parameter in search queries to make it possible to set it at runtime while querying the database. The original HNSW library supports it (https://github.com/nmslib/hnswlib/blob/master/ALGO_PARAMS.md) and Qdrant also has this parameter exposed as 'hnsw_ef'.

Please index from mysql source and use sql_query_range

Please index from mysql source and use sql_query_range
thanks.

crash on select when too many ft fields

If you run this script:

#!/usr/bin/php  
<?php  
if (count($argv) < 5) die("Usage: ".__FILE__." <batch size> <concurrency> <docs> <number of fields>\n");  
  
// This function waits for an idle mysql connection for the $query, runs it and exits  
function process($query) {  
    global $all_links;  
    global $requests;  
    foreach ($all_links as $k=>$link) {  
        if (@$requests[$k]) continue;  
        mysqli_query($link, $query, MYSQLI_ASYNC);  
        @$requests[$k] = microtime(true);  
        return true;  
    }  
    do {  
        $links = $errors = $reject = array();  
        foreach ($all_links as $link) {  
            $links[] = $errors[] = $reject[] = $link;  
        }  
        $count = @mysqli_poll($links, $errors, $reject, 0, 1000);  
        if ($count > 0) {  
            foreach ($links as $j=>$link) {  
                $res = @mysqli_reap_async_query($links[$j]);  
                foreach ($all_links as $i=>$link_orig) if ($all_links[$i] === $links[$j]) break;  
                if ($link->error) {  
                    echo "ERROR: {$link->error}\n";  
                    if (!mysqli_ping($link)) {  
                        echo "ERROR: mysql connection is down, removing it from the pool\n";  
                        unset($all_links[$i]); // remove the original link from the pool  
                        unset($requests[$i]); // and from the $requests too  
                    }  
                    return false;  
                }  
                if ($res === false and !$link->error) continue;  
                if (is_object($res)) {  
                    mysqli_free_result($res);  
                }  
                $requests[$i] = microtime(true);  
		mysqli_query($link, $query, MYSQLI_ASYNC); // making next query  
                return true;  
            }  
        };  
    } while (true);  
    return true;  
}  
  
$all_links = [];  
$requests = [];  
$c = 0;  
for ($i=0;$i<$argv[2];$i  ) {  
  $m = @mysqli_connect('127.0.0.1', '', '', '', 9306);  
      if (mysqli_connect_error()) die("Cannot connect to Manticore\n");  
      $all_links[] = $m;  
  }  
  
// init  
mysqli_query($all_links[0], "drop table if exists t2");  
$create = "create table t2(";  
$ar=[]; for ($n=1;$n<=$argv[4];$n  ) $ar[] = "f$n text"; $create .= implode(',', $ar);  
$create .= ", s string) engine='columnar'";  
mysqli_query($all_links[0], $create);  
  
$batch = [];  
$query_start = "insert into t2(";  
$ar=[]; for ($n=1;$n<=$argv[4];$n  ) $ar[] = "f$n"; $query_start .= implode(',', $ar);  
$query_start .= ",s) values ";  
  
srand(1);  
echo "preparing...\n";  
$error = false;  
$cache_file_name = '/tmp/dmitry'.md5($query_start).'_'.$argv[1].'_'.$argv[3];  
$c = 0;  
if (!file_exists($cache_file_name)) {  
    $batches = [];  
    while ($c < $argv[3]) {  
      $ar = [];  
      for ($n=0;$n<$argv[4];$n  ) $ar[] = "'".substr(md5(rand()), 0, 4)."'";  
      $ar[] = "'".uniqid()."'";  
      $batch[] = "(".implode(',', $ar).")";  
      $c  ;  
      if (floor($c/1000) == $c/1000) echo "\r".($c/$argv[3]*100)."%       ";  
        if (count($batch) == $argv[1]) {  
          $batches[] = $query_start.implode(',', $batch);  
          $batch = [];  
        }  
    }  
    if ($batch) $batches[] = $query_start.implode(',', $batch);  
    file_put_contents($cache_file_name, serialize($batches));  
} else {  
    echo "found in cache\n";  
    $batches = unserialize(file_get_contents($cache_file_name));  
}  
  
echo "querying...\n";  
  
$t = microtime(true);  
  
foreach ($batches as $batch) {  
  if (!process($batch)) die("ERROR\n");  
}  
  
// wait until all the workers finish  
do {  
  $links = $errors = $reject = array();  
  foreach ($all_links as $link)  $links[] = $errors[] = $reject[] = $link;  
  $count = @mysqli_poll($links, $errors, $reject, 0, 100);  
} while (count($all_links) != count($links)   count($errors)   count($reject));  
  
echo "finished inserting\n";  
echo "Total time: ".(microtime(true) - $t)."\n";  
echo round($argv[3] / (microtime(true) - $t))." docs per sec\n";  

like this:

rm /tmp/dmitry*  
php load_sql_columnar_crash.php 1000 4 50000 47  

against version Server version: 5.0.3 aea4e54ff@220727 dev (columnar 1.16.1 637ddc6@220727) git branch HEAD (no branch)

then the following will cause a crash:

snikolaev@dev:~$ mysql -P9306 -h0 -e "select * from t2 where s='abc'"  
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query  

The config is :

snikolaev@dev:~$ cat /etc/manticoresearch/manticore.conf  
common {  
    plugin_dir = /usr/local/lib/manticore  
    lemmatizer_base = /var/lib/manticore/  
}  
  
searchd {  
    listen = 127.0.0.1:9312  
    listen = 127.0.0.1:9306:mysql  
    listen = 127.0.0.1:9308:http  
    log = /var/log/manticore/searchd.log  
    query_log = /var/log/manticore/query.log  
    pid_file = /var/run/manticore/searchd.pid  
    data_dir = /var/lib/manticore  
    query_log_format = sphinxql  
    binlog_max_log_size = 1M  
    max_open_files = max  
    max_packet_size = 128M  
}  

Notes:

• it crashes in the latest release as well
• if I remove engine='columnar' it doesn't crash
• if I lower the number of ft fields to 40 (php load_sql_columnar_crash.php 1000 4 50000 40) it doesn't crash
• when it crashes you have to remove the index dir manually, otherwise you can't create the table again:

  snikolaev@dev:~$ mysql -P9306 -h0 -e "select * from t2 where s='abc'"  
  ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query  
  snikolaev@dev:~$ sudo cat /var/lib/manticore/manticore.json  
  {  
  	"clusters":	{  
  	},  
  	"indexes":	{  
  		"chinese_tokenization":	{  
  			"type":	"rt",  
  			"path":	"chinese_tokenization"  
  		},  
  		"t1":	{  
  			"type":	"rt",  
  			"path":	"t1"  
  		},  
  		"t2":	{  
  			"type":	"rt",  
  			"path":	"t2"  
  		}  
  	}  
  }snikolaev@dev:~$mysql -P9306 -h0 -e "show tables"  
   ---------------------- ------   
  | Index                | Type |  
   ---------------------- ------   
  | chinese_tokenization | rt   |  
  | t1                   | rt   |  
   ---------------------- ------   
  

• archive of the index is t2.tgz
• the crash looks really bad (no crashlog):

   [Wed Jul 27 18:14:16.110 2022] [334175] prereading 2 indexes  
   [Wed Jul 27 18:14:16.110 2022] [334171] accepting connections  
   [Wed Jul 27 18:14:16.120 2022] [334175] preread 2 indexes in 0.010 sec  
   [Wed Jul 27 18:19:12.955 2022] [334174] rt: index t2: diskchunk 0(1), segments 29  saved in 0.443840 (0.535320) sec, RAM saved/new 48616545/0 ratio 0.950000 (soft limit 127506841, conf limit 134217728)  
   [Wed Jul 27 18:21:04.950 2022] [327687] watchdog: main process 334171 killed dirtily with signal 11, core dumped, will be restarted  
   [Wed Jul 27 18:21:04.950 2022] [327687] watchdog: main process 337621 forked ok  
   [Wed Jul 27 18:21:04.951 2022] [337621] starting daemon version '5.0.3 aea4e54ff@220727 dev (columnar 1.16.1 637ddc6@220727)' ...  
   [Wed Jul 27 18:21:04.951 2022] [337621] listening on 127.0.0.1:9312 for sphinx and http(s)  
   [Wed Jul 27 18:21:04.951 2022] [337621] listening on 127.0.0.1:9306 for mysql  
  

Can publish to homebrew?

Can't experience using columnar on macOS

Crash!!! Handling signal 11 on import to table with multi64 field

Doing an import to a table which uses columnar Manticore crashes with message Crash!!! Handling signal 11. (SIGSEGV) No further details in query log and it is not related to a specific document. The import always crashes after a few seconds however. I have imported much larger tables with success.

The issue seems to be related to a multi64 field (which i did not use with the successful tables). When i set engine='rowwise' on this field, the issue goes away and i can import the full table.

Global config options on the table are: engine='columnar' dict='keywords' expand_keywords='1' index_exact_words='1' min_infix_len='3'

Im on the latest Manticore release 6.0.4 and it's in docker.

Is it possible to limit the memory usage of the searchd process

I find that after batch insert data to manticore, the usage of the searchd process not free the memory?

This is my config file

index hacker_new {
    type = rt
    rt_mem_limit = 2048M
    path = /home/mcsearch/index/hacker_new
    
    rt_attr_bigint = story_id
    rt_attr_timestamp = story_time
    rt_field = story_url
    rt_field = story_text
    rt_attr_string = story_author
    rt_attr_uint = comment_id
    rt_field = comment_text
    rt_attr_string = comment_author
    rt_attr_uint = comment_ranking
    rt_attr_uint = author_comment_count
    rt_attr_uint = story_comment_count

    stored_fields = story_url, story_text, comment_text
}

searchd
{
	listen = 9306:mysql41
	listen = 127.0.0.1:9308:http
	query_log = /root/query.log
	log = /root/searchd.log
	pid_file = /root/searchd.pid
#	data_dir = /home/mcsearch/data
	binlog_path = 

	net_workers = 12
	rt_flush_period = 30
   	 # jobs_queue_size = 	
	max_open_files = 10000
    	max_packet_size = 128M
	subtree_docs_cache = 512M
	subtree_hits_cache = 512M

	qcache_max_bytes = 0
}

My test batch insert code is:

import time
import os

times = []
numbers = 100
batch = 30000
for i in range(numbers):
    start = time.time()
    os.system(f'mysql -P9306 -h0 < sql_batch_insert_{batch}.sql')
    spend = time.time() - start
    print(f'#{str(i+1)} spend', spend)
    times.append(spend)
print(times)
print(f'#{numbers}，#{batch}，avg time：', sum(times)/len(times))

After the batch program finish. The usage of memory is still very high.

Memory limit

Hi. After reading your benchmark test, the results look pretty good.
I am curious about how you limit the memory usage of `columnar`.
In addition, your data set looks too small, it is not clear whether the author has tested it on a larger data set, such as a data volume of 1 billion or more.

error on attr string generated by mysql uuid

In mysql , the column value is

7cbe3f8-a185-11ea-82b4-00163e0eafb3`  
  
but when it index to the sphinx, use columnar, it becomes to  

e7cbe3f8-a185-11ea-82b4-00163e0eafb```

Failed to build for arm

Failed to build for arm64 debian 10.
Apparently the FastPFOR library only supports the x64 platform.
Please add build from source instructions to the documentation.

You might be interested in https://github.com/powturbo/TurboPFor-Integer-Compression

OS/Compiler (64 bits):
Windows: MinGW-w64 makefile
Windows: Visual c++ (>=VS2008) - makefile.vs (for nmake)
Windows: Visual Studio project file - vs/vs2017 - Thanks to PavelP
Linux amd64: GNU GCC (>=4.6)
Linux amd64: Clang (>=3.2)
Linux arm64: 64 bits aarch64 ARMv8: gcc (>=6.3)
Linux arm64: 64 bits aarch64 ARMv8: clang
MaxOS: XCode (>=9)
PowerPC ppc64le (incl. SIMD): gcc (>=8.0)

Thanks.

mac MCL package doesn't include the knn lib

➜  ~ searchd -v
Manticore 6.2.13 5689bfca3@231117 dev (columnar 2.2.5 975172c@231117) (secondary 2.2.5 975172c@231117)

Is this expected?

failed tests at c6dbbc

c6dbbcb didn't pass testing - https://github.com/manticoresoftware/columnar/runs/16049753723

Add lz4 compression to strings

Currently strings are stored basically uncompressed (only lengths are compressed) when table compression is not applicable. It makes sense to try to add LZ4 compression to such strings to save disk space (string hases used for grouping and filters are stored as a separate attribute).

CI failed at da216d8a

CI failed at commit da216d8

The task is to figure out why and fix the issue.

Is it possible to achieve additional effects in plain mode?

SI affects result set

SI enabled:

mysql> select id from product_main where corp_id = 230028829261824 and id = 627417850052609;
Empty set (0.00 sec)

SI disabled:

mysql> set global secondary_indexes=0;
Query OK, 0 rows affected (0.00 sec)

mysql> select id from product_main where corp_id = 230028829261824 and id = 627417850052609;
+-----------------+
| id              |
+-----------------+
| 627417850052609 |
+-----------------+
1 row in set (0.00 sec)

"ALTER REBUILD SECONDARY" doesn't help.

The version is:

mysql> show status like 'version';
+---------+---------------------------------------------------------------------------------------------+
| Counter | Value                                                                                       |
+---------+---------------------------------------------------------------------------------------------+
| version | 6.2.1 b924aaa5d@230815 dev (columnar 2.0.5 c0d2628@230620) (secondary 2.0.5 c0d2628@230620) |
+---------+---------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Can be reproduced in /home/snikolaev/si_27 on dev2.

to_string(a) requires a to be in the select list

to_string(<bool>) works only if the <bool> is in the select list too, otherwise it returns an empty value:

mysql> drop table if exists t; create table t(a bool) engine='columnar'; insert into t(id, a) values(1, 1); select to_string(a) from t; select a, to_string(a) from t;  
--------------  
drop table if exists t  
--------------  
  
Query OK, 0 rows affected (0.01 sec)  
  
--------------  
create table t(a bool) engine='columnar'  
--------------  
  
Query OK, 0 rows affected (0.00 sec)  
  
--------------  
insert into t(id, a) values(1, 1)  
--------------  
  
Query OK, 1 row affected (0.00 sec)  
  
--------------  
select to_string(a) from t  
--------------  
  
 --------------   
| to_string(a) |  
 --------------   
|              |  
 --------------   
1 row in set (0.00 sec)  
  
--------------  
select a, to_string(a) from t  
--------------  
  
 ------ --------------   
| a    | to_string(a) |  
 ------ --------------   
|    1 | 1            |  
 ------ --------------   
1 row in set (0.00 sec)  

benchmark's conf

Hi. I'm interested in your project.

Can you pull your config file to github? thanks

partial results with SI on

If you run this https://github.com/sanikolaev/Eclipsium_manticore_crash/tree/ac07a3560416f81bf32eaa3d0bcdcff5c1b1490b (git clone; docker compose up) you'll see that Manticore finds significantly less results with SI on compared to when SI is off:

mysql> set global secondary_indexes=1;
Query OK, 0 rows affected (0.06 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|     4000 |
+----------+
1 row in set (0.16 sec)

mysql> set global secondary_indexes=0;
Query OK, 0 rows affected (0.00 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|  2614447 |
+----------+
1 row in set (0.01 sec)

mysql> set global secondary_indexes=1;
Query OK, 0 rows affected (0.00 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|    12000 |
+----------+
1 row in set (0.01 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|    14000 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|    16000 |
+----------+
1 row in set (0.00 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|    16000 |
+----------+
1 row in set (0.01 sec)

mysql> set global secondary_indexes=0;
Query OK, 0 rows affected (0.02 sec)

mysql> select count(*) from posts_idx where uploaded_at >= 1696494232;
+----------+
| count(*) |
+----------+
|  2624437 |
+----------+
1 row in set (0.45 sec)

The version is:

snikolaev@dev2:~/issue_1458$ mysql -P49306 -h0
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2084
Server version: 6.2.13 f94555a29@230908 dev (columnar 2.2.5 709b9ac@230908) (secondary 2.2.5 709b9ac@230908) git branch master...origin/master

Same w/o count(*):

mysql> set global secondary_indexes=1;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from posts_idx where uploaded_at >= 1696494232 limit 0 option cutoff=0; show meta;
Empty set (0.00 sec)

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 0     |
| total_found    | 0     |
| total_relation | eq    |
| time           | 0.001 |
+----------------+-------+
4 rows in set (0.00 sec)

mysql> select * from posts_idx where uploaded_at >= 1696494232 limit 0 option cutoff=0; show meta;
Empty set (0.00 sec)

+----------------+-------+
| Variable_name  | Value |
+----------------+-------+
| total          | 1     |
| total_found    | 4000  |
| total_relation | eq    |
| time           | 0.003 |
+----------------+-------+
4 rows in set (0.00 sec)

mysql> set global secondary_indexes=0;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from posts_idx where uploaded_at >= 1696494232 limit 0 option cutoff=0; show meta;
Empty set (0.06 sec)

+----------------+---------+
| Variable_name  | Value   |
+----------------+---------+
| total          | 1       |
| total_found    | 2690363 |
| total_relation | eq      |
| time           | 0.065   |
+----------------+---------+
4 rows in set (0.01 sec)

columnar: integrate streamvbyte library

streamvbyte is a codec that uses AVX2, it is included in the fastpfor library. However, most recent version is developed as a separate library, https://github.com/lemire/streamvbyte. We need to integrate that library with MCL as this will allow us to use a most recent version of the codec and use a version of it that has built-in differential coding. Using that built-in version should be faster than using external code to calc deltas and inverse deltas. The downside is that it requires implementing a specialized codec in the MCL.

Update deps in the daemon after CI

The task is to make an automated PR in the daemon repo to update deps.txt if the columnar commit passes tests. Exactly like it's done in the Buddy repo.

RANDOM CRASH

Faced such a problem that when I have an open client for a long time, the manticore crashes on a search query and does not respond to sql queries, sometimes the processor is loaded into the ceiling. I was looking for what the problem was for a very long time, but now I was able to catch the following crash log in the logs on my kube.

[Wed May 17 12:12:34.723 2023] [40] [BUDDY] restart amount of attempts (3) has been exceeded    
[Wed May 17 12:13:14.526 2023] [38] preread 2 tables in 40.321 sec    
------- FATAL: CRASH DUMP -------    
[Wed May 17 18:35:25.437 2023] [    1]    
    
--- crashed SphinxQL request dump ---    
INSERT I%comments�4(ache` (`�4(0�4( $ G4(    
                                         a�G4(pEJ4(     
$�a4(�H@$pEJ4(�4�$P�    
--- request dump end ---    
--- local index:���@(p�>(    
Manticore 6.0.4 1a3a4ea82@230314 (columnar 2.0.4 5a49bd7@230306) (secondary 2.0.4 5a49bd7@230306)    
Handling signal 6    
-------------- backtrace begins here ---------------    
Program compiled with Clang 15.0.4    
Configured with flags: Configured with these definitions: -DDISTR_BUILD=focal -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore    
Built on Linux aarch64 for Linux x86_64 (focal)    
Stack bottom = 0x7f2836287620, thread stack size = 0x20000    
Trying manual backtrace:    
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)    
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f2836290000, stacksize=0x20000)    
Trying system backtrace:    
begin of system symbols:    
searchd(_Z12sphBacktraceib 0x22a)[0x55b6f83b316a]    
searchd(_ZN11CrashLogger11HandleCrashEi 0x355)[0x55b6f8276715]    
/lib/x86_64-linux-gnu/libpthread.so.0( 0x14420)[0x7f2840bd2420]    
/lib/x86_64-linux-gnu/libc.so.6(gsignal 0xcb)[0x7f2840a0f00b]    
/lib/x86_64-linux-gnu/libc.so.6(abort 0x12b)[0x7f28409ee859]    
/lib/x86_64-linux-gnu/libc.so.6( 0x8d26e)[0x7f2840a5926e]    
/lib/x86_64-linux-gnu/libc.so.6( 0x952fc)[0x7f2840a612fc]    
/lib/x86_64-linux-gnu/libc.so.6( 0x96fac)[0x7f2840a62fac]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x20cb97)[0x7f284062cb97]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x20cc89)[0x7f284062cc89]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x5912ec)[0x7f28409b12ec]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x591215)[0x7f28409b1215]    
searchd(_ZNK9RtIndex_c15WriteAttributesER21SaveDiskDataContext_tR10CSphString 0xcc3)[0x55b6f865d9d3]    
searchd(_ZNK9RtIndex_c12SaveDiskDataEPKcRK11VecTraits_TI17CSphRefcountedPtrIK11RtSegment_tEERK12ChunkStats_tR10CSphString 0x16e)[0x55b6f865f48e]    
searchd(_ZN9RtIndex_c13SaveDiskChunkEbbb 0x5c7)[0x55b6f865b557]    
searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEE11VecTraits_TIhEENUlN5boost7context6detail10transfer_tEE_8__invokeES9_ 0x1c)[0x55b6f8b7f19c]    
searchd(make_fcontext 0x37)[0x55b6f8b9dec7]    
Trying boost backtrace:    

I saw something similar in the main manticore repository:
manticoresoftware/manticoresearch#1127

Precision issue while querying the vector on the Qdrant vector benchmark test

While trying to integrate Manticore Search into the Qdrant vector database benchmark, I discovered that we have very low mean precision.

In the current test that I have attached here, Manticore Search's mean precision equals 0.16, while Qdrant's is 0.99.

We should investigate this behavior and try to understand why it happens.

I have also attached the original test set in the attachments to initialize the database.

I checked also it with https://newtum.com/calculators/maths/cosine-similarity-calculator

So Manticore top 1 result gives 0.89 similarity while the expected one is 0.91.

Path to the dump: /tmp/bench.tar.gz on dev2.

The original Request vector used for search [k=100, ef=128]

0.011699999682605267, 0.011500000022351742, 0.008700000122189522, 0.009999999776482582, 0.07850000262260437, 0.10000000149011612, 0.07840000092983246, 0.05299999937415123, 0.052400000393390656, 0.08190000057220459, 0.0658000037074089, 0.057999998331069946, 0.01590000092983246, 0.017000000923871994, 0.04610000178217888, 0.02419999986886978, 0.00839999970048666, 0.006399999838322401, 0.007199999876320362, 0.010200000368058681, 0.030400000512599945, 0.06790000200271606, 0.05889999866485596, 0.057100001722574234, 0.0333000011742115, 0.07859999686479568, 0.08919999748468399, 0.04230000078678131, 0.013799999840557575, 0.013299999758601189, 0.028999999165534973, 0.021900000050663948, 0.008999999612569809, 0.012199999764561653, 0.010700000450015068, 0.01080000028014183, 0.026599999517202377, 0.03849999979138374, 0.057100001722574234, 0.052000001072883606, 0.03550000116229057, 0.04879999905824661, 0.06920000165700912, 0.05310000106692314, 0.014399999752640724, 0.014299999922513962, 0.01489999983459711, 0.02500000037252903, 0.017100000753998756, 0.016100000590085983, 0.010599999688565731, 0.03240000084042549, 0.02710000053048134, 0.04580000042915344, 0.05310000106692314, 0.06239999830722809, 0.031599998474121094, 0.06080000102519989, 0.06610000133514404, 0.08129999786615372, 0.01860000006854534, 0.022700000554323196, 0.011599999852478504, 0.03370000049471855, 0.024700000882148743, 0.011800000444054604, 0.010700000450015068, 0.06390000134706497, 0.039500001817941666, 0.040300000458955765, 0.05249999836087227, 0.0957999974489212, 0.05510000139474869, 0.06759999692440033, 0.08579999953508377, 0.17489999532699585, 0.024399999529123306, 0.02810000069439411, 0.008700000122189522, 0.05119999870657921, 0.01489999983459711, 0.00860000029206276, 0.012400000356137753, 0.035599999129772186, 0.03280000016093254, 0.03869999945163727, 0.046300001442432404, 0.048900000751018524, 0.04919999837875366, 0.0640999972820282, 0.07050000131130219, 0.11640000343322754, 0.020600000396370888, 0.016200000420212746, 0.012500000186264515, 0.04019999876618385, 0.013500000350177288, 0.011300000362098217, 0.007400000002235174, 0.011800000444054604, 0.032999999821186066, 0.02979999966919422, 0.0364999994635582, 0.041999999433755875, 0.04410000145435333, 0.051899999380111694, 0.06589999794960022, 0.05270000174641609, 0.013899999670684338, 0.016200000420212746, 0.01510000042617321, 0.02239999920129776, 0.011800000444054604, 0.008799999952316284, 0.007799999788403511, 0.010900000110268593, 0.037300001829862595, 0.07050000131130219, 0.06279999762773514, 0.03909999877214432, 0.02419999986886978, 0.055799998342990875, 0.07209999859333038, 0.04479999840259552, 0.01679999940097332, 0.013399999588727951, 0.03310000151395798, 0.020800000056624413, 0.004100000020116568, 0.008999999612569809, 0.016499999910593033, 0.011699999682605267, 0.09619999676942825, 0.13590000569820404, 0.12150000035762787, 0.08799999952316284, 0.06390000134706497, 0.11739999800920486, 0.09700000286102295, 0.07029999792575836, 0.012600000016391277, 0.021900000050663948, 0.07460000365972519, 0.0421999990940094, 0.003700000001117587, 0.00839999970048666, 0.009999999776482582, 0.013199999928474426, 0.03200000151991844, 0.06129999831318855, 0.07509999722242355, 0.09139999747276306, 0.042899999767541885, 0.0640999972820282, 0.0746999979019165, 0.04659999907016754, 0.008700000122189522, 0.028599999845027924, 0.051500000059604645, 0.027899999171495438, 0.007300000172108412, 0.01489999983459711, 0.02250000089406967, 0.018200000748038292, 0.01810000091791153, 0.01759999990463257, 0.08619999885559082, 0.06400000303983688, 0.027499999850988388, 0.028999999165534973, 0.055799998342990875, 0.060499999672174454, 0.01140000019222498, 0.014499999582767487, 0.02070000022649765, 0.02850000001490116, 0.020800000056624413, 0.019099999219179153, 0.012000000104308128, 0.04500000178813934, 0.023399999365210533, 0.023099999874830246, 0.053300000727176666, 0.08179999887943268, 0.03970000147819519, 0.024299999698996544, 0.03889999911189079, 0.10819999873638153, 0.022099999710917473, 0.030300000682473183, 0.01600000075995922, 0.05339999869465828, 0.02710000053048134, 0.01889999955892563, 0.01640000008046627, 0.08810000121593475, 0.043699998408555984, 0.017000000923871994, 0.028200000524520874, 0.13899999856948853, 0.060100000351667404, 0.03909999877214432, 0.03709999844431877, 0.20309999585151672, 0.03620000183582306, 0.0340999998152256, 0.011900000274181366, 0.07450000196695328, 0.016699999570846558, 0.0052999998442828655, 0.010400000028312206, 0.04560000076889992, 0.022199999541044235, 0.020099999383091927, 0.015599999576807022, 0.04580000042915344, 0.04309999942779541, 0.04019999876618385, 0.04439999908208847, 0.13459999859333038, 0.020600000396370888, 0.019200000911951065, 0.014999999664723873, 0.046799998730421066, 0.010300000198185444, 0.006599999964237213, 0.012000000104308128, 0.017899999395012856, 0.01850000023841858, 0.019500000402331352, 0.04399999976158142, 0.03290000185370445, 0.02979999966919422, 0.0706000030040741, 0.09549999982118607, 0.05810000002384186, 0.009700000286102295, 0.015699999406933784, 0.03189999982714653, 0.0406000018119812, 0.007400000002235174, 0.009800000116229057, 0.010300000198185444, 0.011599999852478504, 0.040699999779462814, 0.08299999684095383, 0.06750000268220901, 0.035599999129772186, 0.03519999980926514, 0.08489999920129776, 0.07400000095367432, 0.057500001043081284, 0.00800000037997961, 0.010499999858438969, 0.050200000405311584, 0.03790000081062317, 0.00989999994635582, 0.021299999207258224, 0.0364999994635582, 0.024700000882148743, 0.04809999838471413, 0.07810000330209732, 0.094200000166893, 0.08399999886751175, 0.04170000180602074, 0.05249999836087227, 0.07289999723434448, 0.06639999896287918, 0.010099999606609344, 0.023099999874830246, 0.0868000015616417, 0.06459999829530716, 0.033799998462200165, 0.02160000056028366, 0.01899999938905239, 0.03700000047683716, 0.027400000020861626, 0.044599998742341995, 0.04349999874830246, 0.10090000182390213, 0.05249999836087227, 0.06769999861717224, 0.057500001043081284, 0.04639999940991402, 0.022700000554323196, 0.04170000180602074, 0.04749999940395355, 0.024000000208616257, 0.06080000102519989, 0.033399999141693115, 0.03680000081658363, 0.0763000026345253, 0.053300000727176666, 0.021900000050663948, 0.06080000102519989, 0.11249999701976776, 0.0674000009894371, 0.021800000220537186, 0.05270000174641609, 0.13120000064373016, 0.05889999866485596, 0.03310000151395798, 0.02160000056028366, 0.06530000269412994, 0.022099999710917473, 0.017400000244379044, 0.030700000002980232, 0.030300000682473183, 0.027300000190734863, 0.029400000348687172, 0.06279999762773514, 0.033799998462200165, 0.04010000079870224, 0.043699998408555984, 0.05900000035762787, 0.07609999924898148, 0.028599999845027924, 0.01889999955892563, 0.04179999977350235, 0.06960000097751617, 0.00800000037997961, 0.007899999618530273, 0.009499999694526196, 0.011099999770522118, 0.08179999887943268, 0.10530000180006027, 0.08699999749660492, 0.05310000106692314, 0.05180000141263008, 0.08860000222921371, 0.07509999722242355, 0.06889999657869339, 0.015799999237060547, 0.018200000748038292, 0.051600001752376556, 0.028599999845027924, 0.006200000178068876, 0.004800000227987766, 0.006800000090152025, 0.009700000286102295, 0.03370000049471855, 0.06939999759197235, 0.07209999859333038, 0.061799999326467514, 0.0364999994635582, 0.08240000158548355, 0.11050000041723251, 0.04879999905824661, 0.013899999670684338, 0.01269999984651804, 0.03139999881386757, 0.020899999886751175, 0.010300000198185444, 0.008500000461935997, 0.009999999776482582, 0.00860000029206276, 0.029200000688433647, 0.03709999844431877, 0.06930000334978104, 0.05209999904036522, 0.031199999153614044, 0.04650000110268593, 0.0835999995470047, 0.04859999939799309, 0.014100000262260437, 0.011900000274181366, 0.014600000344216824, 0.02280000038444996, 0.01600000075995922, 0.012299999594688416, 0.010099999606609344, 0.030300000682473183, 0.027400000020861626, 0.04969999939203262, 0.0706000030040741, 0.0674000009894371, 0.029999999329447746, 0.05730000138282776, 0.05469999834895134, 0.08820000290870667, 0.015699999406933784, 0.020099999383091927, 0.011900000274181366, 0.03530000150203705, 0.02250000089406967, 0.01119999960064888, 0.009700000286102295, 0.05999999865889549, 0.037300001829862595, 0.04410000145435333, 0.05939999967813492, 0.08869999647140503, 0.049400001764297485, 0.06599999964237213, 0.08269999921321869, 0.18400000035762787, 0.023000000044703484, 0.02410000003874302, 0.009800000116229057, 0.05009999871253967, 0.013899999670684338, 0.006399999838322401, 0.01119999960064888, 0.033799998462200165, 0.03150000050663948, 0.03929999843239784, 0.04740000143647194, 0.04089999943971634, 0.04259999841451645, 0.06599999964237213, 0.07320000231266022, 0.125, 0.02239999920129776, 0.01549999974668026, 0.01209999993443489, 0.037300001829862595, 0.01119999960064888, 0.006300000008195639, 0.007499999832361937, 0.01119999960064888, 0.03200000151991844, 0.028300000354647636, 0.04800000041723251, 0.041200000792741776, 0.04190000146627426, 0.05829999968409538, 0.07840000092983246, 0.051600001752376556, 0.013500000350177288, 0.0142000000923872, 0.016899999231100082, 0.019500000402331352, 0.009700000286102295, 0.00570000009611249, 0.005900000222027302, 0.00800000037997961, 0.03819999843835831, 0.07479999959468842, 0.061900001019239426, 0.05050000175833702, 0.02710000053048134, 0.0608999989926815, 0.08449999988079071, 0.05510000139474869, 0.015799999237060547, 0.012000000104308128, 0.03689999878406525, 0.021299999207258224, 0.006200000178068876, 0.014399999752640724, 0.02280000038444996, 0.011300000362098217, 0.10400000214576721, 0.148499995470047, 0.14489999413490295, 0.1062999963760376, 0.06589999794960022, 0.13019999861717224, 0.11670000106096268, 0.09629999846220016, 0.015200000256299973, 0.025599999353289604, 0.08219999819993973, 0.04540000110864639, 0.004699999932199717, 0.00559999980032444, 0.010999999940395355, 0.010099999606609344, 0.03500000014901161, 0.06650000065565109, 0.09539999812841415, 0.11710000038146973, 0.045899998396635056, 0.07039999961853027, 0.09549999982118607, 0.07150000333786011, 0.009700000286102295, 0.028999999165534973, 0.05739999935030937, 0.025100000202655792, 0.008500000461935997, 0.010200000368058681, 0.019999999552965164, 0.014800000004470348, 0.024800000712275505, 0.01850000023841858, 0.09279999881982803, 0.07010000199079514, 0.029999999329447746, 0.033799998462200165, 0.07490000128746033, 0.07490000128746033, 0.01489999983459711, 0.017400000244379044, 0.020999999716877937, 0.03319999948143959, 0.018200000748038292, 0.012199999764561653, 0.008299999870359898, 0.04190000146627426, 0.025200000032782555, 0.024000000208616257, 0.05310000106692314, 0.07620000094175339, 0.048900000751018524, 0.024700000882148743, 0.032999999821186066, 0.1257999986410141, 0.017500000074505806, 0.025599999353289604, 0.013100000098347664, 0.052400000393390656, 0.026100000366568565, 0.01489999983459711, 0.015699999406933784, 0.08609999716281891, 0.04340000078082085, 0.02419999986886978, 0.03620000183582306, 0.1395999938249588, 0.05649999901652336, 0.04270000010728836, 0.04309999942779541, 0.21359999477863312, 0.03449999913573265, 0.025100000202655792, 0.014299999922513962, 0.07569999992847443, 0.014600000344216824, 0.006000000052154064, 0.00930000003427267, 0.041099999099969864, 0.025100000202655792, 0.018699999898672104, 0.03099999949336052, 0.04699999839067459, 0.035100001841783524, 0.05299999937415123, 0.054099999368190765, 0.13439999520778656, 0.016699999570846558, 0.01759999990463257, 0.01489999983459711, 0.04390000179409981, 0.009200000204145908, 0.005499999970197678, 0.01360000018030405, 0.018400000408291817, 0.01860000006854534, 0.02329999953508377, 0.06780000030994415, 0.04050000011920929, 0.024800000712275505, 0.07890000194311142, 0.11249999701976776, 0.0681999996304512, 0.008999999612569809, 0.014399999752640724, 0.033799998462200165, 0.03889999911189079, 0.006599999964237213, 0.012900000438094139, 0.011599999852478504, 0.009100000374019146, 0.041600000113248825, 0.09130000323057175, 0.08060000091791153, 0.04610000178217888, 0.037300001829862595, 0.09560000151395798, 0.07819999754428864, 0.06019999831914902, 0.009200000204145908, 0.013199999928474426, 0.0551999993622303, 0.04190000146627426, 0.009499999694526196, 0.018799999728798866, 0.040300000458955765, 0.028200000524520874, 0.052799999713897705, 0.08269999921321869, 0.11349999904632568, 0.10689999908208847, 0.04320000112056732, 0.05559999868273735, 0.08980000019073486, 0.08179999887943268, 0.011900000274181366, 0.028999999165534973, 0.10509999841451645, 0.07090000063180923, 0.03280000016093254, 0.015799999237060547, 0.019099999219179153, 0.03629999980330467, 0.02800000086426735, 0.045099999755620956, 0.048700001090765, 0.10890000313520432, 0.055799998342990875, 0.07259999960660934, 0.07270000129938126, 0.05249999836087227, 0.02070000022649765, 0.04600000008940697, 0.058400001376867294, 0.025499999523162842, 0.0551999993622303, 0.02630000002682209, 0.03319999948143959, 0.07329999655485153, 0.04749999940395355, 0.025200000032782555, 0.06620000302791595, 0.1120000034570694, 0.062199998646974564, 0.033900000154972076, 0.06599999964237213, 0.1362999975681305, 0.05119999870657921, 0.02759999968111515, 0.024900000542402267, 0.06830000132322311, 0.019899999722838402, 0.01590000092983246, 0.02930000051856041, 0.028599999845027924, 0.02810000069439411, 0.03519999980926514, 0.07850000262260437, 0.03709999844431877, 0.039000000804662704, 0.05090000107884407, 0.07649999856948853, 0.08209999650716782, 0.02500000037252903, 0.015200000256299973, 0.0471000000834465, 0.07530000060796738, 0.007699999958276749, 0.009200000204145908, 0.008500000461935997, 0.008700000122189522, 0.08950000256299973, 0.11569999903440475, 0.11339999735355377, 0.07410000264644623, 0.05290000140666962, 0.1014999970793724, 0.09799999743700027, 0.10660000145435333, 0.013299999758601189, 0.017400000244379044, 0.05990000069141388, 0.028699999675154686, 0.0066999997943639755, 0.008100000210106373, 0.008200000040233135, 0.007699999958276749, 0.03319999948143959, 0.07590000331401825, 0.08829999715089798, 0.08799999952316284, 0.040800001472234726, 0.10760000348091125, 0.1386999934911728, 0.07429999858140945, 0.01080000028014183, 0.012900000438094139, 0.03830000013113022, 0.021400000900030136, 0.006300000008195639, 0.007499999832361937, 0.011599999852478504, 0.007400000002235174, 0.030300000682473183, 0.039000000804662704, 0.08299999684095383, 0.05609999969601631, 0.031099999323487282, 0.06440000236034393, 0.11240000277757645, 0.050200000405311584, 0.011699999682605267, 0.010499999858438969, 0.016599999740719795, 0.022299999371170998, 0.013399999588727951, 0.01140000019222498, 0.012400000356137753, 0.027400000020861626, 0.02590000070631504, 0.05290000140666962, 0.08720000088214874, 0.0575999990105629, 0.02979999966919422, 0.06759999692440033, 0.09369999915361404, 0.08139999955892563, 0.013299999758601189, 0.017799999564886093, 0.01119999960064888, 0.0340999998152256, 0.020800000056624413, 0.01209999993443489, 0.012400000356137753, 0.052799999713897705, 0.029500000178813934, 0.044599998742341995, 0.09889999777078629, 0.066600002348423, 0.043699998408555984, 0.07599999755620956, 0.10930000245571136, 0.1460999995470047, 0.02319999970495701, 0.019300000742077827, 0.01360000018030405, 0.04839999973773956, 0.012000000104308128, 0.007499999832361937, 0.012199999764561653, 0.030899999663233757, 0.029999999329447746, 0.04399999976158142, 0.07959999889135361, 0.040800001472234726, 0.034299999475479126, 0.06960000097751617, 0.09059999883174896, 0.1151999980211258, 0.018300000578165054, 0.01080000028014183, 0.01769999973475933, 0.0364999994635582, 0.010200000368058681, 0.00570000009611249, 0.007899999618530273, 0.010099999606609344, 0.03220000118017197, 0.030899999663233757, 0.07159999758005142, 0.052299998700618744, 0.0340999998152256, 0.06459999829530716, 0.1177000030875206, 0.061400000005960464, 0.011699999682605267, 0.013199999928474426, 0.01899999938905239, 0.0203000009059906, 0.009800000116229057, 0.006599999964237213, 0.0066999997943639755, 0.007600000128149986, 0.04129999876022339, 0.08470000326633453, 0.09740000218153, 0.07699999958276749, 0.02969999983906746, 0.07050000131130219, 0.11559999734163284, 0.08709999918937683, 0.01209999993443489, 0.012299999594688416, 0.04129999876022339, 0.02329999953508377, 0.006099999882280827, 0.012199999764561653, 0.02280000038444996, 0.010999999940395355, 0.12189999967813492, 0.17100000381469727, 0.16410000622272491, 0.14069999754428864, 0.07479999959468842, 0.15860000252723694, 0.15629999339580536, 0.15080000460147858, 0.017100000753998756, 0.034299999475479126, 0.10440000146627426, 0.05310000106692314, 0.005200000014156103, 0.006200000178068876, 0.01209999993443489, 0.011500000022351742, 0.03929999843239784, 0.0697999969124794, 0.10480000078678131, 0.16290000081062317, 0.04520000144839287, 0.09139999747276306, 0.1193000003695488, 0.11640000343322754, 0.007199999876320362, 0.03400000184774399, 0.07320000231266022, 0.030899999663233757, 0.005400000140070915, 0.008500000461935997, 0.01979999989271164, 0.013899999670684338, 0.030700000002980232, 0.02160000056028366, 0.10180000215768814, 0.07959999889135361, 0.03449999913573265, 0.04540000110864639, 0.0957999974489212, 0.0869000032544136, 0.009600000455975533, 0.01889999955892563, 0.026399999856948853, 0.03610000014305115, 0.014700000174343586, 0.011699999682605267, 0.00839999970048666, 0.03440000116825104, 0.032499998807907104, 0.03269999846816063, 0.054999999701976776, 0.06849999725818634, 0.04899999871850014, 0.03020000085234642, 0.04089999943971634, 0.13590000569820404, 0.01590000092983246, 0.016200000420212746, 0.012799999676644802, 0.05389999970793724, 0.015399999916553497, 0.013000000268220901, 0.015200000256299973, 0.06840000301599503, 0.03099999949336052, 0.02889999933540821, 0.049300000071525574, 0.1177000030875206, 0.046799998730421066, 0.0560000017285347, 0.0674000009894371, 0.20250000059604645, 0.026900000870227814, 0.015799999237060547, 0.018400000408291817, 0.06769999861717224, 0.013299999758601189, 0.009700000286102295, 0.009600000455975533, 0.03319999948143959, 0.023499999195337296, 0.024700000882148743, 0.06599999964237213, 0.04699999839067459, 0.022299999371170998, 0.06840000301599503, 0.08169999718666077, 0.12200000137090683, 0.010700000450015068, 0.0142000000923872, 0.01860000006854534, 0.041999999433755875, 0.009200000204145908, 0.006599999964237213, 0.01549999974668026, 0.01730000041425228, 0.02410000003874302, 0.02810000069439411, 0.10459999740123749, 0.07100000232458115, 0.020400000736117363, 0.09690000116825104, 0.15240000188350677, 0.09470000118017197, 0.009499999694526196, 0.013299999758601189, 0.039400000125169754, 0.03909999877214432, 0.007000000216066837, 0.012900000438094139, 0.01360000018030405, 0.00989999994635582, 0.05000000074505806, 0.10570000112056732, 0.09459999948740005, 0.08479999750852585, 0.0430000014603138, 0.1177000030875206, 0.09070000052452087, 0.08169999718666077, 0.008799999952316284, 0.018200000748038292, 0.06419999897480011, 0.04859999939799309, 0.011699999682605267, 0.022700000554323196, 0.05469999834895134, 0.03720000013709068, 0.06459999829530716, 0.093299999833107, 0.15809999406337738, 0.15289999544620514, 0.0494999997317791, 0.05950000137090683, 0.12929999828338623, 0.11240000277757645, 0.011300000362098217, 0.03680000081658363, 0.13169999420642853, 0.07530000060796738, 0.026799999177455902, 0.013000000268220901, 0.021800000220537186, 0.03460000082850456, 0.027499999850988388, 0.04529999941587448, 0.06939999759197235, 0.12280000001192093, 0.05299999937415123, 0.08389999717473984, 0.11110000312328339, 0.06790000200271606, 0.021400000900030136, 0.05640000104904175, 0.07660000026226044, 0.023800000548362732, 0.03959999978542328, 0.01640000008046627, 0.02319999970495701, 0.055799998342990875, 0.03889999911189079, 0.0357000008225441, 0.08150000125169754, 0.10930000245571136, 0.04780000075697899, 0.06700000166893005, 0.10379999876022339, 0.14419999718666077, 0.03739999979734421, 0.021299999207258224, 0.019700000062584877, 0.060100000351667404, 0.0142000000923872, 0.01730000041425228, 0.03280000016093254, 0.02070000022649765, 0.029999999329447746, 0.05119999870657921, 0.11420000344514847, 0.05050000175833702, 0.03370000049471855, 0.07190000265836716, 0.1242000013589859, 0.10620000213384628, 0.023399999365210533, 0.012400000356137753, 0.06030000001192093, 0.08659999817609787

Expected result with doc IDS

[786559, 624520, 726827, 478228, 910443, 562031, 836073, 350515, 81360, 926288, 284987, 238816, 325973, 386465, 222086, 672007, 681296, 118458, 514474, 124745, 841134, 147438, 497229, 638357, 615210, 326242, 870358, 65659, 115992, 782607, 72752, 960557, 347002, 10531, 712060, 592139, 550209, 836076, 848622, 47315, 95832, 928103, 794886, 377281, 928097, 377926, 736827, 235455, 84976, 103598, 639964, 154680, 679251, 497199, 321677, 124164, 916930, 313472, 318892, 736856, 831947, 696595, 940406, 741999, 722818, 103505, 37825, 155950, 345985, 270368, 957972, 130411, 552560, 686710, 144534, 773781, 167832, 254296, 737310, 294368, 938387, 935201, 937942, 836077, 341838, 757955, 289394, 509877, 23353, 359941, 952767, 976170, 810146, 133944, 765305, 723422, 307262, 562022, 809944, 230737]

Original response in format (id, knn_dist) by manticoresearch

[(81360, 0.66139686), (325973, 0.68221766), (497229, 0.69750249), (47315, 0.71973938), (736827, 0.72894084), (84976, 0.72937477), (736856, 0.74145496), (741999, 0.74830496), (37825, 0.75178444), (270368, 0.75606418), (144534, 0.762124), (254296, 0.76440591), (737310, 0.76475453), (289394, 0.7720077), (23353, 0.7731936), (230737, 0.778316), (55327, 0.78544134), (280393, 0.78602827), (268684, 0.78796375), (58782, 0.79785007)]

Original response with score (looks like knn dist) in QDRant

[(786559, 0.7532910108566284), (624520, 0.7597317099571228), (726827, 0.7852178812026978), (478228, 0.7859099507331848), (910443, 0.786249577999115), (562031, 0.7894607782363892), (836073, 0.8059011697769165), (350515, 0.8092456459999084), (81360, 0.8132631182670593), (926288, 0.815800666809082), (284987, 0.820436954498291), (238816, 0.8224855065345764), (325973, 0.8259646892547607), (386465, 0.8286159634590149), (222086, 0.8301216959953308), (672007, 0.8306664228439331), (681296, 0.832095742225647), (118458, 0.8321948051452637), (514474, 0.8323566317558289), (124745, 0.8334718942642212), (841134, 0.8341284990310669), (147438, 0.8341652154922485), (497229, 0.8351660966873169), (638357, 0.8358363509178162), (615210, 0.8368045687675476), (326242, 0.8372406959533691), (870358, 0.8387916088104248), (65659, 0.8389719724655151), (115992, 0.8411566615104675), (782607, 0.8415353894233704), (72752, 0.841549813747406), (960557, 0.8426862359046936), (347002, 0.8431869149208069), (10531, 0.8434194326400757), (712060, 0.8434333205223083), (592139, 0.8443025350570679), (550209, 0.8459287285804749), (836076, 0.8469060063362122), (848622, 0.8479349613189697), (95832, 0.8483744263648987), (47315, 0.8483744263648987), (928103, 0.850817084312439), (794886, 0.8522862792015076), (377281, 0.8523092269897461), (928097, 0.8528671860694885), (377926, 0.8536683917045593), (736827, 0.8537802696228027), (235455, 0.8538074493408203), (84976, 0.854034423828125), (103598, 0.8544076085090637), (639964, 0.8554280996322632), (154680, 0.856099545955658), (679251, 0.856232762336731), (497199, 0.8566629886627197), (321677, 0.8578798770904541), (124164, 0.8584803938865662), (916930, 0.8585527539253235), (313472, 0.8589867353439331), (318892, 0.8608410358428955), (736856, 0.8610777854919434), (831947, 0.8626039028167725), (696595, 0.8629926443099976), (940406, 0.86502605676651), (741999, 0.8650462031364441), (722818, 0.866686999797821), (103505, 0.8668213486671448), (37825, 0.8670549988746643), (155950, 0.8678973913192749), (345985, 0.868622362613678), (270368, 0.8695195913314819), (957972, 0.8696350455284119), (130411, 0.8705007433891296), (552560, 0.8711114525794983), (686710, 0.8723277449607849), (144534, 0.8729971647262573), (773781, 0.8730197548866272), (167832, 0.8733871579170227), (254296, 0.8743031024932861), (737310, 0.874502420425415), (938387, 0.8764157295227051), (935201, 0.8766061663627625), (937942, 0.8767733573913574), (836077, 0.8782885670661926), (341838, 0.8783756494522095), (757955, 0.8783957958221436), (289394, 0.8786397576332092), (509877, 0.8788956999778748), (23353, 0.8793143033981323), (359941, 0.8800841569900513), (952767, 0.8800858855247498), (976170, 0.880486786365509), (810146, 0.8805068731307983), (133944, 0.8808507323265076), (765305, 0.8810558319091797), (723422, 0.8815526962280273), (307262, 0.881637692451477), (562022, 0.8817274570465088), (809944, 0.8817701935768127), (230737, 0.8822223544120789), (192664, 0.8826940059661865)]

doc id 81360 vector

0.028300,0.034500,0.039400,0.038500,0.076500,0.086300,0.068100,0.061200,0.044900,0.088800,0.107100,0.061400,0.025900,0.025900,0.034500,0.043700,0.012000,0.014700,0.024900,0.024600,0.082600,0.076200,0.076800,0.059000,0.057400,0.085100,0.096600,0.041200,0.009600,0.012500,0.022900,0.025100,0.004400,0.005700,0.008700,0.020500,0.051800,0.040200,0.060100,0.066400,0.031000,0.071700,0.088300,0.023800,0.003900,0.006200,0.015200,0.012100,0.004600,0.005100,0.006600,0.015100,0.028100,0.028900,0.049900,0.084500,0.016800,0.057400,0.046600,0.020300,0.005600,0.006000,0.011900,0.008000,0.007200,0.003400,0.006400,0.010500,0.019500,0.027700,0.039400,0.058000,0.012300,0.028300,0.043100,0.061400,0.008000,0.006400,0.008900,0.011400,0.003400,0.003400,0.007800,0.006100,0.023100,0.050000,0.039100,0.025000,0.009800,0.028800,0.044200,0.073400,0.004300,0.005300,0.007700,0.015400,0.005500,0.005000,0.015100,0.007300,0.035000,0.050700,0.056000,0.023400,0.010400,0.025600,0.060200,0.050200,0.003100,0.004700,0.011100,0.017500,0.013400,0.014900,0.020500,0.020500,0.047800,0.083700,0.075200,0.038000,0.016600,0.050000,0.075200,0.051400,0.010500,0.011300,0.017900,0.026000,0.035900,0.025500,0.028600,0.035800,0.056200,0.066900,0.048100,0.042100,0.059200,0.077700,0.111600,0.086200,0.034700,0.037400,0.057900,0.062100,0.022000,0.017900,0.027600,0.029400,0.060600,0.061500,0.058000,0.062400,0.069400,0.080600,0.080200,0.053700,0.011400,0.023900,0.047900,0.027200,0.007400,0.010300,0.020600,0.032400,0.067400,0.047500,0.042100,0.064300,0.051500,0.062800,0.075300,0.038400,0.004200,0.009500,0.023900,0.022800,0.005000,0.007800,0.014800,0.027200,0.053100,0.050200,0.051800,0.085700,0.033400,0.068900,0.048300,0.021000,0.004300,0.010500,0.023300,0.017700,0.008600,0.005000,0.014700,0.019000,0.029900,0.033600,0.059100,0.060800,0.019300,0.023700,0.057800,0.060700,0.006000,0.009800,0.020600,0.022000,0.002900,0.007400,0.014200,0.011000,0.029500,0.043900,0.028400,0.023000,0.030400,0.031600,0.044200,0.081500,0.004100,0.007500,0.018800,0.029300,0.009800,0.009300,0.018100,0.014200,0.032900,0.043000,0.031500,0.014500,0.015800,0.037400,0.069700,0.072700,0.003900,0.008600,0.027400,0.032900,0.021500,0.010700,0.023200,0.029000,0.037000,0.059300,0.050900,0.040000,0.021600,0.061700,0.062500,0.072900,0.016100,0.014800,0.024300,0.047500,0.039300,0.030900,0.060300,0.039400,0.087900,0.079800,0.031400,0.028000,0.078600,0.078600,0.033400,0.026800,0.050700,0.055600,0.053600,0.033600,0.016200,0.019400,0.024600,0.035200,0.059100,0.031500,0.029100,0.056000,0.049600,0.067200,0.039300,0.050800,0.023200,0.034400,0.039600,0.020800,0.013200,0.008000,0.012400,0.020800,0.064000,0.035400,0.045700,0.092900,0.051500,0.044500,0.061600,0.092200,0.011600,0.007000,0.019300,0.019800,0.023700,0.026500,0.037600,0.035900,0.055600,0.051200,0.026200,0.043900,0.034200,0.033700,0.037300,0.055700,0.026300,0.016200,0.022100,0.031000,0.023400,0.030100,0.036900,0.036100,0.073300,0.091600,0.095800,0.061800,0.043800,0.104000,0.133800,0.073100,0.018100,0.021000,0.032900,0.039600,0.009800,0.012500,0.026700,0.024600,0.080000,0.075200,0.108200,0.063300,0.053500,0.092500,0.116300,0.048800,0.007300,0.010600,0.025300,0.028800,0.003700,0.006300,0.011200,0.021300,0.054700,0.037600,0.072400,0.063900,0.031200,0.089200,0.115200,0.028100,0.003100,0.005100,0.020100,0.014200,0.004500,0.007100,0.010800,0.018600,0.030800,0.025300,0.058000,0.078100,0.017600,0.071400,0.061700,0.025400,0.004900,0.007200,0.013800,0.011400,0.008200,0.006600,0.010100,0.011500,0.020400,0.021100,0.051400,0.071200,0.010900,0.036500,0.065700,0.079900,0.006800,0.006700,0.010700,0.015800,0.004900,0.007600,0.012000,0.006800,0.027600,0.051600,0.042400,0.035400,0.012800,0.031800,0.058900,0.083200,0.003400,0.004800,0.010700,0.018800,0.005400,0.006000,0.018600,0.009000,0.038100,0.055200,0.071500,0.024200,0.011700,0.033100,0.082000,0.065900,0.003100,0.006900,0.013300,0.019900,0.011300,0.013700,0.024900,0.018800,0.047400,0.086100,0.080000,0.039800,0.014900,0.058800,0.096300,0.068700,0.007500,0.010600,0.019000,0.026200,0.027100,0.025300,0.028000,0.035000,0.056400,0.074500,0.050300,0.046200,0.063100,0.099100,0.144900,0.116200,0.018700,0.028300,0.061300,0.060700,0.016600,0.015500,0.032400,0.032300,0.058400,0.058600,0.080300,0.061000,0.068300,0.087400,0.100500,0.077100,0.009100,0.024900,0.056600,0.042600,0.008500,0.012100,0.028500,0.037800,0.073200,0.043600,0.048500,0.071000,0.052800,0.078100,0.100100,0.060600,0.003700,0.015700,0.033800,0.029400,0.006500,0.010600,0.026700,0.038400,0.060700,0.040200,0.069800,0.099800,0.036900,0.087700,0.060000,0.032400,0.004600,0.015800,0.033800,0.026400,0.010500,0.011900,0.026600,0.026700,0.032700,0.032600,0.085700,0.086400,0.022700,0.038700,0.084000,0.099800,0.007200,0.012600,0.028800,0.033900,0.005000,0.017400,0.026200,0.016000,0.033200,0.056500,0.043600,0.035500,0.039600,0.024400,0.054900,0.117600,0.005600,0.009000,0.028100,0.042400,0.008800,0.013800,0.023800,0.017200,0.040800,0.055000,0.053900,0.023500,0.022500,0.053300,0.118600,0.104900,0.005200,0.012800,0.037900,0.044200,0.016400,0.011400,0.027200,0.024700,0.034100,0.056800,0.071000,0.046200,0.021100,0.085400,0.105100,0.112000,0.009100,0.015400,0.032600,0.052500,0.027400,0.029100,0.052200,0.033400,0.082900,0.097900,0.061800,0.050200,0.073700,0.086600,0.047900,0.043300,0.033100,0.049000,0.065900,0.040900,0.016100,0.014800,0.021700,0.047200,0.068700,0.045500,0.065100,0.096600,0.048200,0.087200,0.054100,0.073400,0.018600,0.037100,0.053700,0.019900,0.014400,0.008400,0.021400,0.027300,0.093300,0.079600,0.104900,0.149500,0.076700,0.076100,0.104600,0.139600,0.014900,0.009000,0.025800,0.028000,0.016000,0.026100,0.037500,0.028600,0.055700,0.064100,0.046500,0.073100,0.040400,0.033900,0.058900,0.076100,0.023700,0.011500,0.025400,0.044300,0.025000,0.033000,0.041400,0.042000,0.086500,0.108700,0.122800,0.079900,0.046600,0.123200,0.172900,0.090400,0.021800,0.021700,0.038500,0.046700,0.010500,0.013400,0.030600,0.027300,0.090100,0.082800,0.124700,0.076200,0.055700,0.119200,0.147100,0.062100,0.009600,0.011500,0.029900,0.032200,0.004200,0.005400,0.013600,0.024700,0.059100,0.038300,0.082000,0.073600,0.031800,0.098900,0.146800,0.035600,0.003500,0.007300,0.023100,0.019700,0.004500,0.006100,0.012600,0.023700,0.032000,0.027700,0.063800,0.090200,0.020100,0.085000,0.072000,0.033200,0.005600,0.008900,0.017900,0.014200,0.007800,0.008400,0.011800,0.016700,0.022300,0.044000,0.060200,0.082700,0.011000,0.043100,0.072000,0.093700,0.007600,0.006500,0.015200,0.020800,0.003900,0.011100,0.015600,0.009300,0.030200,0.082500,0.053500,0.036700,0.015900,0.029300,0.071300,0.099400,0.004400,0.007200,0.014200,0.025200,0.005300,0.009100,0.022000,0.011700,0.040500,0.077800,0.085700,0.034500,0.018800,0.041800,0.099000,0.083900,0.004200,0.010400,0.018400,0.025600,0.012000,0.014700,0.029400,0.021800,0.046600,0.086900,0.094900,0.053000,0.016900,0.074900,0.124200,0.087500,0.009700,0.012000,0.022000,0.032600,0.031800,0.033200,0.035100,0.043500,0.064300,0.094900,0.071800,0.057100,0.076500,0.119900,0.184500,0.144100,0.024400,0.035100,0.076200,0.075800,0.018700,0.017500,0.041300,0.040700,0.064900,0.065900,0.092900,0.066300,0.075900,0.110300,0.125800,0.091000,0.012500,0.031200,0.070900,0.050600,0.009400,0.010600,0.036000,0.048100,0.080300,0.047500,0.057300,0.086600,0.057500,0.092200,0.122300,0.073200,0.005200,0.020700,0.042100,0.038500,0.007500,0.012500,0.032600,0.050000,0.064400,0.034500,0.064300,0.133900,0.042800,0.114000,0.078400,0.043300,0.007900,0.025000,0.046500,0.034400,0.010300,0.016600,0.032900,0.036800,0.034000,0.062700,0.088700,0.115700,0.026400,0.060800,0.106700,0.123400,0.006800,0.016500,0.040300,0.044100,0.004700,0.025200,0.034400,0.022800,0.041400,0.099100,0.054700,0.042800,0.048000,0.037800,0.071100,0.135000,0.007700,0.013700,0.037600,0.056900,0.010900,0.021900,0.033700,0.023800,0.050000,0.084700,0.068300,0.032500,0.034100,0.075500,0.148300,0.129200,0.009900,0.021400,0.052400,0.060500,0.017800,0.015700,0.035400,0.029000,0.036900,0.065200,0.096900,0.061400,0.030300,0.107300,0.131200,0.146300,0.011800,0.017500,0.038000,0.070100,0.032200,0.033300,0.055500,0.038100,0.085600,0.114200,0.071300,0.058100,0.083300,0.098700,0.057800,0.052800,0.041300,0.056300,0.080300,0.049300,0.020800,0.013800,0.024600,0.057500,0.075500,0.062500,0.090000,0.119700,0.054200,0.112100,0.068900,0.091400,0.025000,0.045800,0.067000,0.025200,0.017100,0.011800,0.027300,0.033800,0.117200,0.113800,0.144000,0.189300,0.096500,0.106200,0.140100,0.175100,0.017200,0.011900,0.034300,0.035800,0.017000,0.029500,0.043600,0.033700,0.063600,0.088100,0.060000,0.094100,0.048500,0.043900,0.073600,0.093300,0.028600,0.013800,0.033400,0.056100

doc id 786559 vector

0.011400,0.007700,0.013600,0.021600,0.018600,0.067400,0.076900,0.057100,0.021000,0.075300,0.091100,0.062700,0.012000,0.007900,0.013800,0.018800,0.006400,0.006200,0.007100,0.012900,0.019000,0.053300,0.043600,0.036600,0.018200,0.041800,0.073300,0.043800,0.007900,0.004500,0.012000,0.014300,0.004800,0.003800,0.005800,0.012600,0.018500,0.021100,0.033300,0.043500,0.017000,0.026000,0.054300,0.026100,0.004000,0.003700,0.010400,0.009700,0.009200,0.003100,0.012600,0.012200,0.027800,0.020300,0.042900,0.086400,0.022100,0.023400,0.070400,0.039900,0.008500,0.003000,0.013700,0.014700,0.018900,0.003700,0.025300,0.021900,0.045200,0.017200,0.056200,0.102600,0.042400,0.018400,0.068800,0.089700,0.020900,0.003800,0.021000,0.028800,0.008300,0.003500,0.012500,0.008400,0.024200,0.020000,0.052900,0.036000,0.026700,0.021200,0.033300,0.072000,0.010700,0.003300,0.010500,0.013500,0.003500,0.004300,0.007800,0.005500,0.016400,0.024100,0.046700,0.017600,0.015800,0.026400,0.032400,0.045500,0.003500,0.003700,0.008600,0.009500,0.006200,0.005800,0.010200,0.013200,0.019400,0.035100,0.068100,0.048800,0.018100,0.054700,0.059700,0.053100,0.008100,0.006300,0.007300,0.011800,0.008800,0.008400,0.031000,0.041300,0.037700,0.108800,0.091700,0.089200,0.038200,0.116600,0.102800,0.084200,0.011900,0.012600,0.034300,0.036600,0.011200,0.007200,0.014400,0.031500,0.043800,0.097200,0.086600,0.068100,0.040700,0.094300,0.117100,0.074800,0.010800,0.007300,0.031800,0.032000,0.010800,0.005200,0.011500,0.026100,0.044500,0.032100,0.051200,0.064100,0.033800,0.040600,0.055700,0.049000,0.004900,0.005000,0.013700,0.019600,0.010400,0.003600,0.016100,0.021200,0.040800,0.030400,0.040300,0.096000,0.030000,0.049700,0.072400,0.045000,0.007000,0.007700,0.014800,0.022900,0.011600,0.006800,0.031300,0.019000,0.051100,0.029700,0.046800,0.100400,0.050300,0.031500,0.057500,0.086500,0.011800,0.006300,0.025300,0.018400,0.009100,0.005700,0.015000,0.013400,0.036200,0.046000,0.052700,0.039200,0.040500,0.033200,0.032000,0.064700,0.009400,0.004800,0.009400,0.016600,0.003600,0.007900,0.014100,0.012400,0.032700,0.040900,0.051500,0.028800,0.037200,0.033400,0.054300,0.028700,0.006600,0.006100,0.010900,0.013500,0.008300,0.007600,0.027500,0.028500,0.042100,0.084900,0.112500,0.069200,0.042600,0.097100,0.106400,0.053700,0.013900,0.012600,0.019700,0.027700,0.011900,0.019800,0.049900,0.051700,0.030100,0.101000,0.068800,0.076000,0.027700,0.099000,0.075200,0.062700,0.009600,0.017600,0.054500,0.048600,0.013000,0.008600,0.015200,0.030900,0.029400,0.064200,0.042300,0.044600,0.041500,0.053400,0.071700,0.056500,0.009000,0.018800,0.046200,0.023500,0.009700,0.008100,0.024700,0.020300,0.029600,0.049900,0.045700,0.031300,0.028800,0.052800,0.048400,0.032600,0.010200,0.006400,0.019400,0.018800,0.007700,0.016400,0.043900,0.023000,0.044100,0.052000,0.065600,0.061400,0.030900,0.063800,0.048700,0.044000,0.011700,0.007600,0.017700,0.030000,0.011000,0.009600,0.014300,0.021100,0.022200,0.080900,0.080500,0.057800,0.025000,0.098100,0.116300,0.073300,0.011300,0.012300,0.018000,0.022100,0.006600,0.007000,0.006000,0.013100,0.020100,0.063200,0.048800,0.036600,0.022900,0.054200,0.090800,0.054100,0.008000,0.007100,0.014500,0.016700,0.003600,0.003900,0.005500,0.010000,0.019700,0.020600,0.040700,0.044600,0.020200,0.031400,0.063800,0.030800,0.005200,0.004000,0.009200,0.009600,0.009600,0.003600,0.013500,0.011500,0.032900,0.021000,0.054800,0.090300,0.027700,0.027900,0.073200,0.049000,0.008700,0.004100,0.011300,0.013800,0.018600,0.004100,0.027200,0.021700,0.050200,0.022200,0.059500,0.110100,0.051200,0.026400,0.075200,0.104600,0.022700,0.004500,0.019000,0.027900,0.008100,0.003500,0.014300,0.008200,0.025200,0.024700,0.055100,0.040400,0.032400,0.026600,0.043100,0.083500,0.010700,0.003600,0.010500,0.012300,0.004100,0.004800,0.006900,0.007000,0.017300,0.029300,0.051700,0.019400,0.019400,0.029200,0.036900,0.054000,0.003500,0.004400,0.006300,0.008300,0.005900,0.005200,0.011500,0.013700,0.021300,0.043900,0.078300,0.047100,0.021000,0.071000,0.075800,0.060500,0.006100,0.007500,0.008200,0.011900,0.008800,0.007700,0.032200,0.039200,0.046800,0.131700,0.099700,0.094400,0.049800,0.144900,0.133200,0.108600,0.012500,0.011800,0.044300,0.043700,0.008500,0.006800,0.014700,0.027600,0.046600,0.113300,0.096300,0.070600,0.051600,0.117400,0.143800,0.095800,0.014200,0.010200,0.041300,0.039800,0.008700,0.006900,0.009600,0.020100,0.048200,0.037900,0.049600,0.062300,0.041600,0.048700,0.061100,0.055100,0.004600,0.006700,0.013300,0.017800,0.011800,0.005100,0.017200,0.022900,0.051800,0.035000,0.036600,0.101100,0.042500,0.057800,0.063800,0.047200,0.008600,0.010500,0.014400,0.024400,0.012200,0.006900,0.037200,0.021600,0.064500,0.036100,0.046900,0.114500,0.067600,0.039200,0.057100,0.108800,0.014400,0.006800,0.028400,0.026400,0.010000,0.007900,0.017800,0.017100,0.043800,0.054700,0.050300,0.047200,0.052000,0.042400,0.033700,0.076900,0.011700,0.007400,0.010500,0.020000,0.004600,0.009800,0.014400,0.015100,0.036200,0.048600,0.056800,0.034400,0.045800,0.041700,0.056400,0.032700,0.007600,0.006300,0.009300,0.014800,0.009100,0.007700,0.031700,0.032500,0.048400,0.105900,0.134700,0.079400,0.048200,0.123200,0.133100,0.070400,0.010100,0.011600,0.022300,0.029500,0.013100,0.023400,0.059200,0.057100,0.034200,0.118800,0.085000,0.080800,0.032600,0.119000,0.094300,0.076300,0.012700,0.021600,0.067600,0.062500,0.015700,0.009200,0.018400,0.036200,0.037100,0.074100,0.046200,0.054600,0.049600,0.064200,0.087900,0.072200,0.012900,0.025000,0.058300,0.029000,0.011400,0.009000,0.031100,0.026800,0.037100,0.061500,0.051700,0.044500,0.037600,0.063700,0.053800,0.048900,0.010200,0.006400,0.024100,0.024500,0.008300,0.019700,0.052100,0.028400,0.051700,0.060100,0.080300,0.074900,0.039200,0.075100,0.058800,0.060300,0.013500,0.007400,0.023800,0.037600,0.013500,0.008900,0.015000,0.027500,0.023700,0.089100,0.083200,0.062900,0.028100,0.120300,0.117100,0.078000,0.026800,0.015600,0.019800,0.035600,0.006800,0.006800,0.006600,0.014900,0.021100,0.068600,0.054600,0.040300,0.027300,0.065000,0.099500,0.058200,0.012100,0.010300,0.017400,0.020500,0.003700,0.004100,0.006800,0.009800,0.020000,0.021000,0.048500,0.051000,0.021900,0.033000,0.069000,0.041500,0.004300,0.003000,0.008200,0.009600,0.010000,0.003300,0.011900,0.013400,0.034100,0.020800,0.057800,0.091800,0.028800,0.030700,0.070600,0.059900,0.011600,0.005000,0.012000,0.014300,0.018700,0.003700,0.028200,0.019800,0.052200,0.023400,0.059600,0.111000,0.053800,0.027600,0.077000,0.119700,0.026500,0.005400,0.020100,0.027500,0.008500,0.003300,0.013700,0.009800,0.026100,0.023700,0.048000,0.042100,0.033700,0.028000,0.050700,0.093900,0.012800,0.005400,0.011200,0.013700,0.004100,0.004300,0.007000,0.007100,0.019700,0.031200,0.051600,0.026500,0.021900,0.030800,0.045200,0.055000,0.004500,0.005700,0.008700,0.010200,0.006800,0.006000,0.012400,0.016500,0.023100,0.049700,0.084700,0.053400,0.024200,0.081600,0.073200,0.061500,0.010900,0.008900,0.008200,0.018300,0.012600,0.007400,0.038800,0.048900,0.051300,0.145700,0.106900,0.105000,0.055600,0.162400,0.135200,0.118400,0.020100,0.009800,0.045400,0.056400,0.010500,0.006900,0.016900,0.029500,0.050000,0.123400,0.106200,0.075600,0.059200,0.133400,0.153400,0.104800,0.016200,0.013300,0.040000,0.039300,0.009100,0.006700,0.009700,0.020800,0.050500,0.039100,0.061600,0.074800,0.045600,0.047600,0.067300,0.065900,0.006100,0.005300,0.013800,0.018200,0.013000,0.004300,0.018300,0.028400,0.054900,0.035400,0.047000,0.118100,0.047000,0.053400,0.058900,0.055700,0.013100,0.011300,0.016000,0.028500,0.013100,0.007100,0.045500,0.026100,0.070400,0.037100,0.051100,0.129400,0.075600,0.041300,0.058500,0.119000,0.017500,0.007600,0.033500,0.029900,0.011500,0.007700,0.018400,0.021700,0.046500,0.049500,0.034000,0.052400,0.056600,0.043100,0.050100,0.085100,0.014700,0.010600,0.012800,0.022700,0.005000,0.008100,0.012800,0.015600,0.040900,0.049000,0.059600,0.042900,0.052400,0.043900,0.071400,0.044900,0.010000,0.007600,0.012300,0.021000,0.011200,0.008000,0.036600,0.035500,0.053500,0.116800,0.148000,0.089900,0.051200,0.133000,0.135200,0.074700,0.014100,0.011700,0.021900,0.037600,0.015200,0.026600,0.060200,0.058600,0.036500,0.134600,0.099800,0.087100,0.034600,0.137100,0.107100,0.079400,0.014400,0.028700,0.073400,0.063000,0.018300,0.010900,0.020100,0.038500,0.042700,0.079600,0.050400,0.063400,0.053300,0.074900,0.096400,0.084000,0.016700,0.031900,0.066300,0.033300,0.012900,0.011000,0.037500,0.031700,0.041400,0.064700,0.059100,0.055900,0.044900,0.069600,0.055500,0.065700,0.011200,0.007500,0.029900,0.028800,0.008600,0.021400,0.054500,0.031500,0.056100,0.067900,0.092100,0.086400,0.046800,0.082700,0.060100,0.067100,0.013700,0.008700,0.024900,0.036300

latest tests failures

test 452 and 466 failed here

Write amplification details?

As a longtime user of Sphinx/Manticore, I'm really excited you guys are working on this. Thank you so much.

I love that you're designing the data structures and implementation in a way that requires so little memory. I've done some testing of mmap()ing disk-based array performance, and I'm really glad to see this project taking advantage of modern storage capabilities.

The question I have is around disk write amplification. RocksDB, LevelDB, and Clickhouse all suffer from pretty high write aplification. As someone who has had catastrophic failure from write amplifications like this, I'm extra cautious.

Do you have benchmarks/an overview of how the data is written to disk, and how it compares to other engines? If the implementation is straightforward and you'd rather point me there, is there somewhere in the code you could point me toward?

Again, I'm really excited you're working on this. I really love what you've all done in the last three years and feel that Manticore is such a far under-appreciated application. The engineering you're all doing is fantastic. Thank you.

Crash on loading ~400M docs with MVA

How to reproduce

On our perf server:

Start searchd:

Latest dev in /root/mva-crash-issue-test-situation/test

root@perf ~/mva-crash-issue-test-situation/test # searchd -c manticore.conf  
Manticore 6.0.5 ec5105cfb@230315 dev (columnar 2.0.5 bc5e8a2@230313) (secondary 2.0.5 bc5e8a2@230313)  
...  

Create table

mysql -P9315 -h0 < schema.sql  

Start loading data

root@perf ~/mva-crash-issue-test-situation/test # mysql -P9315 -h0 < data.sql  
ERROR 2013 (HY000) at line 375373: Lost connection to MySQL server during query  

Crash

The crash log is:

[Thu Mar 16 03:42:05.688 2023] [11106] rt: table person: diskchunk 213(102), segments 28  saved in 4.374480 (4.374876) sec, RAM saved/new 123154168/11144032 ratio 0.917020 (soft limit 123080373, conf limit 134217728)  
[Thu Mar 16 03:42:06.109 2023] [11095] rt: table person: diskchunk 214(103), segments 26  saved in 0.427138 (0.452878) sec, RAM saved/new 11077648/929376 ratio 0.867020 (soft limit 116369487, conf limit 134217728)  
[Thu Mar 16 03:42:46.655 2023] [11104] rt: table person: diskchunk 215(104), segments 24  saved in 4.534651 (4.535344) sec, RAM saved/new 116526232/10680192 ratio 0.916040 (soft limit 122948870, conf limit 134217728)  
[Thu Mar 16 03:43:26.491 2023] [11105] rt: table person: diskchunk 217(104), segments 26  saved in 4.708940 (4.709241) sec, RAM saved/new 122952060/10913976 ratio 0.918471 (soft limit 123275078, conf limit 134217728)  
------- FATAL: CRASH DUMP -------  
[Thu Mar 16 03:44:05.076 2023] [11094]  
  
--- crashed SphinxQL request dump ---  
INSERT INTO `person` (`id`, `social_platform_id`) VALUES ('375127001', ()), ('375127002',  
 ()), ('375127003', ()), ('375127004', ()), ('375127005', ()), ('375127006', ()),  
('375127007', ()), ('375127008', ()), ('375127009', ()), ('375127010', ()), ('375127011',  
 ()), ('375127012', ()), ('375127013', ()), ('375127014', ()), ('375127015', ()),  
('375127016', ()), ('375127017', ()), ('375127018', ()), ('375127019', ()), ('375127020',  
 ()), ('375127021', ()), ('375127022', ()), ('375127023', ()), ('375127024', ()),  
('375127025', ()), ('375127026', ()), ('375127027', ()), ('375127028', ()), ('375127029',  
 ()), ('375127030', ()), ('375127031', ()), ('375127032', ()), ('375127033', ()),  
('375127034', ()), ('375127035', ()), ('375127036', ()), ('375127037', ()), ('375127038',  
 ()), ('375127039', ()), ('375127040', ()), ('375127041', ()), ('375127042', ()),  
('375127043', ()), ('375127044', ()), ('375127045', ()), ('375127046', ()), ('375127047',  
 ()), ('375127048', ()), ('375127049', ()), ('375127050', ()), ('375127051', ()),  
('375127052', ()), ('375127053', ()), ('375127054', ()), ('375127055', ()), ('375127056',  
 ()), ('375127057', ()), ('375127058', ()), ('375127059', ()), ('375127060', ()),  
('375127061', ()), ('375127062', ()), ('375127063', ()), ('375127064', ()), ('375127065',  
 ()), ('375127066', ()), ('375127067', ()), ('375127068', ()), ('375127069', ()),  
...  
 ()), ('375127986', ()), ('375127987', ()), ('375127988', ()), ('375127989', ()),  
('375127990', ()), ('375127991', ()), ('375127992', ()), ('375127993', ()), ('375127994',  
 ()), ('375127995', ()), ('375127996', ()), ('375127997', ()), ('375127998', ()),  
('375127999', ()), ('375128000', ())  
--- request dump end ---  
--- local index:�5:  
Manticore 6.0.5 ec5105cfb@230315 dev (columnar 2.0.5 bc5e8a2@230313) (secondary 2.0.5 bc5e8a2@230313)  
Handling signal 11  
-------------- backtrace begins here ---------------  
Program compiled with Clang 15.0.4  
Configured with flags: Configured with these definitions: -DDISTR_BUILD=bionic -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.20 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore  
Built on Linux x86_64 (bionic) (cross-compiled)  
Stack bottom = 0x7f0b9c40b970, thread stack size = 0x20000  
Trying manual backtrace:  
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x1)  
Wrong stack limit or frame pointer, manual backtrace failed (fp=0x1, stack=0x7f0b9c410000, stacksize=0x20000)  
Trying system backtrace:  
begin of system symbols:  
searchd(_Z12sphBacktraceib 0x22a)[0x55a3542d9eca]  
searchd(_ZN11CrashLogger11HandleCrashEi 0x355)[0x55a35419b2b5]  
/lib/x86_64-linux-gnu/libpthread.so.0( 0x12980)[0x7f0bc4096980]  
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x2173d9)[0x7f0bc39093d9]  
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x216be4)[0x7f0bc3908be4]  
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x214bc4)[0x7f0bc3906bc4]  
/usr/share/manticore/modules/lib_manticore_columnar.so(_ZN8columnar9Builder_c7SetAttrEiPKli 0x40)[0x7f0bc3c7cb90]  
searchd(_Z15SetColumnarAttri8ESphAttrPN8columnar9Builder_iERSt10unique_ptrINS0_10Iterator_iESt14default_deleteIS4_EERN3sph8Vector_TIlNS9_13DefaultCopy_TIlEENS9_14DefaultRelimitENS9_16DefaultStorage_TIlEEEE 0x1b9)[0x55a3546d4819]  
searchd(_ZNK9RtIndex_c15WriteAttributesER21SaveDiskDataContext_tR10CSphString 0x6b2)[0x55a354586b22]  
searchd(_ZNK9RtIndex_c12SaveDiskDataEPKcRK11VecTraits_TI17CSphRefcountedPtrIK11RtSegment_tEERK12ChunkStats_tR10CSphString 0x16e)[0x55a354588bde]  
searchd(_ZN9RtIndex_c13SaveDiskChunkEbbb 0x5c7)[0x55a354584cb7]  
searchd(_ZZN7Threads11CoRoutine_c13CreateContextESt8functionIFvvEESt4pairIN5boost7context13stack_contextENS_14StackFlavour_EEEENUlNS6_6detail10transfer_tEE_8__invokeESB_ 0x1c)[0x55a354aa781c]  
searchd(make_fcontext 0x2f)[0x55a354ac7e9f]  
Trying boost backtrace:  
 0# sphBacktrace(int, bool) in searchd  
 1# CrashLogger::HandleCrash(int) in searchd  
 2# 0x00007F0BC4096980 in /lib/x86_64-linux-gnu/libpthread.so.0  
 3# 0x00007F0BC39093D9 in /usr/share/manticore/modules/lib_manticore_columnar.so  
 4# 0x00007F0BC3908BE4 in /usr/share/manticore/modules/lib_manticore_columnar.so  
 5# 0x00007F0BC3906BC4 in /usr/share/manticore/modules/lib_manticore_columnar.so  
 6# columnar::Builder_c::SetAttr(int, long const*, int) in /usr/share/manticore/modules/lib_manticore_columnar.so  
 7# SetColumnarAttr(int, ESphAttr, columnar::Builder_i*, std::unique_ptr<columnar::Iterator_i, std::default_delete<columnar::Iterator_i> >&, sph::Vector_T<long, sph::DefaultCopy_T<long>, sph::DefaultRelimit, sph::DefaultStorage_T<long> >&) in searchd  
 8# RtIndex_c::WriteAttributes(SaveDiskDataContext_t&, CSphString&) const in searchd  
 9# RtIndex_c::SaveDiskData(char const*, VecTraits_T<CSphRefcountedPtr<RtSegment_t const> > const&, ChunkStats_t const&, CSphString&) const in searchd  
10# RtIndex_c::SaveDiskChunk(bool, bool, bool) in searchd  
11# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::__invoke(boost::context::detail::transfer_t) in searchd  
12# make_fcontext in searchd  
  
-------------- backtrace ends here ---------------  
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)  
and attach there:  
a) searchd log, b) searchd binary, c) searchd symbols.  
Look into the chapter 'Reporting bugs' in the manual  
(https://manual.manticoresearch.com/Reporting_bugs)  
Dump with GDB via watchdog  
--- active threads ---  
thd 0 (work_11), proto mysql, state query, command insert  
--- Totally 3 threads, and 1 client-working threads ---  
------- CRASH DUMP END -------  
[Thu Mar 16 03:44:08.341 2023] [11093] watchdog: main process 11094 crashed via CRASH_EXIT (exit code 2), will be restarted  
[Thu Mar 16 03:44:08.341 2023] [11093] watchdog: main process 12266 forked ok  
[Thu Mar 16 03:44:08.385 2023] [12266] starting daemon version '6.0.5 ec5105cfb@230315 dev (columnar 2.0.5 bc5e8a2@230313) (secondary 2.0.5 bc5e8a2@230313)' ...  
[Thu Mar 16 03:44:08.385 2023] [12266] listening on 127.0.0.1:9315 for mysql  

This time it failed at ~372M docs:

mysql> select count(*) from person;  
 -----------   
| count(*)  |  
 -----------   
| 372380996 |  
 -----------   
1 row in set (0.47 sec)  

incorporate autotag into CI

Currently when we sync from gitlab to github this function is called which creates the needed tag and propagates it further. When we sync from github to gitlab it doesn't happen -

columnar/.github/workflows/mirror.yml

Lines 28 to 33 in 975172c

	echo "${{ secrets.GITLAB_SSH_KEY }}" > /tmp/ssh.key
	chmod 600 /tmp/ssh.key
	git remote add gitlab [email protected]:manticoresearch/columnar.git
	git checkout ${{ github.ref }}
	GIT_SSH_COMMAND='ssh -i /tmp/ssh.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -F /dev/null' git fetch gitlab
	GIT_SSH_COMMAND='ssh -i /tmp/ssh.key -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -F /dev/null' git push -u gitlab ${{ github.ref }}

The task is to fix it.

mysqlimport support?

Does your project support mysqlimport? I want to import the csv file into a table in rt mode through mysqlimport

Add SSE code to columnar scan

Some places in MCL code were intended to be written in SSE (and marked as such in comments), but were temporarily implemented in plain instructions. This needs to be fixed in order to improve search performance over columnar attributes.

lib_manticore_columnar wrong path

Hello
Under Debian 10

After Compiling Manticore columnar lib install in /share/manticore/modules
But manticore not found it

Then I copy libs to
/usr/share/manticore/modules/lib_manticore_columnar.so

And everything worked
Manticore 5.0.3 cff178311@220703 dev (columnar 1.16.1 637ddc6@220727) (secondary 1.16.1 637ddc6@220727)

Maybe it's a bug with the library directory

int -> bigint ALTER

Implement int->bigint ALTER support.

Crash on loading index.

I'm getting the following error trying to load the index from Postgresql materialized view.

  what():  It is possible we have a buffer overrun. You reported having allocated 8192 bytes for the compressed data but we needed 8228 bytes. Please increase the available memory for compressed data or check the value of the last parameter provided  to the encodeArray method.    
*** Oops, indexer crashed! Please send the following report to developers.    
Manticore 6.0.2 89c7a5139@230210 (columnar 2.0.0 a7c703d@230130) (secondary 2.0.0 a7c703d@230130)    
-------------- report begins here ---------------    
Current document: docid=851968, hits=14138020    
Current batch: minid=0, maxid=0    
Hit pool start: docid=0, hit=0    
-------------- backtrace begins here ---------------    
Program compiled with Clang 15.0.4    
Configured with flags: Configured with these definitions: -DDISTR_BUILD=focal -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=libzstd.so.1 -DWITH_CURL=1 -DDL_CURL=1 -DCURL_LIB=libcurl.so.4 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=libodbc.so.2 -DWITH_EXPAT=1 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.so.21 -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/lib/manticore/data -DFULL_SHARE_DIR=/usr/share/manticore    
Built on Linux x86_64 (focal) (cross-compiled)    
Stack bottom = 0x0, thread stack size = 0x20000    
Trying system backtrace:    
begin of system symbols:    
indexer(_Z12sphBacktraceib 0x22a)[0x55ed70d4a76a]    
indexer(_Z7sigsegvi 0xbb)[0x55ed70c86e5b]    
/lib/x86_64-linux-gnu/libpthread.so.0( 0x14420)[0x7f67858b8420]    
/lib/x86_64-linux-gnu/libc.so.6(gsignal 0xcb)[0x7f67856f500b]    
/lib/x86_64-linux-gnu/libc.so.6(abort 0x12b)[0x7f67856d4859]    
/lib/x86_64-linux-gnu/libstdc  .so.6( 0x9e911)[0x7f6785982911]    
/lib/x86_64-linux-gnu/libstdc  .so.6( 0xaa38c)[0x7f678598e38c]    
/lib/x86_64-linux-gnu/libstdc  .so.6( 0xaa3f7)[0x7f678598e3f7]    
/lib/x86_64-linux-gnu/libstdc  .so.6( 0xaa6a9)[0x7f678598e6a9]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x1cda26)[0x7f67852d4a26]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x1cc133)[0x7f67852d3133]    
/usr/share/manticore/modules/lib_manticore_columnar.so(_ZN4util10IntCodec_c6EncodeERKNS_6Span_TImEERSt6vectorIjSaIjEE 0x7b)[0x7f67852cd37b]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x215bd1)[0x7f678531cbd1]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x20fa8a)[0x7f6785316a8a]    
/usr/share/manticore/modules/lib_manticore_columnar.so( 0x20cce4)[0x7f6785313ce4]    
/usr/share/manticore/modules/lib_manticore_columnar.so(_ZN8columnar9Builder_c7SetAttrEil 0x3c)[0x7f678569782c]    
indexer( 0x672d9a)[0x55ed70ca6d9a]    
indexer(_ZN13CSphIndex_VLN5BuildERKN3sph8Vector_TIP10CSphSourceNS0_13DefaultCopy_TIS3_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS3_EEEEiiR17CSphIndexProgress 0x100d)[0x55ed70ca443d]    
indexer(_Z7DoIndexRK17CSphConfigSectionPKcRK15CSphOrderedHashIS_10CSphString15CSphStrHashFuncLi256EEP8_IO_FILE 0x1f09)[0x55ed70c84be9]    
indexer(main 0x27b9)[0x55ed70c89d29]    
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main 0xf3)[0x7f67856d6083]    
indexer(_start 0x2e)[0x55ed70c7bc7e]    
Trying boost backtrace:    
 0# sphBacktrace(int, bool) in indexer    
 1# sigsegv(int) in indexer    
 2# 0x00007F67858B8420 in /lib/x86_64-linux-gnu/libpthread.so.0    
 3# gsignal in /lib/x86_64-linux-gnu/libc.so.6    
 4# abort in /lib/x86_64-linux-gnu/libc.so.6    
 5# 0x00007F6785982911 in /lib/x86_64-linux-gnu/libstdc  .so.6    
 6# 0x00007F678598E38C in /lib/x86_64-linux-gnu/libstdc  .so.6    
 7# 0x00007F678598E3F7 in /lib/x86_64-linux-gnu/libstdc  .so.6    
 8# 0x00007F678598E6A9 in /lib/x86_64-linux-gnu/libstdc  .so.6    
 9# 0x00007F67852D4A26 in /usr/share/manticore/modules/lib_manticore_columnar.so    
10# 0x00007F67852D3133 in /usr/share/manticore/modules/lib_manticore_columnar.so    
11# util::IntCodec_c::Encode(util::Span_T<unsigned long> const&, std::vector<unsigned int, std::allocator<unsigned int> >&) in /usr/share/manticore/modules/lib_manticore_columnar.so    
12# 0x00007F678531CBD1 in /usr/share/manticore/modules/lib_manticore_columnar.so    
13# 0x00007F6785316A8A in /usr/share/manticore/modules/lib_manticore_columnar.so    
14# 0x00007F6785313CE4 in /usr/share/manticore/modules/lib_manticore_columnar.so    
15# columnar::Builder_c::SetAttr(int, long) in /usr/share/manticore/modules/lib_manticore_columnar.so    
16# 0x000055ED70CA6D9A in indexer    
17# CSphIndex_VLN::Build(sph::Vector_T<CSphSource*, sph::DefaultCopy_T<CSphSource*>, sph::DefaultRelimit, sph::DefaultStorage_T<CSphSource*> > const&, int, int, CSphIndexProgress&) in indexer    
18# DoIndex(CSphConfigSection const&, char const*, CSphOrderedHash<CSphConfigSection, CSphString, CSphStrHashFunc, 256> const&, _IO_FILE*) in indexer    
19# main in indexer    
20# __libc_start_main in /lib/x86_64-linux-gnu/libc.so.6    
21# _start in indexer    
    
-------------- backtrace ends here ---------------    

The config

{    
    
    sql_query       = SELECT * FROM test;    
    
    sql_attr_uint       = osm_id    
    sql_attr_string     = city_type    
    sql_attr_string     = subclass    
    sql_attr_string     = type    
    sql_attr_string     = country    
    sql_field_string    = postcode    
    sql_field_string    = region    
    sql_field_string    = district    
    sql_field_string    = city    
    sql_field_string    = place    
    sql_field_string    = street    
    sql_field_string    = housenumber    
    sql_field_string    = additional_housenumber    
    sql_field_string    = additional_text    
    sql_attr_uint       = city_level    
    sql_attr_float      = lat    
    sql_attr_float      = lon    
    
    
index streets    
{    
    columnar_attrs = id,geohash_coords,type    
    source            = streets    
    path            = /var/lib/manticore/streets    
    morphology        = stem_enru    
    index_exact_words = 1    
    expand_keywords   = 1    
    min_word_len      = 1    
    min_prefix_len    = 2    
    min_infix_len = 3    
}    
    
indexer    
{    
    mem_limit       = 2G    
    write_buffer    = 1G    
}    
    
searchd    
{    
    listen = 0.0.0.0:9312    
    listen = 0.0.0.0:9306:mysql    
    listen = 0.0.0.0:9308:http    
    log = /var/log/manticore/searchd.log    
    query_log = /var/log/manticore/query.log    
    pid_file = /var/run/manticore/searchd.pid    
    query_log_format = sphinxql    
    pseudo_sharding = 0    
    threads = 60    
    
   qcache_max_bytes = 0    
}    

The command:

indexer streets --rotate --config /etc/manticore.conf

What is weird is that almost the same dataset works fine. The difference is the size of it. About 400 MB.

Improve filtering performance by using SIMD predication

When columnar storage processes a filter, there are two heavy stages - unpacking data and processing the filter. Filtering performance can be improved by replacing conditions like if ( ( i==(ACCESSOR_VALUES)m_tValue ) ^ (!EQ) ) with predication using _mm256_cmpeq_epi32 and similar intrinsics.

Try Annoy instead of HNSW

We recently added HNSW support for KNN to this library. Together with Manticore Search it works so - https://manual.manticoresearch.com/dev/Searching/KNN#K-nearest-neighbor-search

It would be great to implement, test and bechmark https://github.com/spotify/annoy vs the integrated hnsw library in the context of Manticore Search.

Manticore columnar crashed on indexing

Manticore 3.5.5 02802ecb@210325 release (columnar 0.9.9 d39c836@210325) Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com) Copyright (c) 2017-2021, Manticore Software LTD (https://manticoresearch.com)
using config file '/etc/manticoresearch/manticore_actual.conf'...
indexing index 'tenders_arch'...
*** Oops, indexer crashed! Please send the following report to developers.
Manticore 3.5.5 02802ecb@210325 release
-------------- report begins here ---------------
Current document: docid=65536, hits=5620846
Current batch: minid=36615277, maxid=36865276
Hit pool start: docid=0, hit=0
-------------- backtrace begins here ---------------
Program compiled with 7
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=bionic -DUSE_SSL=ON
-DDL_UNIXODBC=1 -DUNIXODBC_LIB=libodbc.so.2 -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.so.1 -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysql
client.so.20 -DDL_PGSQL=1 -DPGSQL_LIB=libpq.so.5 -DLOCALDATADIR=/var/data -DFULL_SHARE_DIR=/usr/share/manticore -DUSE_RE2=1 -DUSE_ICU=
1 -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=1 -DWITH_ICONV=ON -DWITH_MYSQL=1 -DWITH_ODBC=ON -DWITH_PGSQL=1 -DWITH_RE2=1
-DWITH_STEMMER=1 -DWITH_ZLIB=ON -DGALERA_SONAME=libgalera_manticore.so.31 -DSYSCONFDIR=/etc/manticoresearch
Host OS is Linux x86_64
Stack bottom = 0x0, thread stack size = 0x20000
Trying system backtrace:
begin of system symbols:
/usr/bin/indexer(_Z12sphBacktraceib+0xcb)[0x5614cccf5a3b]
/usr/bin/indexer(_Z7sigsegvi+0xac)[0x5614ccc1313c]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980)[0x7f7a3f92b980]
/usr/lib/lib_manticore_columnar.so(_ZN11FastPForLib5Delta9fastDeltaIjEEvPT_m+0xc3)[0x7f7a3e1f7c42]
/usr/lib/lib_manticore_columnar.so(+0x36bc76)[0x7f7a3e1ebc76]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar13ComputeDeltasEPjib+0x2d)[0x7f7a3e1ec278]
/usr/lib/lib_manticore_columnar.so(ZN8columnar12Packer_MVA_TIjE13PrepareValuesERNS_6Span_TIjEERKS3+0x81)[0x7f7a3e1e4777]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar12Packer_MVA_TIjE21WritePacked_DeltaPFOREb+0x2ea)[0x7f7a3e1e3cbc]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar12Packer_MVA_TIjE11WriteToFileENS_12MvaPacking_eE+0x80)[0x7f7a3e1e2134]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar12Packer_MVA_TIjE5FlushEv+0x69)[0x7f7a3e1e17b3]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar12Packer_MVA_TIjE6AddDocEPKli+0x4e)[0x7f7a3e1e14ca]
/usr/lib/lib_manticore_columnar.so(_ZN8columnar9Builder_c7SetAttrEiPKli+0x4c)[0x7f7a3e23943a]
/usr/bin/indexer(_ZN13CSphIndex_VLN24Build_StoreColumnarAttrsElRN8columnar9Builder_iER10CSphSourceR19QueryMvaContainer_c+0xa3)[0x5614ccc97c23]
/usr/bin/indexer(_ZN13CSphIndex_VLN5BuildERKN3sph8Vector_TIP10CSphSourceNS0_13DefaultCopy_TIS3_EENS0_14DefaultRelimitENS0_16DefaultStorage_TIS3_EEEEii+0x10a1)[0x5614ccca4591]
/usr/bin/indexer(_Z7DoIndexRK17CSphConfigSectionPKcRK15CSphOrderedHashIS_10CSphString15CSphStrHashFuncLi256EEP8_IO_FILE+0x1df2)[0x5614ccc1c172]
/usr/bin/indexer(main+0x14d5)[0x5614ccc11dc5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f7a3e7e9bf7]
/usr/bin/indexer(_start+0x2a)[0x5614ccc12faa]
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the manual
(https://manual.manticoresearch.com/Reporting_bugs)
Will run gdb on '/usr/bin/indexer', pid '7769
'
Illegal process-id: 7769
.
No threads.
Main thread:
No stack.
Local variables:
No frame selected.
The program is not being run.

crash after 709b9aca

709b9ac leads to a crash which can be reproduced so:

snikolaev@dev2:~$ cat configless.conf
searchd {
   listen = 127.0.0.1:9315:mysql
   listen = 127.0.0.1:9316:http
   data_dir = data
   pid_file = 9315.pid
   log = searchd.log
   binlog_path =
}

snikolaev@dev2:~$ ~/manticore_github/build/src/searchd -c configless.conf
Manticore 6.2.13 f94555a29@230908 dev (columnar 2.2.5 709b9ac@230908) (secondary 2.2.5 709b9ac@230908)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

[56:31.368] [3994359] using config file '/home/snikolaev/configless.conf' (158 chars)...
starting daemon version '6.2.13 f94555a29@230908 dev (columnar 2.2.5 709b9ac@230908) (secondary 2.2.5 709b9ac@230908)' ...
listening on 127.0.0.1:9315 for mysql
listening on 127.0.0.1:9316 for sphinx and http(s)

snikolaev@dev2:~$ ~/manticore_github/test/clt-tests/mysqldump/scripts/generate-1m-records.sh |mysql -P9315 -h0

snikolaev@dev2:~$ mysql -P9315 -h0
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 6.2.13 f94555a29@230908 dev (columnar 2.2.5 709b9ac@230908) (secondary 2.2.5 709b9ac@230908) git branch HEAD (no branch)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> select * from t where s = 'Psm' order by id asc limit 20;
ERROR 2013 (HY000): Lost connection to MySQL server during query

backtrace:

 0# sphBacktrace(int, bool) in /home/snikolaev/manticore_github/build/src/searchd
 1# CrashLogger::HandleCrash(int) in /home/snikolaev/manticore_github/build/src/searchd
 2# 0x00007FA7029CC520 in /lib/x86_64-linux-gnu/libc.so.6
 3# 0x000055690504835A in /home/snikolaev/manticore_github/build/src/searchd
 4# bool CSphMatchQueue<MatchGeneric1_fn, false>::PushT<CSphMatch const&, CSphMatchQueue<MatchGeneric1_fn, false>::Push(CSphMatch const&)::{lambda(CSphMatch&, CSphMatch const&)#1}>(CSphMatch const&, CSphMa
tchQueue<MatchGeneric1_fn, false>::Push(CSphMatch const&)::{lambda(CSphMatch&, CSphMatch const&)#1}&&) in /home/snikolaev/manticore_github/build/src/searchd
 5# CSphMatchQueue<MatchGeneric1_fn, false>::Push(CSphMatch const&) in /home/snikolaev/manticore_github/build/src/searchd
 6# 0x0000556904F26299 in /home/snikolaev/manticore_github/build/src/searchd
 7# CSphIndex_VLN::RunFullscanOnIterator(RowidIterator_i*, CSphQueryContext const&, CSphQueryResultMeta&, VecTraits_T<ISphMatchSorter*> const&, CSphMatch&, int, bool, int, long) const in /home/snikolaev/ma
nticore_github/build/src/searchd
 8# CSphIndex_VLN::MultiScan(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&, long) const in /home/snikolaev/manticore_github/build/src/searchd
 9# CSphIndex_VLN::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /home/snikolaev/manticore_github/build/src/searchd
10# 0x0000556905CE4353 in /home/snikolaev/manticore_github/build/src/searchd
11# 0x0000556905E3100E in /home/snikolaev/manticore_github/build/src/searchd
12# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in /home/snikolaev/manticore_github/build/src/searchd
13# RtIndex_c::MultiQuery(CSphQueryResult&, CSphQuery const&, VecTraits_T<ISphMatchSorter*> const&, CSphMultiQueryArgs const&) const in /home/snikolaev/manticore_github/build/src/searchd
14# 0x0000556904DFEE05 in /home/snikolaev/manticore_github/build/src/searchd
15# 0x0000556905E3100E in /home/snikolaev/manticore_github/build/src/searchd
16# Threads::Coro::ExecuteN(int, std::function<void ()>&&) in /home/snikolaev/manticore_github/build/src/searchd
17# SearchHandler_c::RunLocalSearches() in /home/snikolaev/manticore_github/build/src/searchd
18# SearchHandler_c::RunSubset(int, int) in /home/snikolaev/manticore_github/build/src/searchd
19# SearchHandler_c::RunQueries() in /home/snikolaev/manticore_github/build/src/searchd
20# HandleMysqlSelect(RowBuffer_i&, SearchHandler_c&) in /home/snikolaev/manticore_github/build/src/searchd
21# ClientSession_c::Execute(std::pair<char const*, int>, RowBuffer_i&) in /home/snikolaev/manticore_github/build/src/searchd
22# ProcessSqlQueryBuddy(std::pair<char const*, int>, unsigned char&, GenericOutputBuffer_c&) in /home/snikolaev/manticore_github/build/src/searchd
23# SqlServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >) in /home/snikolaev/manticore_github/build/src/searchd
24# MultiServe(std::unique_ptr<AsyncNetBuffer_c, std::default_delete<AsyncNetBuffer_c> >, std::pair<int, unsigned short>, Proto_e) in /home/snikolaev/manticore_github/build/src/searchd
25# 0x0000556904E99DA1 in /home/snikolaev/manticore_github/build/src/searchd
26# Threads::CoRoutine_c::CreateContext(std::function<void ()>, std::pair<boost::context::stack_context, Threads::StackFlavour_E>)::{lambda(boost::context::detail::transfer_t)#1}::_FUN(boost::context::deta
il::transfer_t) in /home/snikolaev/manticore_github/build/src/searchd
27# make_fcontext in /home/snikolaev/manticore_github/build/src/searchd

Notes

• generate-1m-records.sh can be found here https://github.com/manticoresoftware/manticoresearch/blob/master/test/clt-tests/mysqldump/scripts/generate-1m-records.sh
• disabling SI prevents the crash
• it doesn't crash in the previous commit in the MCL