[Crash/Broken] Memory Issues after 5000 Inserts about manticoresearch

[tomatolog]

from the error message I see the crash is related to hnsw vector - could you provide your data as described in the manual or copy /var/lib/manticore/binlog/ from the container as it seems crash daemon as well on replaying commits

[def-roth]

I already guessed so as the error message is directly from hnswlib/hnswalg:1209.

Problem:
No vectors are provided at this point and they are empty. So in theory this shouldnt have any effect.

Background:
We are building a test suite and haven't reached this point yet.

Data
I've created a test set with lorem ipsum. Here's the binlog.tar.gz

[def-roth]

Update minimum reproducible example.

Seems like the schema is invalid / the combination not possible.

Info

Changing parameters changes the error message e.g.
knn_dims 1 -> 1024
hnsw_m 1 -> 16

[Wed Jul 10 14:53:10.300 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (351 chars)...
starting daemon version '6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.2:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9308 --bind=127.0.0.1  --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:37975
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[BUDDY]   local: 
[BUDDY]   extra: 
Crash!!! Handling signal 11
  1900K .......... .......... .......... .......... ........  100% 6.03M=0.4s

Minimal reproducible example in NodeJS

Following schemas work:
schema with knn, knn is ok (Dims 1, M 1)
schema with text, knn is ok (Dims 1024, M 16)

Following schemas error:
schema with text, knn not ok (Dims 1, M 1)
schema with text, knn, knn (1/1 & 1024/16)

const axios = require("axios");

await axios.post(
"http://localhost:9308/cli",
"CREATE TABLE knn_1(tex text, kvec float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='COSINE' hnsw_m='1',kvec1 float_vector knn_type='hnsw' knn_dims='1' hnsw_similarity='COSINE' hnsw_m='1')",
{
    headers: {
      'Content-Type': 'application/x-www-form-urlencoded'
}
});

const jsn = { tex: "test" };

  const ins = async () => {
    for (let i=0;i<25_001;i++) {
  
      try {
        const x = await axios.post(
            `http://localhost:9308/insert`,
            {
              index: tablename,
              id: 0,
              doc: jsn
            }
        );

        if (x.data.error)  return;

      } catch (e) {
        return;
      }
    }
  };

  const jobs = [];
  const x = 8;

  for (let i=0;i<x;i++) {
    jobs.push(ins());
  }

  await Promise.all(jobs);

[sanikolaev]

@def-roth how do I run your MRE? I get this:

snikolaev@dev2:~$ node 2403.js
/home/snikolaev/2403.js:4
await axios.post(
^^^^^

SyntaxError: await is only valid in async functions and the top level bodies of modules
    at internalCompileFunction (node:internal/vm:77:18)
    at wrapSafe (node:internal/modules/cjs/loader:1288:20)
    at Module._compile (node:internal/modules/cjs/loader:1340:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1435:10)
    at Module.load (node:internal/modules/cjs/loader:1207:32)
    at Module._load (node:internal/modules/cjs/loader:1023:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
    at node:internal/main/run_main_module:28:49

Node.js v20.11.1

sorry I don't have any experience with Node JS.

[tomatolog]

to replay binlog file you provided I need the table table ms_2983b6f8311c645b80730840429e807122c660_43586d99bf62550729c0f183ba12e39e88345ec9lyed3b9y difinition, could you post the output of the

show create table table ms_2983b6f8311c645b80730840429e807122c660_43586d99bf62550729c0f183ba12e39e88345ec9lyed3b9y;

[def-roth]

@sanikolaev sorrry I assumed a little experience, so this time i created a full drop in and added some console logs and comments.

Info:
I reran the MRE on another machine and it required a longer string to be indexed as text and more time to crash than on the lower spec machine - which is weird as I assumed manticore would impose limits on the memory usage (at least on the default settings I am running). My guess would be a memory leak.

Required packages

npm i axios 

JS

const axios = require("axios");


// the test function
async function run() {


    try {
        // drop the table if it exists
        const response = await axios.post(
            "http://localhost:9308/cli",
            "DROP TABLE knn_1",
            {
                headers: {
                    'Content-Type': 'application/x-www-form-urlencoded'
                }
            });
        console.log("Table dropped.")
    }
    catch (e) {
        // catch the error if the table does not exist
        console.error("Table did not exist.")
    }

    // create the table
    // if this errors there is something wrong so we dont catch it
    await axios.post(
        "http://localhost:9308/cli",
        "CREATE TABLE knn_1(fl float, tex text, kvec float_vector knn_type='hnsw' knn_dims='1024' hnsw_similarity='COSINE' hnsw_m='16', kvec1 float_vector knn_type='hnsw' knn_dims='1024' hnsw_similarity='COSINE' hnsw_m='16')",
        {
            headers: {
                'Content-Type': 'application/x-www-form-urlencoded'
            }
        });
    console.log("Table created.")


    // this is the inserted document
    const jsn = {
        fl: 0,
        tex: lorem,
    };

    // for stats we keep a counter of how many documents we have been inserted
    let cntr = 0;
    
    // this is the insertion function we run in parallel
    const ins = async () => {
        for (let i=0;i<25_001;i++) {

            // every 1000 documents we log the progress
            // nodeJS is single threaded (if you dont use worker threads)
            // so the counter keeps track of all parallel insertions
            ++cntr % 1000 === 0 && console.log(cntr, "inserted.");

            try {
                // we change the fl value to the counter to keep track of the insertions
                jsn.fl = cntr;
                const x = await axios.post(
                    `http://localhost:9308/insert`,
                    {
                        index: "knn_1",     // the table name
                        id: 0,              // enable auto id
                        doc: jsn            // the document
                    }
                );

                if (x.data.error) {
                    // if there is an error we log it and stop here
                    console.error(x.data.error);
                    return;
                }

            } catch (e) {
                // if there is an error we log it and stop here

                console.error("Manticore probably crashed.")
                console.error(e)
                console.error("This is bad.")

                // we throw the error to stop the process
                throw e;
            }
        }
    };

    // we create an array to hold the insertion functions and their promises
    const jobs = [];

    // we run 64 insertions in parallel to make use of libuv
    // this is the number of available threads in the thread pool
    const x = 64;
    for (let i=0;i<x;i++) {
        // we push the insertion function to the jobs array
        // which returns a promise
        jobs.push(ins());
    }

    // we wait for all insertions to finish
    // by waiting for all promises to resolve
    await Promise.all(jobs);

}


// this runs the test
run();


// this is the long text that is inserted into to manticore
// the backticks are used to create a multiline string
const lorem = `Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent at porta elit. Nunc semper dignissim arcu et laoreet. Nulla fermentum libero et mi varius, vitae euismod quam egestas. Nunc sodales odio dolor, id porta sem congue a. Donec id est sollicitudin, ornare nisl blandit, ultrices magna. Praesent quis ligula tortor. Proin neque est, facilisis eget faucibus a, tempus et neque. Donec malesuada posuere nibh eu bibendum. Donec aliquam iaculis magna, in venenatis elit auctor et. Morbi erat sem, porta et risus a, consectetur gravida nulla. Sed vel mi eu ante vulputate pulvinar.

Sed orci arcu, facilisis quis bibendum id, tincidunt non ligula. Pellentesque sem neque, hendrerit at vestibulum sed, accumsan non purus. Integer euismod tempor mattis. Quisque ullamcorper pulvinar interdum. Nunc eget ornare elit. Aliquam placerat sit amet metus tincidunt volutpat. Proin tincidunt neque quis est volutpat rhoncus. Vestibulum porta, libero ut dignissim sodales, magna tortor auctor orci, ut hendrerit lorem lacus mattis magna. Nullam massa ante, euismod vel magna quis, scelerisque dictum metus. Proin facilisis non purus vitae commodo. Praesent non urna id ante fermentum facilisis id eu enim. Nulla sodales risus eget nulla posuere, non gravida eros ultricies. Sed ac tellus molestie, tincidunt magna in, cursus odio. Nam quis rutrum nunc, vitae faucibus magna. Nullam cursus blandit nisi et posuere.

Vestibulum pulvinar nunc eu varius commodo. Vivamus interdum leo consequat, tincidunt nulla porttitor, pharetra mi. Suspendisse iaculis facilisis magna nec feugiat. Nunc ultricies dignissim fermentum. Sed enim leo, varius eget scelerisque ut, tempus a lorem. Aenean porta purus nisl, a viverra ipsum mattis consectetur. Integer id velit et metus vehicula vulputate. Vestibulum varius sem arcu, sit amet bibendum urna egestas eget.

Praesent vitae vestibulum eros. Nullam in pulvinar magna, sed auctor magna. Nunc sagittis, leo at placerat rutrum, odio odio sagittis lectus, sed imperdiet purus leo id ex. Mauris volutpat maximus dui ut commodo. Cras tincidunt, purus et aliquet sagittis, lacus tellus tincidunt turpis, quis bibendum nunc libero ut sapien. Etiam urna felis, dictum sed est non, ullamcorper sollicitudin massa. Maecenas tempus bibendum lectus, ut facilisis urna gravida egestas. Vivamus dictum elit eget lectus mattis, eget imperdiet eros varius. Quisque venenatis nibh risus, quis dapibus metus ornare vel. Curabitur nec sem ac nisi congue ornare. Suspendisse potenti. Donec sed ultrices mi. Morbi ut odio dictum, imperdiet ipsum fringilla, tincidunt arcu. Morbi purus felis, egestas vel risus eget, euismod dictum turpis. Pellentesque in metus malesuada, lacinia nulla id, auctor urna.

Aenean malesuada purus vitae nisl blandit, nec elementum quam varius. Cras pulvinar id sapien tristique accumsan. Fusce viverra lorem nisl, a pretium nunc lobortis sit amet. Curabitur commodo libero feugiat magna ullamcorper pretium. Suspendisse vel massa purus. Donec tincidunt mattis tortor ac porttitor. Curabitur id aliquet est, vitae tempus tortor. Phasellus ipsum est, vulputate maximus eleifend nec, sagittis at risus. Nunc pellentesque varius neque, nec commodo nisl tempus in. Morbi eget est mollis, maximus elit vitae, laoreet augue. Nunc eleifend leo consequat, vulputate urna sit amet, auctor velit.

Sed suscipit eros enim, id efficitur elit ultrices eu. Duis euismod neque felis. Mauris felis lectus, accumsan ut eros ac, dictum imperdiet arcu. Phasellus lacinia enim ac tellus mattis vestibulum. Aliquam sit amet vehicula sapien, in congue libero. Donec tellus risus, condimentum in consectetur quis, malesuada eu diam. Quisque bibendum semper ante ac hendrerit. Pellentesque mi ante, malesuada quis porta ut, consequat quis augue. Suspendisse tempus ultricies molestie. Etiam a odio bibendum, elementum purus a, sagittis enim. Nulla sodales magna malesuada accumsan mattis. Vestibulum dictum risus id rhoncus egestas. Sed imperdiet, lorem vel sodales elementum, quam nisi congue nibh, aliquet suscipit arcu ex vel tortor.

Vestibulum in est id ligula consequat tristique. Nulla quis ante a ligula pretium fermentum et ut augue. Sed eu leo orci. Aenean ultrices nunc sed elit dapibus, sed porta purus imperdiet. Duis nisl quam, fermentum sed interdum placerat, blandit ac erat. Praesent cursus, felis non commodo euismod, ipsum urna tincidunt leo, ut lacinia enim ex et magna. Aenean vitae orci nec diam efficitur tristique non et tellus. Sed ligula nunc, tristique eleifend hendrerit id, consectetur finibus ex. Mauris sed mattis mi, pellentesque finibus elit. Suspendisse odio nunc, posuere scelerisque fringilla sit amet, faucibus ac felis. Quisque mattis dapibus tellus quis hendrerit. Nunc in nulla ex. Sed tortor ligula, bibendum rutrum sapien ut, pellentesque iaculis velit. Pellentesque luctus, nunc in eleifend consequat, velit sapien ultricies magna, tincidunt vestibulum tortor nibh blandit sem. Maecenas finibus dui vitae dui varius, sit amet tristique arcu cursus.

Integer in mollis nisl, in eleifend est. Pellentesque et odio sit amet enim semper tincidunt. Nullam ac metus mattis, iaculis nunc at, facilisis massa. Sed malesuada fermentum turpis, vel faucibus lorem eleifend sed. Donec enim felis, laoreet pretium efficitur et, fringilla a risus. Mauris eu porta est. Class aptent taciti sociosqu ad litora torquent per conubia nostra, per inceptos himenaeos. Integer maximus risus aliquet, mollis felis lacinia, facilisis diam. Morbi varius aliquam nibh, varius mollis eros rutrum nec. Duis dolor neque, faucibus ut ultrices vel, efficitur sed velit. Duis eu tempus tortor, sed placerat purus. Donec nisl turpis, imperdiet pretium ultricies nec, aliquet nec sem.

Vivamus venenatis fringilla consequat. Proin quis risus et lectus maximus pretium. Nullam placerat scelerisque felis vitae vestibulum. Integer mauris purus, porta et mollis sed, elementum ac quam. Ut et tincidunt nulla. Duis eget congue nibh. Pellentesque sodales dictum nibh faucibus dictum. Integer laoreet maximus bibendum. Etiam sit amet placerat dolor. Fusce facilisis erat in orci lacinia pharetra. Nullam dignissim sit amet orci at fermentum. Donec nec tellus felis. Aenean tincidunt velit at eros convallis, sit amet pulvinar eros dapibus. Morbi aliquet orci justo, a varius elit laoreet imperdiet. Duis ac efficitur odio. Suspendisse fringilla ut ipsum at sagittis.

Morbi laoreet vestibulum purus. Vestibulum libero leo, ultricies vel arcu id, dictum luctus dui. Suspendisse potenti. Ut magna sapien, ullamcorper iaculis rutrum in, tristique ut est. Vestibulum vestibulum pharetra lorem, at consequat turpis convallis eget. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Vestibulum in velit malesuada, tristique velit in, laoreet metus. Vivamus quam massa, auctor eget elementum nec, malesuada imperdiet orci. Nullam mollis leo ut odio sagittis placerat. Nulla ac ornare erat. Vivamus at vulputate massa, eget feugiat lacus. Donec varius aliquet orci eu commodo.

Quisque eu turpis porta, interdum quam suscipit, ultrices magna. Proin congue faucibus urna, et porttitor dolor aliquam at. Maecenas non finibus nulla. Integer fermentum mauris a mauris auctor tempor. Interdum et malesuada fames ac ante ipsum primis in faucibus. Nulla nec rhoncus orci. Proin ultrices, erat et lobortis venenatis, nunc urna ullamcorper dui, ut interdum nisi libero sit amet arcu. Nam non finibus tellus, sed tincidunt turpis. Ut rhoncus tincidunt lacus eu tincidunt. Integer luctus nunc vel faucibus sodales.

Fusce ullamcorper in est suscipit euismod. Aenean molestie nisl vitae rutrum tincidunt. Quisque nec velit ac eros mollis molestie. Aenean aliquet vestibulum fringilla. Morbi vitae ex at magna semper tincidunt. Donec ultrices ullamcorper lectus, vitae blandit turpis vestibulum sit amet. Nunc quam ligula, ultricies quis tempus et, imperdiet ut elit. Sed ut auctor massa, vehicula scelerisque orci. Vivamus tincidunt augue et felis lacinia volutpat. Quisque ut venenatis lorem. Curabitur diam nibh, scelerisque quis hendrerit in, blandit quis felis. Donec sit amet libero sit amet dui malesuada suscipit a eget elit. Quisque commodo non risus eget viverra. Nulla volutpat augue in aliquet auctor. Sed eget ligula mi. Nulla a erat erat.

Vestibulum bibendum, purus in cursus pulvinar, leo ligula vestibulum mi, egestas sodales metus odio eget diam. In vestibulum, est vel fringilla maximus, felis diam volutpat est, eget dapibus metus dui et risus. Nam eu rhoncus orci, a consequat neque. Duis porttitor egestas nunc iaculis efficitur. Nullam tristique vitae nulla id viverra. Pellentesque in augue nec tellus lobortis interdum non at arcu. Cras maximus dictum luctus. Quisque sed tincidunt tortor. Vivamus rhoncus, metus id auctor ultricies, ligula nisi ultricies metus, malesuada tempor felis augue at dui.

Curabitur urna velit, vestibulum vitae lorem et, iaculis volutpat massa. Sed sagittis efficitur nisl pharetra convallis. Etiam pulvinar sed ex quis viverra. Aliquam erat volutpat. In quam arcu, porttitor eu convallis a, pretium id nisl. Etiam ultricies mollis sodales. Praesent cursus ullamcorper lectus, a lobortis velit suscipit quis. Morbi sit amet dapibus arcu. Quisque quis mollis mi. Fusce vulputate, quam non vulputate lacinia, ligula ligula laoreet sem, non mollis arcu est vulputate lacus. Quisque at nulla est.

Nullam sit amet posuere purus, quis dictum neque. Etiam commodo scelerisque arcu ut porta. Cras finibus nisl gravida laoreet pellentesque. Aenean quis justo ornare, suscipit ipsum vitae, vulputate urna. Nullam ut enim vitae leo dapibus aliquam quis vitae leo. Maecenas volutpat tempus facilisis. Suspendisse sed mauris sed quam elementum pellentesque.`

[def-roth]

@tomatolog Here's /var/lib/manticore/ for the errors as there are two different errors.

Files have been uploaded to filebin due to github's size restrictions. Link

Error 1

One is to the inputs for hnsw where dim='1' m='1' causes the following error.
(This is highly constructed and will never run in production anywhere)

[Thu Jul 11 07:28:09.419 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (351 chars)...
starting daemon version '6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.2:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9308 --bind=127.0.0.1  --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:41627
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[BUDDY]   local: 
[BUDDY]   extra: 
  1900K .......... .......... .......... .......... ........  100% 11.4M=0.1sterminate called after throwing an instance of 'std::runtime_error'
  what():  Not enough memory: addPoint failed to allocate linklist
Crash!!! Handling signal 6
terminate called recursively
Crash!!! Handling signal 11

Error 2

This one is with dim='1024' m='16' where the vector size is from gte large and M is the default and recommended by the authors
(This very probable to run in production)

[Thu Jul 11 07:36:00.423 2024] [1] using config file '/etc/manticoresearch/manticore.conf.sh' (351 chars)...
starting daemon version '6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206)' ...
listening on all interfaces for mysql, port=9306
listening on UNIX socket /var/run/mysqld/mysqld.sock
listening on all interfaces for sphinx and http(s), port=9308
listening on 172.17.0.2:9312 for sphinx and http(s)
prereading 0 tables
preread 0 tables in 0.000 sec
accepting connections
[BUDDY] started v2.3.10 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://0.0.0.0:9308 --bind=127.0.0.1  --threads=8 --skip=manticoresoftware/buddy-plugin-sharding --skip=manticoresoftware/buddy-plugin-queue' at http://127.0.0.1:34737
[BUDDY] Loaded plugins:
[BUDDY]   core: empty-string, backup, emulate-elastic, create, insert, alias, select, show, cli-table, plugin, test, alter-distributed-table, alter-rename-table, modify-table, knn, replace
[BUDDY]   local: 
[BUDDY]   extra: 
rt: table knn_1: diskchunk 0(1), segments 30  saved in 0.115931 (0.116133) sec, RAM saved/new 44746575/3237877 ratio 0.932522 (soft limit 125161034, conf limit 134217728)
rt: table knn_1: diskchunk 1(2), segments 31  saved in 0.343974 (0.344190) sec, RAM saved/new 125163373/9221341 ratio 0.931381 (soft limit 125007845, conf limit 134217728)
rt: table knn_1: diskchunk 2(3), segments 28  saved in 0.036312 (0.036526) sec, RAM saved/new 9060208/957419 ratio 0.881381 (soft limit 118296959, conf limit 134217728)
rt: table knn_1: diskchunk 3(4), segments 31  saved in 0.323049 (0.350102) sec, RAM saved/new 118297115/8393085 ratio 0.933751 (soft limit 125325952, conf limit 134217728)
rt: table knn_1: diskchunk 4(5), segments 31  saved in 0.339837 (0.340034) sec, RAM saved/new 125332937/9072841 ratio 0.932497 (soft limit 125157581, conf limit 134217728)
rt: table knn_1: diskchunk 5(6), segments 29  saved in 0.029090 (0.029364) sec, RAM saved/new 8888689/818426 ratio 0.882497 (soft limit 118446694, conf limit 134217728)
rt: table knn_1: diskchunk 6(7), segments 31  saved in 0.335010 (0.363862) sec, RAM saved/new 118448352/8559940 ratio 0.932603 (soft limit 125171895, conf limit 134217728)
rt: table knn_1: diskchunk 7(8), segments 32  saved in 0.366435 (0.366673) sec, RAM saved/new 125175993/9291686 ratio 0.930900 (soft limit 124943313, conf limit 134217728)
rt: table knn_1: diskchunk 8(9), segments 29  saved in 0.030866 (0.031076) sec, RAM saved/new 9046150/808816 ratio 0.880900 (soft limit 118232427, conf limit 134217728)
rt: table knn_1: diskchunk 9(10), segments 31  saved in 0.340033 (0.368973) sec, RAM saved/new 118237071/8812308 ratio 0.930639 (soft limit 124908214, conf limit 134217728)
rt: table knn_1: diskchunk 10(11), segments 31  saved in 0.357873 (0.385585) sec, RAM saved/new 124910645/9209918 ratio 0.931331 (soft limit 125001137, conf limit 134217728)
Crash!!! Handling signal 11
  1900K .......... .......... .......... .......... ........  100% 45.5M=0.09s

[def-roth]

Update describe

    +-------+--------------+----------------+
    | Field | Type         | Properties     |
    +-------+--------------+----------------+
    | id    | bigint       |                |
    | tex   | text         | indexed stored |
    | kvec  | float_vector | knn            |
    | kvec1 | float_vector | knn            |
    +-------+--------------+----------------+