Can't import table with wordforms about manticoresearch

[sanikolaev]

I see the searchd does:

[pid 1252352] openat(AT_FDCWD, "/var/lib/manticore/t2/t2.meta", O_RDONLY) = 2106
[pid 1252352] read(2106, "{\n\"meta_created_time_utc\":\"2024-"..., 635) = 635
[pid 1252352] close(2106)               = 0
[pid 1252352] openat(AT_FDCWD, "wf", O_RDONLY) = -1 ENOENT (No such file or directory)

i.e. tries to open wf in the current dir unlike t2.meta which is opened by its absolute path.

echo "a > b" > /var/lib/manticore/wf

lets import table t2 from '/var/lib/manticore/wftest/wftest' pass.

This task blocks #1788

[sanikolaev]

The issue is not new:

root@a2ad520fb15e:/var/lib/manticore# searchd -v
Manticore 6.2.12 dc5144d35@230822

root@a2ad520fb15e:/var/lib/manticore# echo "a > b" > /tmp/wf;
root@a2ad520fb15e:/var/lib/manticore# mysql -h0 -P9306 -e "create table wftest(f text) wordforms='/tmp/wf'"
mysql -h0 -P9306 -e "insert into wftest(id) values(1)"
mysql -h0 -P9306 -e "import table t2 from '/var/lib/manticore/wftest/wftest'"
ERROR 1064 (42000) at line 1: unable to import table 't2': failed to open wf: No such file or directory

[PavelShilin89]

The same case with the exceptions

mysql> import table table3 from '/var/lib/manticore/table2/table2';
ERROR 1064 (42000): unable to import table 'table3': failed to prealloc: 'synonyms': failed to open exceptions.txt: No such file or directory

[tomatolog]

fixed at b5a3b7f now import table should copy external files well

[tomatolog]

@PavelShilin89
however the tests should be developed. All the tests should end as

• show index imported settings and all external files should be there
• daemon restart and should be no warning about missed external files
• after restart show index imported settings and all external files should be there

all indexes should have exceptions and multiple stopwords files and multiple wordforms files

cases:

• RT index defined in the config (not RT mode) with all external files located NOT at the index folder ( maybe /tmp ) should be imported into another table well
• RT index created in RT mode with all external files should be imported into another table well
• RT index created in RT mode with all external files and additional disk chunks should be imported into another table well

there could be another cases that I am unaware about maybe @PavelShilin89 or @sanikolaev could find more useful cases

[djklim87]

It partitionally work:
I tested it on test kit from your commit ghcr.io/manticoresoftware/manticoresearch:test-kit-b5a3b7f

docker run -it --rm ghcr.io/manticoresoftware/manticoresearch:test-kit-b5a3b7f bash

# searchd
Manticore 6.2.13 b5a3b7f1f@24051010 dev (columnar 2.2.5 478fff2@240417) (secondary 2.2.5 478fff2@240417) (knn 2.2.5 478fff2@240417)
....

# echo "a => b" > /var/lib/manticore/wordforms1.txt; echo "c => d" > /var/lib/manticore/exc.txt; echo "abcstop" > /var/lib/manticore/stop.txt

# ls -1 /var/lib/manticore/
binlog
exc.txt
state.sql
stop.txt
wordforms1.txt

mysql -h0 -P9306
mysql> CREATE TABLE table1 (title TEXT, tag INTEGER) exceptions='/var/lib/manticore/exc.txt' wordforms='/var/lib/manticore/wordforms1.txt' stopwords='/var/lib/manticore/stop.txt';
mysql> INSERT INTO table1 (id, title, tag) values (1, 'zxczxc', 77);
mysql> import table table2 from '/var/lib/manticore/table1/table1';
mysql> import table table3 from '/var/lib/manticore/table2/table2';
ERROR 1064 (42000): unable to import table 'table3': failed to prealloc: 'synonyms': failed to open exceptions.txt: No such file or directory

[tomatolog]

seems case more complex and needs one of the following

• drop support of the indexes with binary meta and sph headers for import table and rewrite load, fixup, save headers for import table
• hack loading of the indexes and push there new boolean to not generate error on missed external but keep thфе info about that external

[tomatolog]

at the 56052bf I fixed import table to copy external files.

[tomatolog]

it could be better to add to test described here #2154 (comment) the case to test that all externals are applied (exceptions, workdforms, stopwords) to prevent the case like #2207

[PavelShilin89]

Done in #2222