Comments (10)
sanikolaev
commented on June 30, 2024
Data with Minio according to your docs
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097
I can't find issue-2097 on our side. Can you pls check it has been copied?
from manticoresearch.
sanikolaev
commented on June 30, 2024
How come it only use less then 1GB RAM when everything is mlock and preopened?
I see 27.8GB:
am I missing smth?
from manticoresearch.
Lot-Art
commented on June 30, 2024
htop memory
My bad, sorry.
Minio data on your amazon storage
This is from my history
696 curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o $HOME/minio-binaries/mc
697 chmod +x $HOME/minio-binaries/mc
700 ./mc config host add manticore http://s3.manticoresearch.com:9000 manticore manticore
715 sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097
(so the last step is with sudo)
And the contents:
$ ls -alhR /home/ME/minio-binaries/issue-2097/manticore
/home/ME/minio-binaries/issue-2097/manticore:
total 519M
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 .
drwxr-xr-x 3 root root 4.0K Apr 27 15:39 ..
-rw-r--r-- 1 root root 519M Apr 27 15:39 core
drwxr-xr-x 2 root root 12K Apr 27 15:43 la8_items_all
drwxr-xr-x 2 root root 4.0K Apr 27 15:43 la8_items_upcaft
-rw-r--r-- 1 root root 184 Apr 27 15:43 manticore.json
-rw-r--r-- 1 root root 0 Apr 27 15:43 state.sql
/home/ME/minio-binaries/issue-2097/manticore/la8_items_all:
total 40G
drwxr-xr-x 2 root root 12K Apr 27 15:43 .
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 ..
-rw-r--r-- 1 root root 355M Apr 27 15:39 la8_items_all.1.spa
-rw-r--r-- 1 root root 272M Apr 27 15:39 la8_items_all.1.spb
-rw-r--r-- 1 root root 561M Apr 27 15:40 la8_items_all.1.spd
-rw-r--r-- 1 root root 17M Apr 27 15:39 la8_items_all.1.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.1.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.1.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.1.spi
-rw-r--r-- 1 root root 302M Apr 27 15:39 la8_items_all.1.spidx
-rw-r--r-- 1 root root 466K Apr 27 15:39 la8_items_all.1.spm
-rw-r--r-- 1 root root 260M Apr 27 15:39 la8_items_all.1.spp
-rw-r--r-- 1 root root 20M Apr 27 15:39 la8_items_all.1.spt
-rw-r--r-- 1 root root 390M Apr 27 15:39 la8_items_all.10.spa
-rw-r--r-- 1 root root 314M Apr 27 15:40 la8_items_all.10.spb
-rw-r--r-- 1 root root 622M Apr 27 15:40 la8_items_all.10.spd
-rw-r--r-- 1 root root 18M Apr 27 15:39 la8_items_all.10.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.10.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.10.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.10.spi
-rw-r--r-- 1 root root 380M Apr 27 15:40 la8_items_all.10.spidx
-rw-r--r-- 1 root root 511K Apr 27 15:39 la8_items_all.10.spm
-rw-r--r-- 1 root root 229M Apr 27 15:39 la8_items_all.10.spp
-rw-r--r-- 1 root root 21M Apr 27 15:39 la8_items_all.10.spt
-rw-r--r-- 1 root root 389M Apr 27 15:40 la8_items_all.11.spa
-rw-r--r-- 1 root root 313M Apr 27 15:40 la8_items_all.11.spb
-rw-r--r-- 1 root root 621M Apr 27 15:40 la8_items_all.11.spd
-rw-r--r-- 1 root root 18M Apr 27 15:39 la8_items_all.11.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:39 la8_items_all.11.sph
-rw-r--r-- 1 root root 183K Apr 27 15:39 la8_items_all.11.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:39 la8_items_all.11.spi
-rw-r--r-- 1 root root 375M Apr 27 15:40 la8_items_all.11.spidx
-rw-r--r-- 1 root root 510K Apr 27 15:39 la8_items_all.11.spm
-rw-r--r-- 1 root root 231M Apr 27 15:40 la8_items_all.11.spp
-rw-r--r-- 1 root root 21M Apr 27 15:39 la8_items_all.11.spt
-rw-r--r-- 1 root root 397M Apr 27 15:40 la8_items_all.12.spa
-rw-r--r-- 1 root root 318M Apr 27 15:40 la8_items_all.12.spb
-rw-r--r-- 1 root root 620M Apr 27 15:40 la8_items_all.12.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.12.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.12.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.12.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.12.spi
-rw-r--r-- 1 root root 377M Apr 27 15:40 la8_items_all.12.spidx
-rw-r--r-- 1 root root 521K Apr 27 15:40 la8_items_all.12.spm
-rw-r--r-- 1 root root 237M Apr 27 15:40 la8_items_all.12.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.12.spt
-rw-r--r-- 1 root root 395M Apr 27 15:40 la8_items_all.13.spa
-rw-r--r-- 1 root root 310M Apr 27 15:40 la8_items_all.13.spb
-rw-r--r-- 1 root root 614M Apr 27 15:40 la8_items_all.13.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.13.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.13.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.13.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.13.spi
-rw-r--r-- 1 root root 372M Apr 27 15:40 la8_items_all.13.spidx
-rw-r--r-- 1 root root 518K Apr 27 15:40 la8_items_all.13.spm
-rw-r--r-- 1 root root 237M Apr 27 15:40 la8_items_all.13.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.13.spt
-rw-r--r-- 1 root root 392M Apr 27 15:40 la8_items_all.14.spa
-rw-r--r-- 1 root root 312M Apr 27 15:40 la8_items_all.14.spb
-rw-r--r-- 1 root root 612M Apr 27 15:41 la8_items_all.14.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.14.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.14.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.14.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.14.spi
-rw-r--r-- 1 root root 367M Apr 27 15:40 la8_items_all.14.spidx
-rw-r--r-- 1 root root 515K Apr 27 15:40 la8_items_all.14.spm
-rw-r--r-- 1 root root 238M Apr 27 15:40 la8_items_all.14.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.14.spt
-rw-r--r-- 1 root root 393M Apr 27 15:40 la8_items_all.15.spa
-rw-r--r-- 1 root root 312M Apr 27 15:40 la8_items_all.15.spb
-rw-r--r-- 1 root root 611M Apr 27 15:41 la8_items_all.15.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.15.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.15.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.15.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.15.spi
-rw-r--r-- 1 root root 365M Apr 27 15:41 la8_items_all.15.spidx
-rw-r--r-- 1 root root 516K Apr 27 15:40 la8_items_all.15.spm
-rw-r--r-- 1 root root 250M Apr 27 15:40 la8_items_all.15.spp
-rw-r--r-- 1 root root 22M Apr 27 15:40 la8_items_all.15.spt
-rw-r--r-- 1 root root 389M Apr 27 15:41 la8_items_all.16.spa
-rw-r--r-- 1 root root 308M Apr 27 15:41 la8_items_all.16.spb
-rw-r--r-- 1 root root 607M Apr 27 15:41 la8_items_all.16.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.16.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.16.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.16.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.16.spi
-rw-r--r-- 1 root root 359M Apr 27 15:41 la8_items_all.16.spidx
-rw-r--r-- 1 root root 510K Apr 27 15:40 la8_items_all.16.spm
-rw-r--r-- 1 root root 254M Apr 27 15:41 la8_items_all.16.spp
-rw-r--r-- 1 root root 21M Apr 27 15:40 la8_items_all.16.spt
-rw-r--r-- 1 root root 380M Apr 27 15:41 la8_items_all.2.spa
-rw-r--r-- 1 root root 290M Apr 27 15:41 la8_items_all.2.spb
-rw-r--r-- 1 root root 608M Apr 27 15:41 la8_items_all.2.spd
-rw-r--r-- 1 root root 18M Apr 27 15:40 la8_items_all.2.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:40 la8_items_all.2.sph
-rw-r--r-- 1 root root 183K Apr 27 15:40 la8_items_all.2.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:40 la8_items_all.2.spi
-rw-r--r-- 1 root root 322M Apr 27 15:41 la8_items_all.2.spidx
-rw-r--r-- 1 root root 499K Apr 27 15:41 la8_items_all.2.spm
-rw-r--r-- 1 root root 285M Apr 27 15:41 la8_items_all.2.spp
-rw-r--r-- 1 root root 21M Apr 27 15:41 la8_items_all.2.spt
-rw-r--r-- 1 root root 325M Apr 27 15:41 la8_items_all.20.spa
-rw-r--r-- 1 root root 247M Apr 27 15:41 la8_items_all.20.spb
-rw-r--r-- 1 root root 490M Apr 27 15:41 la8_items_all.20.spd
-rw-r--r-- 1 root root 15M Apr 27 15:41 la8_items_all.20.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.20.sph
-rw-r--r-- 1 root root 184K Apr 27 15:41 la8_items_all.20.sphi
-rw-r--r-- 1 root root 17M Apr 27 15:41 la8_items_all.20.spi
-rw-r--r-- 1 root root 286M Apr 27 15:41 la8_items_all.20.spidx
-rw-r--r-- 1 root root 426K Apr 27 15:41 la8_items_all.20.spm
-rw-r--r-- 1 root root 211M Apr 27 15:41 la8_items_all.20.spp
-rw-r--r-- 1 root root 18M Apr 27 15:41 la8_items_all.20.spt
-rw-r--r-- 1 root root 679M Apr 27 15:41 la8_items_all.21.spa
-rw-r--r-- 1 root root 519M Apr 27 15:41 la8_items_all.21.spb
-rw-r--r-- 1 root root 1.1G Apr 27 15:42 la8_items_all.21.spd
-rw-r--r-- 1 root root 31M Apr 27 15:41 la8_items_all.21.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.21.sph
-rw-r--r-- 1 root root 184K Apr 27 15:41 la8_items_all.21.sphi
-rw-r--r-- 1 root root 26M Apr 27 15:41 la8_items_all.21.spi
-rw-r--r-- 1 root root 583M Apr 27 15:42 la8_items_all.21.spidx
-rw-r--r-- 1 root root 892K Apr 27 15:41 la8_items_all.21.spm
-rw-r--r-- 1 root root 471M Apr 27 15:41 la8_items_all.21.spp
-rw-r--r-- 1 root root 37M Apr 27 15:41 la8_items_all.21.spt
-rw-r--r-- 1 root root 755M Apr 27 15:42 la8_items_all.22.spa
-rw-r--r-- 1 root root 584M Apr 27 15:42 la8_items_all.22.spb
-rw-r--r-- 1 root root 1.2G Apr 27 15:42 la8_items_all.22.spd
-rw-r--r-- 1 root root 36M Apr 27 15:41 la8_items_all.22.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.22.sph
-rw-r--r-- 1 root root 183K Apr 27 15:41 la8_items_all.22.sphi
-rw-r--r-- 1 root root 24M Apr 27 15:41 la8_items_all.22.spi
-rw-r--r-- 1 root root 645M Apr 27 15:42 la8_items_all.22.spidx
-rw-r--r-- 1 root root 991K Apr 27 15:41 la8_items_all.22.spm
-rw-r--r-- 1 root root 561M Apr 27 15:42 la8_items_all.22.spp
-rw-r--r-- 1 root root 41M Apr 27 15:41 la8_items_all.22.spt
-rw-r--r-- 1 root root 380M Apr 27 15:42 la8_items_all.3.spa
-rw-r--r-- 1 root root 297M Apr 27 15:41 la8_items_all.3.spb
-rw-r--r-- 1 root root 604M Apr 27 15:42 la8_items_all.3.spd
-rw-r--r-- 1 root root 18M Apr 27 15:41 la8_items_all.3.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:41 la8_items_all.3.sph
-rw-r--r-- 1 root root 183K Apr 27 15:41 la8_items_all.3.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:41 la8_items_all.3.spi
-rw-r--r-- 1 root root 323M Apr 27 15:42 la8_items_all.3.spidx
-rw-r--r-- 1 root root 499K Apr 27 15:41 la8_items_all.3.spm
-rw-r--r-- 1 root root 283M Apr 27 15:42 la8_items_all.3.spp
-rw-r--r-- 1 root root 21M Apr 27 15:41 la8_items_all.3.spt
-rw-r--r-- 1 root root 379M Apr 27 15:42 la8_items_all.4.spa
-rw-r--r-- 1 root root 301M Apr 27 15:42 la8_items_all.4.spb
-rw-r--r-- 1 root root 613M Apr 27 15:42 la8_items_all.4.spd
-rw-r--r-- 1 root root 19M Apr 27 15:42 la8_items_all.4.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.4.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.4.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.4.spi
-rw-r--r-- 1 root root 326M Apr 27 15:42 la8_items_all.4.spidx
-rw-r--r-- 1 root root 497K Apr 27 15:42 la8_items_all.4.spm
-rw-r--r-- 1 root root 282M Apr 27 15:42 la8_items_all.4.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.4.spt
-rw-r--r-- 1 root root 375M Apr 27 15:42 la8_items_all.5.spa
-rw-r--r-- 1 root root 295M Apr 27 15:42 la8_items_all.5.spb
-rw-r--r-- 1 root root 600M Apr 27 15:42 la8_items_all.5.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.5.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.5.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.5.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.5.spi
-rw-r--r-- 1 root root 327M Apr 27 15:42 la8_items_all.5.spidx
-rw-r--r-- 1 root root 493K Apr 27 15:42 la8_items_all.5.spm
-rw-r--r-- 1 root root 277M Apr 27 15:42 la8_items_all.5.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.5.spt
-rw-r--r-- 1 root root 397M Apr 27 15:42 la8_items_all.6.spa
-rw-r--r-- 1 root root 312M Apr 27 15:42 la8_items_all.6.spb
-rw-r--r-- 1 root root 596M Apr 27 15:43 la8_items_all.6.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.6.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.6.sph
-rw-r--r-- 1 root root 184K Apr 27 15:42 la8_items_all.6.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.6.spi
-rw-r--r-- 1 root root 352M Apr 27 15:42 la8_items_all.6.spidx
-rw-r--r-- 1 root root 522K Apr 27 15:42 la8_items_all.6.spm
-rw-r--r-- 1 root root 262M Apr 27 15:42 la8_items_all.6.spp
-rw-r--r-- 1 root root 22M Apr 27 15:42 la8_items_all.6.spt
-rw-r--r-- 1 root root 387M Apr 27 15:42 la8_items_all.7.spa
-rw-r--r-- 1 root root 307M Apr 27 15:42 la8_items_all.7.spb
-rw-r--r-- 1 root root 595M Apr 27 15:43 la8_items_all.7.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.7.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.7.sph
-rw-r--r-- 1 root root 184K Apr 27 15:42 la8_items_all.7.sphi
-rw-r--r-- 1 root root 16M Apr 27 15:42 la8_items_all.7.spi
-rw-r--r-- 1 root root 351M Apr 27 15:43 la8_items_all.7.spidx
-rw-r--r-- 1 root root 509K Apr 27 15:42 la8_items_all.7.spm
-rw-r--r-- 1 root root 258M Apr 27 15:43 la8_items_all.7.spp
-rw-r--r-- 1 root root 21M Apr 27 15:42 la8_items_all.7.spt
-rw-r--r-- 1 root root 394M Apr 27 15:43 la8_items_all.8.spa
-rw-r--r-- 1 root root 317M Apr 27 15:43 la8_items_all.8.spb
-rw-r--r-- 1 root root 601M Apr 27 15:43 la8_items_all.8.spd
-rw-r--r-- 1 root root 18M Apr 27 15:42 la8_items_all.8.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:42 la8_items_all.8.sph
-rw-r--r-- 1 root root 183K Apr 27 15:42 la8_items_all.8.sphi
-rw-r--r-- 1 root root 15M Apr 27 15:42 la8_items_all.8.spi
-rw-r--r-- 1 root root 377M Apr 27 15:43 la8_items_all.8.spidx
-rw-r--r-- 1 root root 517K Apr 27 15:42 la8_items_all.8.spm
-rw-r--r-- 1 root root 245M Apr 27 15:43 la8_items_all.8.spp
-rw-r--r-- 1 root root 22M Apr 27 15:42 la8_items_all.8.spt
-rw-r--r-- 1 root root 401M Apr 27 15:43 la8_items_all.9.spa
-rw-r--r-- 1 root root 320M Apr 27 15:43 la8_items_all.9.spb
-rw-r--r-- 1 root root 614M Apr 27 15:43 la8_items_all.9.spd
-rw-r--r-- 1 root root 18M Apr 27 15:43 la8_items_all.9.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_all.9.sph
-rw-r--r-- 1 root root 183K Apr 27 15:43 la8_items_all.9.sphi
-rw-r--r-- 1 root root 17M Apr 27 15:43 la8_items_all.9.spi
-rw-r--r-- 1 root root 391M Apr 27 15:43 la8_items_all.9.spidx
-rw-r--r-- 1 root root 526K Apr 27 15:43 la8_items_all.9.spm
-rw-r--r-- 1 root root 231M Apr 27 15:43 la8_items_all.9.spp
-rw-r--r-- 1 root root 22M Apr 27 15:43 la8_items_all.9.spt
-rw-r--r-- 1 root root 0 Apr 27 15:43 la8_items_all.lock
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_all.meta
-rw-r--r-- 1 root root 28 Apr 27 15:43 la8_items_all.ram
-rw-r--r-- 1 root root 152 Apr 27 15:43 la8_items_all.settings
/home/ME/minio-binaries/issue-2097/manticore/la8_items_upcaft:
total 604M
drwxr-xr-x 2 root root 4.0K Apr 27 15:43 .
drwxr-xr-x 4 root root 4.0K Apr 27 15:43 ..
-rw-r--r-- 1 root root 122M Apr 27 15:43 la8_items_upcaft.0.spa
-rw-r--r-- 1 root root 92M Apr 27 15:43 la8_items_upcaft.0.spb
-rw-r--r-- 1 root root 189M Apr 27 15:43 la8_items_upcaft.0.spd
-rw-r--r-- 1 root root 5.4M Apr 27 15:43 la8_items_upcaft.0.spe
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_upcaft.0.sph
-rw-r--r-- 1 root root 174K Apr 27 15:43 la8_items_upcaft.0.sphi
-rw-r--r-- 1 root root 9.0M Apr 27 15:43 la8_items_upcaft.0.spi
-rw-r--r-- 1 root root 103M Apr 27 15:43 la8_items_upcaft.0.spidx
-rw-r--r-- 1 root root 160K Apr 27 15:43 la8_items_upcaft.0.spm
-rw-r--r-- 1 root root 79M Apr 27 15:43 la8_items_upcaft.0.spp
-rw-r--r-- 1 root root 6.6M Apr 27 15:43 la8_items_upcaft.0.spt
-rw-r--r-- 1 root root 0 Apr 27 15:43 la8_items_upcaft.lock
-rw-r--r-- 1 root root 2.6K Apr 27 15:43 la8_items_upcaft.meta
-rw-r--r-- 1 root root 28 Apr 27 15:43 la8_items_upcaft.ram
-rw-r--r-- 1 root root 152 Apr 27 15:43 la8_items_upcaft.settings
So only root can write. Does that cause a problem?
from manticoresearch.
Lot-Art
commented on June 30, 2024
Actually, diskchunk save (FLUSH RAMCHUNK) also does some damage. Here is the searchd.log
[Sun Apr 28 00:13:29.169 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=108.017 Mb, time delta=15608 sec, took=0.803 sec)
[Sun Apr 28 00:43:09.907 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=61.208 Mb, time delta=15680 sec, took=0.309 sec)
[Sun Apr 28 02:27:57.087 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=133.525 Mb, time delta=8067 sec, took=0.882 sec)
[Sun Apr 28 02:52:41.109 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=72.313 Mb, time delta=7771 sec, took=0.398 sec)
[Sun Apr 28 05:29:14.738 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=152.845 Mb, time delta=10877 sec, took=1.059 sec)
[Sun Apr 28 06:20:37.889 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=89.146 Mb, time delta=12476 sec, took=0.555 sec)
[Sun Apr 28 11:10:18.047 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=190.684 Mb, time delta=20463 sec, took=1.682 sec)
[Sun Apr 28 12:19:31.129 2024] [47321] rt: table la8_items_upcaft: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=105.370 Mb, time delta=21533 sec, took=0.800 sec)
[Sun Apr 28 14:20:17.890 2024] [47321] rt: table la8_items_all: diskchunk 24(15), segments 32 saved in 66.333469 (66.991552) sec, RAM saved/new 715542869/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Sun Apr 28 14:21:27.500 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541020904, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:34:09.807 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 518257623, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:34:11.285 2024] [47319] WARNING: last message repeated 1 times
[Sun Apr 28 14:35:25.443 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:39:08.996 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541020904, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 14:58:40.769 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 536228500, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:08:17.958 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 518254642, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:10:56.929 2024] [47321] rt: table la8_items_all: ramchunk saved ok (mode=forced, last TID=0, current TID=0, ram=841.860 Mb, time delta=3039 sec, took=3.889 sec)
[Sun Apr 28 15:12:23.710 2024] [47321] rt: table la8_items_all: diskchunk 25(16), segments 28 forcibly saved in 81.681049 (81.720773) sec, RAM saved/new 883533090/0 ratio 0.950000 (soft limit 2040109465, conf limit 2147483648)
[Sun Apr 28 15:14:45.663 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:15:05.424 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:15:19.888 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 443784822, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:23:11.464 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 541016708, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:33:31.143 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:33:31.143 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:26.964 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:26.964 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:33.732 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:35:33.732 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.24.spd' at 776702601, from mmap of 223816177, query most probably would FAIL; report the fact to dev!
[Sun Apr 28 15:36:27.853 2024] [47321] WARNING: INTERNAL: out-of-range in ThinMMapReader_c: trying to read '/var/lib/manticore/la8_items_all/la8_items_all.25.spd' at 388795699, from mmap of 276017136, query most probably would FAIL; report the fact to dev!
(.. it keeps goind like that)
You can see after a diskchunk log the warnings are happening. After the 2nd diskchunk save 2 chunks report the same warning (and the log is right, the query does fail).
Should I open a separate issue for this (its the same data, same server, same version, same same same)?
from manticoresearch.
Lot-Art
commented on June 30, 2024
This is in searchd.log after a restart
[Tue Apr 30 22:42:06.976 2024] [469662] watchdog: main process 469663 forked ok
[Tue Apr 30 22:42:06.987 2024] [469663] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 10.164.0.36:9312 for sphinx and http(s)
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9312 for sphinx and http(s)
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 10.164.0.36:9306 for mysql
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9306 for mysql
[Tue Apr 30 22:42:06.987 2024] [469663] listening on 127.0.0.1:9307 for VIP mysql
[Tue Apr 30 22:42:11.589 2024] [469665] WARNING: table 'la8_items_upcaft': disabled at the JSON config
[Tue Apr 30 22:42:11.590 2024] [469665] WARNING: table 'la8_items_upcaft': prealloc: failed to lock /var/lib/manticore/la8_items_upcaft/la8_items_upcaft.lock: Resource temporarily unavailable - NOT SERVING
[Tue Apr 30 22:42:11.592 2024] [469664] prereading 1 tables
[Tue Apr 30 22:42:11.637 2024] [469663] accepting connections
[Tue Apr 30 22:42:12.926 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(124): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+SETT...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:12.941 2024] [469667] [BUDDY] restarting
[Tue Apr 30 22:42:14.073 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(139): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+VARI...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:14.089 2024] [469667] [BUDDY] restarting
[Tue Apr 30 22:42:15.225 2024] [469667] WARNING: [BUDDY] invalid output, should be 'Buddy ver, started address:port', got '
Fatal error: Uncaught Manticoresearch\Buddy\Core\Error\ManticoreSearchClientError: Cannot connect to server at http://10.164.0.36:9312/sql?mode=raw in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php:133
Stack trace:
#0 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(139): Manticoresearch\Buddy\Core\ManticoreSearch\Client->sendRequest('query=SHOW+VARI...')
#1 /usr/share/manticore/modules/manticore-buddy/src/Lib/QueryProcessor.php(95): Manticoresearch\Buddy\Base\Lib\QueryProcessor::fetchManticoreSettings()
#2 /usr/share/manticore/modules/manticore-buddy/src/main.php(30): Manticoresearch\Buddy\Base\Lib\QueryProcessor::init()
#3 {main}
thrown in /usr/share/manticore/modules/manticore-buddy/vendor/manticoresoftware/buddy-core/src/ManticoreSearch/Client.php on line 133
'
[Tue Apr 30 22:42:15.226 2024] [469667] [BUDDY] restart amount of attempts (3) has been exceeded
[Tue Apr 30 22:42:39.258 2024] [469664] preread 1 tables in 27.665 sec"
And cannot connect with mysql -h 0 -P 9306. It just waits forever (until i press CTRL+C to get back my terminal).
Then when I restart manticore again, this is the searchd.log
[Tue Apr 30 22:47:49.898 2024] [469779] watchdog: main process 469780 forked ok
[Tue Apr 30 22:47:49.899 2024] [469780] starting daemon version '6.2.12 dc5144d35@230822 (columnar 2.2.4 5aec342@230822) (secondary 2.2.4 5aec342@230822)' ...
[Tue Apr 30 22:47:49.900 2024] [469780] listening on 10.164.0.36:9312 for sphinx and http(s)
[Tue Apr 30 22:47:49.900 2024] [469780] listening on 127.0.0.1:9312 for sphinx and http(s)
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 10.164.0.36:9306 for mysql
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 127.0.0.1:9306 for mysql
[Tue Apr 30 22:47:49.901 2024] [469780] listening on 127.0.0.1:9307 for VIP mysql
[Tue Apr 30 22:47:53.648 2024] [469782] WARNING: table 'la8_items_upcaft': disabled at the JSON config
[Tue Apr 30 22:47:53.648 2024] [469782] WARNING: table 'la8_items_upcaft': prealloc: failed to lock /var/lib/manticore/la8_items_upcaft/la8_items_upcaft.lock: Resource temporarily unavailable - NOT SERVING
[Tue Apr 30 22:47:53.650 2024] [469783] prereading 1 tables
[Tue Apr 30 22:47:53.689 2024] [469780] accepting connections
[Tue Apr 30 22:47:53.873 2024] [469784] [BUDDY] started v1.0.18 '/usr/share/manticore/modules/manticore-buddy/bin/manticore-buddy --listen=http://10.164.0.36:9312 --threads=4' at http://127.0.0.1:46475
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] Loaded plugins:
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] core: empty-string, backup, emulate-elastic, insert, select, show, cli-table, plugin, test, insert-mva
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] local:
[Tue Apr 30 22:47:54.001 2024] [469784] [BUDDY] extra:
[Tue Apr 30 22:47:58.081 2024] [469783] preread 1 tables in 4.432 sec
At this point I can connect, but one of my indexes is "disabled at the JSON config" (I did not do that), and is not serving.
(note, i always shut it down with systemctl that has the --stopwait option in it
Here is manticore.json
{
"clusters": {
},
"indexes": {
"la8_items_all": {
"type": "rt",
"path": "la8_items_all"
},
"la8_items_upcaft": {
"type": "rt",
"path": "la8_items_upcaft"
}
}
}
It seems like manticore.json is created every time searchd starts. Im not sure if this is the file the log talks about when when it says "disabled at the JSON config", but maybe i need to do something with it, like directory permissions, or set something in the main manticore.conf file.
from manticoresearch.
Lot-Art
commented on June 30, 2024
Hi, is there an update you could give me by any chance?
Did you receive the data?
Is there something I can do about table disabled at the JSON config?
BUDDY failing?
At this point manticore is very unstable: it crashes and even the restart fails (table disabled, and buddy). I don't know if its mlock, preopen_tables...
from manticoresearch.
sanikolaev
commented on June 30, 2024
Hi @Lot-Art
Sorry for the delay. Unfortunately, we still don't have issue-2097 on our side.
You did:
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore /home/ME/minio-binaries/issue-2097
which is wrong. Looks like you copied it to just /home/ME/minio-binaries/issue-2097, not to our server. The correct command according to the docs should be:
sudo /home/ME/minio-binaries/mc config host add manticore http://s3.manticoresearch.com:9000 manticore manticore
sudo /home/ME/minio-binaries/mc cp -r /var/lib/manticore/ manticore/write-only/issue-2097
Is there something I can do about table disabled at the JSON config?
What's important in this case is the next line, e.g.:
[Sat Apr 27 15:49:51.252 2024] [46345] WARNING: table 'la8_items_all': prealloc: failed to lock /var/lib/manticore/la8_items_all/la8_items_all.lock: Resource temporarily unavailable - NOT SERVING
then you need to find out why the searchd can't lock la8_items_all.lock:
- dir doesn't exist?
- another process has already locked it?
- etc.
BUDDY failing?
Unlikely in this case.
from manticoresearch.
Lot-Art
commented on June 30, 2024
I did the data with minio again, maybe this time i got it right. Could you please check?
from manticoresearch.
sanikolaev
commented on June 30, 2024
Could you please check?
Still the same :(
from manticoresearch.
Lot-Art
commented on June 30, 2024
Ok, now I really think i did it right. Please check again (sorry for wasting your time with my fiddles).
With that it should be possible to reproduce the issue in my opening post (the OPTIMIZE fail).
from manticoresearch.
Related Issues (20)
- `range` filter produced random order results even with `order by` condition HOT 1
- `sort` is not respected by manticoresearch
- Improve RELOAD ... switchover=1
- Protected tables in replication of Galera
- Thai chars are in the wrong charsets HOT 9
- Treating document IDs as numbers in /search responses HOT 4
- Daemon crashes on _search request from Kibana HOT 1
- Fix json escaping for mysqldump HOT 2
- Make it possible to heal from dupes in a disk chunk HOT 3
- Search by JSON keys works incorrectly HOT 4
- Possible issue with Buddy on Windows HOT 6
- buddy can not work at Windows if listen is 127.0.0.1 HOT 5
- WARNING: wordlist size mismatch (size=18, checkpoints=0) HOT 3
- Crash if I use an SQL with a reserved word HOT 4
- Escape is not working for HTTP and SQL HOT 4
- IDF calculation issue
- Extended quote and double quote functionality
- Show correct data types in /cli_json HOT 2
- Escaping in wordforms and exceptions HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from manticoresearch.