FATAL: CRASH DUMP - searchd about manticoresearch

[tomatolog]

What package did you use to install the binary?

[CrowdSparkDave]

manticore_2.4.1-171017-3b31a97-release-stemmer.xenial_amd64-bin.deb

Linux 3.13.0-48-generic #80-Ubuntu SMP Thu Mar 12 11:16:15 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[tomatolog]

according to crash log you provided crashes happened on searching realtime_media_941 index.
Could you check that index is valid with following command

./indextool -c your.conf --check realtime_media_941

In case check passed well could you provide that index to replay queries cause crashes and investigate crashes here locally?

[CrowdSparkDave]

root@ec2-sphinxsearch1:~# mysql -u root -h127.0.0.1 -P9306
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
+---------------------+-------+
| Index | Type |
+---------------------+-------+
| realtime_groups | rt |
| realtime_media | rt |
| realtime_media_1147 | rt |
| realtime_media_123 | rt |
| realtime_media_43 | rt |
| realtime_media_629 | rt |
| realtime_media_721 | rt |
| realtime_media_747 | rt |
| realtime_media_877 | rt |
| realtime_media_927 | rt |
| realtime_media_941 | rt |
| realtime_media_987 | rt |
| userdevices | local |
+---------------------+-------+

root@ec2-sphinxsearch1:~# indextool -c /etc/sphinxsearch/sphinx.conf --check realtime_media_941
Manticore 2.5.1 b751d2b@171123 id64-release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)

using config file '/etc/sphinxsearch/sphinx.conf'...
FATAL: index 'realtime_media_941': prealloc failed: failed to lock /var/lib/sphinxsearch/data/realtimemedia_941.lock: Resource temporarily unavailable

[tomatolog]

FATAL: index 'realtime_media_941': prealloc failed: failed to lock

you have to stop daemon or flush RT index then copy ram part along with disk chunks then check RT index at this new location as you can not use RT index along with daemon manage same RT index

[CrowdSparkDave]

root@ec2-sphinxsearch1:~# indextool -c /etc/sphinxsearch/sphinx.conf --check realtime_media_941
Manticore 2.5.1 b751d2b@171123 id64-release
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)
using config file '/etc/sphinxsearch/sphinx.conf'...
checking index 'realtime_media_941'...
checking RT segment 0(26)...
checking RT segment 1(26)...
checking RT segment 2(26)...
checking RT segment 3(26)...
checking RT segment 4(26)...
checking RT segment 5(26)...
checking RT segment 6(26)...
checking RT segment 7(26)...
checking RT segment 8(26)...
checking RT segment 9(26)...
checking RT segment 10(26)...
checking RT segment 11(26)...
checking RT segment 12(26)...
checking RT segment 13(26)...
checking RT segment 14(26)...
checking RT segment 15(26)...
checking RT segment 16(26)...
checking RT segment 17(26)...
checking RT segment 18(26)...
checking RT segment 19(26)...
checking RT segment 20(26)...
checking RT segment 21(26)...
checking RT segment 22(26)...
checking RT segment 23(26)...
checking RT segment 24(26)...
checking RT segment 25(26)...
checking disk chunk 0(4)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 9.5 sec elapsed
checking disk chunk 1(4)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 0.9 sec elapsed
checking disk chunk 2(4)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 1.1 sec elapsed
checking disk chunk 3(4)...
checking rows...
checking dictionary...
checking data...
checking rows...
checking attribute blocks index...
checking kill-list...
check passed, 1.2 sec elapsed
check passed, 13.2 sec elapsed

[tomatolog]

you might upload your data to our write only FTP at 88.99.251.227

user: manticorebugs
pass: shithappens

I will try to reproduce crash here locally by replaying query from crash log your provided and investigate this crash. I also need your config along with index files

[CrowdSparkDave]

done.
Thanks!

[klirichek]

Hm, failed to lock /var/lib/sphinxsearch/data/realtimemedia_941.lock
Check if the daemon could create this lock file in this folder (with it's runtime credentials).
If it is impossible, try to eliminate the reason (say, if such file was created with 0600 mask by another UID and left there - so the daemon will not be able to lock via that file)

[airolg]

Added to backlog, thank you

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Feel free to re-open the issue in case it becomes actual.

[manticoresearch]

Closing as non-actual. Feel free to reopen if you can provide more details to reproduce the issue.