Connecting through mysql protocol on mac about manticoresearch

[barryhunter]

Are all the old 'searchd' processes definitely gone?

I recently did a similar transition, and manticore cept crashing. Seems there was some sort of 'zombie' searchds still running (even though had searchd --stop). once I killall all searchd processes, it suddenly started working.

(this on ubuntu linux, but understand macs are basically linux under the hood)

[gerricom]

Yes, they are definitly gone. I even performed a reboot, just in case some zombies were still hanging around.

The process handling with mac seems to be a bit faulty, too. When running searchd --stop it claims, that it hat sent SIGTERM to the given pid, but nothing happens. I have to kill the process by myself.

[gerricom]

I did a downgrade to sphinx 2.2.11 again and everything is working as normal... mmhh

I gave it a shot to compile manticore from scratch, but I'm running into a bunch of cmake errors and I have not the time to clear things up at the moment. Hopefully I can test some days later...

[tomatolog]

tried to decode addresses from crash log you provided to get BT that could read and seems sym file does not matched to crash log.

You have to rebuild debug version of Manticore then restart debug version of daemon to get BT on crash or provide reproducible example - config \ index and query that recreates crash here locally.

[gerricom]

How exactly can a "rebuild" be done? I see there is a "dbg" package for Mac OS at the github releases page, but I don't know what to do with it. Sorry - when it comes to compiling etc. I'm not that experienced yet.

I can at the moment state that this (no mysql connection and effectless --stop) happens with every Mac I've got for testing so far. Two iMacs and and an Air with current OS 10.13.1. Even on clean systems without sphinx being installed ever.

[tomatolog]

just make build directory at Manticore source then

cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebugInfo -DDISABLE_TESTING=1
make -j4
make install

and provide output when build failed. I'll look at our build environment tomorrow to make sure I miss nothing however build process is described here http://manticoresearch.gitlab.io/dev/installation.html#compiling-manticore-from-source

[klirichek]

@gerricom give try to https://yadi.sk/d/38zGwQFB3QMkVt

comparing to sphinx we moved from 'poll' queue to macos's 'kqueue', so it by design works faster on many connections (on linux on the same place used 'epoll' where possibly). However that proxy code (kqueue specific) wasn't tested, and so, may have issues.
I've found and fixed the one which causes crash when querying distributed index (hope that is the one you stepped on).

--stop however seems to work ok. Literally all the things it does is to look the pidfile location in config, then read the file and send SIGTERM to the proc id written in file.

So, if you know the location, the same effect will be achieved by kill 'cat searchd.pid' (backticks here)

If it doesn't work by some reason, ensure that searchd's pid matches the number located in pidfile. Actually there are two processes, one (with lower num) is watchdog, second is actual daemon. The pidfile will contain last one.

[gerricom]

Sorry - this took a while. The flu got me these days. Ugh.

So back here I tested your new version, @klirichek and the first glance looks good: Manticore starts and the MySQL clients connects. But: I can't perform any queries. This is the full searchd.log from startup to failure:

[Fri Dec  8 10:08:01.388 2017] [795442] watchdog: main process 24024 forked ok
[Fri Dec  8 10:08:01.389 2017] [795443] listening on 127.0.0.1:9306
[Fri Dec  8 10:08:01.504 2017] [795488] prereading 1 indexes
[Fri Dec  8 10:08:01.504 2017] [795443] accepting connections
[Fri Dec  8 10:08:01.504 2017] [795488] prereaded 1 indexes in 0.000 sec
[Fri Dec  8 10:08:01.504 2017] [795489] WARNING: failed to set nodelay option: Operation not supported on socket
[Fri Dec  8 10:08:02.507 2017] [795443] WARNING: last message repeated 1 times
------- FATAL: CRASH DUMP -------
[Fri Dec  8 10:08:14.527 2017] [24024]

--- crashed invalid query ---

--- request dump end ---
Sphinx 2.5.2 ac0efee8@171206 id64-dev
Handling signal 11
-------------- backtrace begins here ---------------
Program compiled with 4.2.1
Configured with flags: Configured by CMake with these definitions: -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDISTR_BUILD=macos -DDL_EXPAT=1 -DEXPAT_LIB=libexpat.dylib -DUSE_LIBICONV=1 -DDL_MYSQL=1 -DMYSQL_LIB=libmysqlclient.dylib -DDL_PGSQL=1 -DPGSQL_LIB=libpq.dylib -DDATADIR=/var/data -DSPLIT_SYMBOLS=ON -DUSE_BISON=ON -DUSE_FLEX=ON -DUSE_SYSLOG=1 -DWITH_EXPAT=ON -DWITH_ICONV=ON -DWITH_MYSQL=ON -DWITH_PGSQL=ON -DWITH_RE2=ON -DWITH_STEMMER=ON -DWITH_ZLIB=ON
Host OS is Darwin Air-Aleksej.klirik.pw 17.2.0 Darwin Kernel Version 17.2.0: Fri Sep 29 18:27:05 PDT 2017
Stack bottom = 0x700006d37edf, thread stack size = 0x100000
Trying manual backtrace:
Something wrong with thread stack, manual backtrace may be incorrect (fp=0x700006d37760)
Stack looks OK, attempting backtrace.
0x10361ea43
0x700006d37df0
0x0
0x10368bef4
0x1037ea954
0x7fff6ee776c1
0x7fff6ee7756d
0x7fff6ee76c5d
0x0
Something wrong in frame pointers, manual backtrace failed (fp=0)
Trying system backtrace:
begin of manual symbols:
1037e72eb
10361ea43
7fff6ee6df5a
0
10368bef4
1037ea954
7fff6ee776c1
7fff6ee7756d
7fff6ee76c5d
-------------- backtrace ends here ---------------
Please, create a bug report in our bug tracker (https://github.com/manticoresoftware/manticore/issues)
and attach there:
a) searchd log, b) searchd binary, c) searchd symbols.
Look into the chapter 'Reporting bugs' in the documentation
(http://docs.manticoresearch.com/latest/html/reporting_bugs.html)
--- BT to source lines (depth 9): ---
conversion failed (error 'No such file or directory'):
  1. Run the command provided below over the crashed binary (for example, './searchd'):
  2. Attach the source.txt to the bug report.
addr2line -e ./searchd 0x37e72eb 0x361ea43 0x6ee6df5a 0x0 0x368bef4 0x37ea954 0x6ee776c1 0x6ee7756d 
0x6ee76c5d > source.txt
--- BT to source lines finished ---
--- 1 active threads ---
thd 0, proto sphinxql, state handshake, command SYSTEM
------- CRASH DUMP END -------
[Fri Dec  8 10:08:15.008 2017] [795442] watchdog: main process 24024 crashed via CRASH_EXIT (exit code 2), will be restarted
[Fri Dec  8 10:08:15.008 2017] [795442] watchdog: main process 24099 forked ok
[Fri Dec  8 10:08:15.009 2017] [796493] listening on 127.0.0.1:9306
[Fri Dec  8 10:08:15.112 2017] [796493] binlog: replaying log /Users/gerricom/Development/eopac/sphinx/binlog.001
[Fri Dec  8 10:08:15.112 2017] [796493] binlog: replay stats: 0 rows in 0 commits; 0 updates, 0 reconfigure; 0 indexes
[Fri Dec  8 10:08:15.112 2017] [796493] binlog: finished replaying /Users/gerricom/Development/eopac/sphinx/binlog.001; 0.0 MB in 0.000 sec
[Fri Dec  8 10:08:15.112 2017] [796493] binlog: finished replaying total 1 in 0.000 sec
[Fri Dec  8 10:08:15.113 2017] [796508] prereading 1 indexes
[Fri Dec  8 10:08:15.113 2017] [796509] WARNING: failed to set nodelay option: Operation not supported on socket
[Fri Dec  8 10:08:15.113 2017] [796508] prereaded 1 indexes in 0.000 sec
[Fri Dec  8 10:08:15.113 2017] [796493] accepting connections
[Fri Dec  8 10:08:15.113 2017] [796509] WARNING: failed to set nodelay option: Operation not supported on socket

This is, what MySQL outputs:

$ sphinxql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 2.5.2 ac0efee8@171206 id64-dev 

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SELECT * FROM realtime WHERE MATCH('test');
ERROR 2013 (HY000): Lost connection to MySQL server during query
mysql> 

Stopping searchd like this looks like:

$ ./searchd --config ~/Development/eopac/sphinx/sphinx.conf --stop
Manticore 2.5.2 ac0efee8@171206 id64-dev
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017, Manticore Software LTD (http://manticoresearch.com)

using config file '/Users/gerricom/Development/eopac/sphinx/sphinx.conf'...
stop: successfully sent SIGTERM to pid 24099

But:

$ ps aux | egrep '[s]earchd'
gerricom    24099   0,0  0,8  4450092 142540   ??  S    10:08am   0:01.54 ./searchd --config /Users/gerricom/Development/eopac/sphinx/sphinx.conf
gerricom    24023   0,0  0,0  4282376    500   ??  S    10:08am   0:00.00 ./searchd --config /Users/gerricom/Development/eopac/sphinx/sphinx.conf

It is still running after --stop. Only force kill stops searchd.

[airolg]

Added to backlog, thank you

[klirichek]

Networking was refactored in 3359bcd
So, I close this issue as labeled about querying.
However shutdown issue wasn't touched; it is related and moved to #45, details will be there.