Comments (23)
Thanks for the effort to simplify debugging!
I've checked that the Python driver crash with that file. The server also shouldn't hang because the driver crash, so we've probably two issues here, but lets leave things on this issue for not splitting the information too much. I'll continue investigating.
from bblfshd.
Good news! I find a good reproducible example for you.
Bug related to code in files I am trying to parse.
here is the code to reproduce:
https://gist.github.com/zurk/07ca43333c02ea15cee53a361f0c6394
and python_files.txt
file:
python_files.txt.zip
It contains the path to data on science-3
. I can make a dump with this files for you if you want, just ask.
from bblfshd.
Update:
I found one file that can hang the server or driver, I am not sure.
Run:
from bblfsh import BblfshClient
BblfshClient("0.0.0.0:9437").parse("bad_file.py", language="Python", timeout=120)
No responce, exit by timeout and nothing in bblfsh logs.
P.S.: End of the file is strange but correct. Also, I think it is not only one example of "hanging" file in that collection. Does bblfsh server provide a timeout for drivers?
from bblfshd.
Yes, I could not work on this yesterday with all the pending releases, I'm working on it currently.
from bblfshd.
To be sure run everything again.
You can find hang bblfsh_test
docker as is. And bblfsh_hang_client
docker with WARNING:source_transformer:Failed to construct model for /storage/konstantin/data/repos/matplotlib/tinytai@caffe-segnet: itemsize cannot be zero in type
May be it is helpful. No need to wait :)
also last logs from server:
time="2017-09-13T17:42:14Z" level=info msg="parsing show_events.py (775 bytes)"
time="2017-09-13T17:42:14Z" level=info msg="parsing setup.py (746 bytes)"
time="2017-09-13T17:42:14Z" level=info msg="container started bblfsh/python-driver:latest (01BSY5B2NRZBWDSSPHGGK4M5DA)"
time="2017-09-13T17:42:14Z" level=info msg="container started bblfsh/python-driver:latest (01BSY5B2W1TDE7D4Y0MQXGX295)"
time="2017-09-13T17:42:14Z" level=info msg="container started bblfsh/python-driver:latest (01BSY5B30TRA608NT4EE45DMNV)"
from bblfshd.
Looks like a problem serializing complex numbers. This changed in python-driver 0.8.1.
from bblfshd.
I think it can be not only one problem. I add logs from bblfsh to my previous message. Here is no error from the container. They are just started.
So, I will try to reproduce it after python-driver the fix.
from bblfshd.
If you see the last error in the server log and scroll to the right you can see:
class__.__name__)\\nTypeError: Object of type \\'complex\\' is not JSON serializable
from bblfshd.
(the server also probably should not hang on this or any other parsing errors, but let's fix this one first).
from bblfshd.
Yes, @juanjux, I saw the error. But check the log output for the second try in this comment #101 (comment) there is no error
from bblfshd.
Could you try again with the server in verbose mode?
docker run --rm --privileged -d -p 9434:9432 --name bblfsh_test bblfsh/server:v0.7.0 --log-level debug
from bblfshd.
Sure, after last logs with code.
time="2017-09-14T12:34:06Z" level=info msg="container started bblfsh/python-driver:latest (01BT063K0FP8ZMZ9TFQ87TEFNP)"
time="2017-09-14T12:34:06Z" level=debug msg="creating container for bblfsh/python-driver:latest"
time="2017-09-14T12:34:06Z" level=debug msg="starting up container bblfsh/python-driver:latest (01BT063K5Q52VNRRWQP644VMMV)"
time="2017-09-14T12:34:06Z" level=info msg="container started bblfsh/python-driver:latest (01BT063K5Q52VNRRWQP644VMMV)"
time="2017-09-14T12:34:06Z" level=debug msg="creating container for bblfsh/python-driver:latest"
time="2017-09-14T12:34:06Z" level=debug msg="starting up container bblfsh/python-driver:latest (01BT063KA4FH0RJPCWCRK6RMVS)"
time="2017-09-14T12:34:06Z" level=info msg="container started bblfsh/python-driver:latest (01BT063KA4FH0RJPCWCRK6RMVS)"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:08Z" level=debug msg="driver exited without error"
time="2017-09-14T12:34:13Z" level=debug msg="Empty code received, returning empty UAST"
time="2017-09-14T12:34:13Z" level=debug msg="Empty code received, returning empty UAST"
time="2017-09-14T12:34:13Z" level=debug msg="Empty code received, returning empty UAST"
time="2017-09-14T12:34:18Z" level=debug msg="Empty code received, returning empty UAST"
And then hangs.
from bblfshd.
Wow, no, It does not actually hang. It continues to produce
"Empty code received, returning empty UAST"
from bblfshd.
The error is probably unrelated, but I released v0.8.2 of the Python driver with the complex numbers serialization fix, could you try?
The empty code error is produced when the server receives and a request with an empty code content, trough it shouldn't loop if there aren't new requests.
from bblfshd.
Thank you @juanjux!
As soon as science-3
will be turned on.
with an empty code content
It was not empty because something happened and then bblfsh gives only empty UAST.
And as I can see in the last logs
time="2017-09-14T12:34:07Z" level=debug msg="driver exited without error"
29 drivers exited simultaneously and no new driver container started. So It is something strange.
from bblfshd.
Yes, I check it one time more.
I do not have a problem with serialization but still from some moment I have this messages:
time="2017-09-14T17:51:35Z" level=debug msg="Empty code received, returning empty UAST"
I attach tail of debug log just in case.
logs.zip
I think it can be somehow related to file content.
from bblfshd.
Just want to mention that problem still alive with server v1.0.0 and latest driver.
from bblfshd.
I just did a PR that fixes the problem with that bad_file.py
, so the python driver doesn't crash anymore on it (or others with the same problem).
Next step for me is to do a micro test case with the same problem (just print 123L
) and, with the unfixed version, check why the server would hang in that case.
from bblfshd.
Python driver 1.0.1 (already tagged as latest so it should be automatically downloaded by the server) have a fix and also links vs a version of the SDK with another fix. With these together I can't reproduce this anymore so I don't think we need to release a new version of the server linking against the new SDK version (but I'm not 100% sure because the fixed part is also used by the server).
@zurk could you please test again and report here if you still find the problem?
from bblfshd.
@juanjux , Great news!
I test it for my example with 6 000 files #101 (comment).
I run it several times, and everything is fine! Hope, that it is true for the rest dataset. In
However, I wonder about sever part, could something like that happen with some other "bad" drivers?
from bblfshd.
There were really several problems. bblfsh/python-pydetector/pull/24 caused a crash in the Python driver when a numeric long literal like 0L was in the exported native AST dictionry. bblfsh/python-driver/pull/92 was logging on the stderr of the Python driver when any crash happened (drivers should not write to stderr, only communicate with the messages, the sdk merges the drivers stderr and stdout into a single stream) and finally bblfsh/sdk/pull/175 would cause incomplete reads if there was something written on stderr that shouldn't be.
Other drivers would be "bad" by writing random stuff in stdout/stderr outside of the expected JSON communication but currently it would be easier to spot. I tough of adding a timeout on the SDK/server read but that could mask real problems so I'll maybe let that for a future, more solid, version of everything.
Thanks for you excellent report!
from bblfshd.
@juanjux please fix the links to the issues in the previous message, they're linking to server issues but they are actually issues in other repos.
from bblfshd.
Fixed.
from bblfshd.
Related Issues (20)
- install --all has the same effect as --recommended HOT 1
- bblfshd and go-client interdependency HOT 1
- Design/Prototype scalability and deployment support
- Design/Prototype local scalable environment
- Prototype native parser scaling mechanism HOT 3
- Test scalability layers
- Drivers discovery mechanism HOT 2
- make test fails HOT 2
- cron job fails in Travis
- Is this still active? HOT 1
- Archive project(s)
- Ongoing maintenance of the Babelfish project HOT 6
- Driver instance not killed after deadline
- Cannot get latest version: module contains a go.mod file, so module path should be github.com/bblfsh/bblfshd/v2 HOT 5
- Security Issue: Request for contact
- bblfshd not run HOT 1
- https://doc.bblf.sh/ down?
- Babelfish run test error
- Is bblfshd dead? HOT 3
- The vulnerability CVE-2021-32825 has been fixed, but no specific tag denotes the patched version.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bblfshd.