Comments (7)
@rguilmont @YaYaB gzip/deflate encryption is handled by https://github.com/oatpp/oatpp-zlib from within https://github.com/oatpp/oatpp. The components are simply added here: https://github.com/jolibrain/deepdetect/blob/master/src/http/app_component.hpp#L114
Running valgrind on dede
with gzip queries only shows the possible leak below. This looks like an init from libz directly, from the oatpp send
function.
@lganzzzo Hi, the ::send
function seems to leak from deflateInit
, have you seen this before, or are we doing something wrong ? Thanks.
Libz init memory reported by valgrind:
==3020638== 536,192 (11,904 direct, 524,288 indirect) bytes in 2 blocks are definitely lost in loss record 4,799 of 4,801
==3020638== at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==3020638== by 0x5AA3418: deflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.11)
==3020638== by 0x5AA3651: deflateInit_ (in /lib/x86_64-linux-gnu/libz.so.1.2.11)
==3020638== by 0x71C063: oatpp::zlib::DeflateEncoder::DeflateEncoder(long, bool, int) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638== by 0x71B2E8: oatpp::zlib::DeflateEncoderProvider::getProcessor() (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638== by 0x6EA02D: oatpp::web::protocol::http::outgoing::Response::send(oatpp::data::stream::OutputStream*, oatpp::data::stream::BufferOutputStream*, oatpp::web::protocol::http::encoding::EncoderProvider*) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638== by 0x6F6DB6: oatpp::web::server::HttpProcessor::processNextRequest(oatpp::web::server::HttpProcessor::ProcessingResources&) (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638== by 0x6FB28F: oatpp::web::server::HttpProcessor::Task::run() (in /home/beniz/projects/deepdetect/dev/deepdetect/build/main/dede)
==3020638== by 0x945BDE3: ??? (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==3020638== by 0x936B608: start_thread (pthread_create.c:477)
==3020638== by 0x9837292: clone (clone.S:95)
from deepdetect.
More info on that : we managed to reproduce this issues with the above Python scripts, but not with curl.
So after testing more with @YaYaB, we had a strong intuition that it has something to do with the HTTP serving.
After analysing HTTP headers, we found that Python requests by default asks for GZIP encoded answer ( Accept-Encoding: gzip, deflate
) while curl doesn't.
So we manually set this header in curl, and finally reproduced the issue with curl too.
We also tested to send gzip-compressed queries, asking for uncompressed responses, and no memory leak was noticed. So really looks like it's something related to GZIP compression.
from deepdetect.
Actualy it is even not related to tensorrt but even with classical caffe predictions with or without gpu
from deepdetect.
Hey @beniz ,
Your code looks good. Most probably it's on oatpp side.
I'll take a closer look.
from deepdetect.
Hi @lganzzzo how are things ? Do you have any fresh lead on this by any chance ? I've seen issues with libz a long time ago, this could still be outside oatpp.
from deepdetect.
Hey @beniz ,
Yes, at this point it looks like a libz
issue.
I'm filing an issue in oatpp to investigate possible fixes.
It might take a while
from deepdetect.
Thanks a lot guys.
FYI we've mitigated this gzip issue by setting an Envoy proxy in front of deepdetect, taking care of compression and decompression of requests.
from deepdetect.
Related Issues (20)
- Inconsistent predictons using refinedet model HOT 12
- Memory leak on constant /predict requests HOT 8
- Refinedet Tensorrt prediction fails HOT 7
- Different prediction with tensorrt on refinedet model for the version v0.18.0 HOT 3
- getting error while training, .solverstate HOT 23
- Chain predictions swapped between images HOT 2
- Simsearch query segfault when using IVF indexes, but not default/flat index HOT 6
- On object detect training call, missing either test or train list causes a segfault
- dd_client not find in this path anyone help HOT 2
- How do I do a face recognition using this? HOT 2
- DeepDetect full rewrite in Pure Java
- 'OCR' object has no attribute 'histogram_equalization' HOT 13
- "best: -1" in predict behaves differently in torch models HOT 2
- Torch v1.12 requires libcupti* but nvidia/cuda:11.6.0-cudnn8-runtime-ubuntu20.04 doesn't include it
- Race condition / pthread error when predicting
- I have error build xgboost HOT 1
- Using `true` or `false` instead of `1` or `0` for query params for status or labels returns a internal server error HOT 1
- Question about hosting the docker image HOT 4
- Graphics problem with tsne algorithm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepdetect.