Comments (13)
was just poking around a little dockerpty may be relevant.
from girder_worker.
from girder_worker.
👍
Looks like this communicates directly with the service over HTTP rather than using a subprocess, which is cool. We'd have to investigate how well its API supports streaming IO.
from girder_worker.
Yes, talking directly to the service seems like a nice step up from using subprocess. When you say "streaming IO" are you taking about the support we currently have for pipes?
from girder_worker.
Yeah, the ability to do non-blocking reads from pipes (a la select
).
from girder_worker.
Recording quote from @zachmullen "we should see about using more modern tooling for docker garbage collection in lieu of the current docker-gc script, which is a third party tool that was the industry standard for a while." Was thinking the same ...
from girder_worker.
Some possible alternative to docker-gc script:
https://github.com/Yelp/docker-custodian (uses docker-py )
from girder_worker.
Docker-py has just recently release a version 2.0. Prior to version 2.0, it is docker-py
on pip, now it is is just docker
. There are substantial breaking changes between the two. We use version 1.9 in the slicer_cli_web and HistomicsTK projects, so if we use version 2, we may want to update those as well.
It almost seems like less work to use the command line interface than docker-py, especially for volumes. Version 2 is a little cleaner than version 1.9. The transition wouldn't be hard.
And, our GC task should run out-of-band; it seems wasteful to delay sending job results until the GC has run, especially when that GC can take a long time.
from girder_worker.
Docker-py has just recently release a version 2.0. Prior to version 2.0, it is docker-py on pip, now it is is just docker. There are substantial breaking changes between the two. We use version 1.9 in the slicer_cli_web and HistomicsTK projects, so if we use version 2, we may want to update those as well.
It almost seems like less work to use the command line interface than docker-py, especially for volumes. Version 2 is a little cleaner than version 1.9. The transition wouldn't be hard.
@manthey Are you saying that the move docker-py isn't worth it despite being able to talk directly to the docker daemon rather than through a sub process?
And, our GC task should run out-of-band; it seems wasteful to delay sending job results until the GC has run, especially when that GC can take a long time.
I agree a celery periodic task could be used for this
from girder_worker.
@cjh1 I don't think switching to docker-py will reduce code complexity. Version 2 won't increase it much, but version 1.x will reduce readability. If it gains us robustness or maintainability, then switch can still be worth while.
from girder_worker.
It would reduce the number of processes used to run a docker task by one, which might help with scalability.
from girder_worker.
As part of this, we should try to mitigate the busy loop behavior while we wait on the docker container to finish.
from girder_worker.
Fixed by #96
from girder_worker.
Related Issues (20)
- RFC: Consider cache configuration in new girder worker tasks HOT 2
- Girder worker integration test plugin is not compatible with girder 3.
- Create "How to write an integration test" documentation for girder worker
- spaces in girder folder name are incorrectly converted to container args. HOT 5
- Check if Celery Pinning is still necessary HOT 24
- Docker in docker permission issue HOT 1
- Girder Client 2.4.0 breaking integration test HOT 7
- girder_plugin settings for worker.api_url, worker.broker, worker.backend are not being picked up correctly HOT 3
- Using kwargs in chained tasks fails to create Job object
- Revoke integration tests are hanging HOT 1
- girder_result_hooks cleanup methods are not called after task is done
- File paths from docker transforms should be wrapped in quotes HOT 1
- Document job event binding and output reference kwargs. HOT 7
- Running NVidia dockers
- docker_run can't process non-string arguments HOT 3
- Exceptions are silently ignored during plugin load
- Girder plugin: use setting default HOT 1
- Pin celery version in requirements.in HOT 1
- How can I pass env variables from within DSA to a image that is being added through slicer web task
- Is the "DIRECT_PATH" checkbox actually used ? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from girder_worker.