modelhub-ai / modelhub Goto Github PK
View Code? Open in Web Editor NEWA collection of deep learning models with a unified API.
Home Page: http://modelhub.ai/
License: MIT License
A collection of deep learning models with a unified API.
Home Page: http://modelhub.ai/
License: MIT License
We need a way to have a blank IP on the front end - and fetch the config from github directly since there will be no cloud instance running.
something like -uc to update contrib src and -ud to update docker (both optional)
With Docker version 19.03 the mechanism to enable GPU acceleration in Docker containers changed, it is no longer necessary to install the full Nvidia-Docker 2 env as there is now native support to pass a GPU argument to docker run
.
See here for the updated usage of Nvidia-Docker: https://github.com/NVIDIA/nvidia-docker#quickstart
Users still need to install the nvidia-docker-toolkit, but that's it. The issue is that this breaks our current start script as the commands to run containers are different:
Old: docker run --runtime=nvidia ...
New: docker run --gpus all ...
Should we still stick to the old syntax or require users to simply update to 19.03 and use the most recent version? Images built with either version are still compatible, so this is only about changing the start script and the requirements.
To clearly define how the outputs are supposed to be formatted, depending on the type of output. will also make integration_test much easier, since we just have to check the output against the schema.
Update: Force a fresh download of the model
List: Discover and list all available models in the repository
Since weight files can be large, the start scripts should be able to download weights from a different source than github
This is a future TODO/TO THINK ABOUT.
Implement a Slicer module that provides a generic interface to use modelhub models via the REST API. We could even integrate the model discovery (from the index) into the Slicer module such that you can search from within Slicer for available models, and if you start it automatically downloads and runs them.
I think this would open modelhub to a large community and create additional impact.
Any in-depth documentation or step-by step tutorials to help set the docker would be wonderful!
It is too cumbersome and error prone to list all files manually in the start script.
Only files in special locations (like large model files which don't fit into the repo) should we listed manually
Currently hardcoded to go through png, then load dicom.
Choose port on which to run API (or expect API in manual mode)
I think it would be quite helpful to have a quick gif with a demo of how modelhub works (not internals, just the summary of the web site features), so that it can be featured in the readme, and included in presentation slides. Would be quite handy for me at the moment!
This project might be an interesting direction to explore to support rendering of the JSONSchema-defined template and UISchema for the representation for the model configuration files, so that input types, attributes can be typed and validated at the time of submission: https://github.com/mozilla-services/react-jsonschema-form#the-uischema-object
When a notebook server is already running, (default) port 8888 is taken, and it appears that the server started within the docker is not available, although there are no errors reported on the console.
The SG running our instances needs updating - needs to point to specific IP's.
I dont want to move them myself because this will mess up your start scripts?
GitHub limits anonymous calls to their API to 60 per hour from the same IP (see here). Our start script requires several calls to the GitHub API when downloading a model (essentially one call per subdirectory in the model directory + 1 call to get the init.json first). This means at least 7 API calls for one model download => a user cannot download more than about 8 models per hour. This will probably not happen too often, but would still be nice to avoid.
One attempt to solve this could be to go through the GitHub Git Trees API and retrieve the whole directory tree. Unfortunately the returned results are not as informative as the current solution and would require a bunch of string fiddling to get the download URLs. It would also still require 3 API calls (as far as I understand the API for now), because we'd have to go through "master root tree" -> "models sha tree" -> "actual model sha tree" to get the sha for the actual model dir tree we're looking for (we cannot just get the root tree, because results are limited to 1000 entries, which we will probably exceed at some point).
Another solution would be for users to provide their GitHub credentials and run authenticated API calls, which are limited to 5000/hour. But I don't think we should expect that all users have a GitHub account.
Note that starting models which were downloaded before and already exist locally won't require any GitHub API calls (but updating them of course will).
Due to the GPU mode check with the online model index, local models (e.g. still in development) are not able to start due to the failing check for the GPU flag with the following error:
(base) christophs-mbp:modelhub christoph$ python start.py brats-modelhub -e Model folder exists already. Skipping download. ERROR: Model startup failed. ERROR DETAIL: 'Model "brats-modelhub" not found in online model index'
I'm mainly thinking about the url and ports. In the long run these should be assigned dynamically by some kind of service, such that we don't have to hardcode them in the index (with a larger number of model this will become infeasible). The index should really just be a list of models which are available via modelhub with the essential information of how/where to find them.
In any case, I expect that the way we store and organize the index will anyway evolve the more models we get.
also change docker image name to match others
When models are submitted, submitters should understand under what license the models will be distributed. That same license should probably be included in this repository and apply to all models stored. It is good to have this issue sorted out early in the process. The submission form says by default it is MIT, and individual submissions can have different licenses as needed.
This would replace the current start script and should have the following sub packages (runnable commands essentially):
Currently these are very long script files. Refactoring steps:
Try a better separation of functionality makes sense, possibly splitting up into classes. However, still keep everything in one file each, such that the user does not have to download several files for starting/testing.
Turn the whole thing into an installable python package. Then we can split functionality over several files to make the architecture even nicer. Usage of the package would then be like:
modelhub start
modelhub list
modelhub test_integration
Download the schema from github instead of expecting it to exist locally. Like this we also make sure that always the newest schema is used in the test.
also visualize these on front end
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.