Comments (7)
I would avoid this kind of a plugin structure. This is how girder's plugins work, and it causes a number of problems with deploying and maintenance. A better approach is to use setuptools' "entry_points". Pylons provides a basic example of how it works here. Doing it this way eliminates the need for manual management of dependencies and makes deployment easier (just pip install).
from gaia.
I agree with @jbeezley, directory based plugins quickly run into scaleability issues. A very compact example of how to use entry-points exists in this PR girder/girder_worker#29 As you can see here we dynamically add celery tasks from other packages by creating a custom 'girder_worker.tasks' entry point. Any installed module that defines this entry_point can add tasks to the celery worker.
from gaia.
Thanks @jbeezley and @kotfic I will use the entry_points approach.
from gaia.
I agree with both @jbeezley @kotfic
from gaia.
@jbeezley and @kotfic when you have a chance could you take a look at:
gaia_plugin_demo
gaia.parser.load_plugins
and let me know if that's generally what you had in mind.
gaia_plugin_demo contains new IO and Processor classes in several modules, with this in its setup.py file:
entry_points={
'gaia.plugins': [
"fake_plugin = my_gaia_plugins.fake_plugin",
"another_fake_plugin = my_gaia_plugins.another_fake_plugin"
]
}
In the gaia plugins branch, there is a load_plugins method which is in turn called in the deserialize method:
# Import plugin classes
def load_plugins():
plugin_modules = []
for ep in pr.iter_entry_points(group='gaia.plugins'):
module = ep.load()
plugin_modules.append(importlib.import_module(module.__name__))
logger.debug("Loaded plugin {}: {}".format(ep.name, module))
for plugin in plugin_modules:
valid_classes.extend([x[0] for x in inspect.getmembers(
plugin, inspect.isclass)])
So when sending a json file like this to Gaia's parser:
{
"_type": "my_gaia_plugins.fake_plugin.FakeProcess",
"inputs": [
{
"name": "input",
"_type": "my_gaia_plugins.fake_plugin.FakeIO",
"uri": "/tmp/bogus.geojson"
},
{
"name": "input",
"_type": "my_gaia_plugins.another_fake_plugin.AnotherFakeIO",
"uri": "/tmp/bogus.tif"
}
],
"var_col": "num_hospitals"
}
The output is:
Created FakeIO
Created AnotherFakeIO
Created FakeProcess
Compute something with FakeProcess
Process complete.
from gaia.
Yup, that looks about right. You'll eventually want to publish these plugins on pypi, so it would be good to establish a naming convention so they are easy to find (something like gaia-*
).
from gaia.
@mbertrand should we close this?
from gaia.
Related Issues (20)
- dependency Fiona
- Different scenario causes cropping error HOT 2
- anaconda / conda-forge installation HOT 1
- Geoserver/Geonode endpoints via Girder plugin HOT 4
- Define API to convert datasets from one format to another format HOT 6
- Create short movies for Annual Review demogaia
- Gaia
- Title template
- Fix failing postgres tests on travis ci
- Use gippy for raster image processing HOT 4
- Gaia integration with Minerva/Girder HOT 12
- Geotrellis integration with Gaia HOT 4
- Define / Discuss ways and roadmap to improve documentation HOT 3
- Conda Package HOT 3
- Use circle ci as improves the testing and debugging
- Dead kernels reading certain raster files HOT 1
- Add coverage reporting to the README
- Review and test Raster clustering PR HOT 1
- Update and review the README.md
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gaia.