Plugin framework for Gaia about gaia

[jbeezley]

I would avoid this kind of a plugin structure. This is how girder's plugins work, and it causes a number of problems with deploying and maintenance. A better approach is to use setuptools' "entry_points". Pylons provides a basic example of how it works here. Doing it this way eliminates the need for manual management of dependencies and makes deployment easier (just pip install).

[kotfic]

I agree with @jbeezley, directory based plugins quickly run into scaleability issues. A very compact example of how to use entry-points exists in this PR girder/girder_worker#29 As you can see here we dynamically add celery tasks from other packages by creating a custom 'girder_worker.tasks' entry point. Any installed module that defines this entry_point can add tasks to the celery worker.

[mbertrand]

Thanks @jbeezley and @kotfic I will use the entry_points approach.

[aashish24]

I agree with both @jbeezley @kotfic

[mbertrand]

@jbeezley and @kotfic when you have a chance could you take a look at:

gaia_plugin_demo
gaia.parser.load_plugins

and let me know if that's generally what you had in mind.

gaia_plugin_demo contains new IO and Processor classes in several modules, with this in its setup.py file:

  entry_points={
    'gaia.plugins': [
            "fake_plugin = my_gaia_plugins.fake_plugin",
            "another_fake_plugin = my_gaia_plugins.another_fake_plugin"
        ]
}

In the gaia plugins branch, there is a load_plugins method which is in turn called in the deserialize method:

# Import plugin classes
def load_plugins():
    plugin_modules = []
    for ep in pr.iter_entry_points(group='gaia.plugins'):
        module = ep.load()
        plugin_modules.append(importlib.import_module(module.__name__))
        logger.debug("Loaded plugin {}: {}".format(ep.name, module))

    for plugin in plugin_modules:
        valid_classes.extend([x[0] for x in inspect.getmembers(
plugin, inspect.isclass)])

So when sending a json file like this to Gaia's parser:

{
  "_type": "my_gaia_plugins.fake_plugin.FakeProcess",
  "inputs": [
    {
      "name": "input",
      "_type": "my_gaia_plugins.fake_plugin.FakeIO",
      "uri": "/tmp/bogus.geojson"
    },
    {
      "name": "input",
      "_type": "my_gaia_plugins.another_fake_plugin.AnotherFakeIO",
      "uri": "/tmp/bogus.tif"
    }
  ],
  "var_col": "num_hospitals"
}

The output is:

Created FakeIO
Created AnotherFakeIO
Created FakeProcess
Compute something with FakeProcess
Process complete.

[jbeezley]

Yup, that looks about right. You'll eventually want to publish these plugins on pypi, so it would be good to establish a naming convention so they are easy to find (something like gaia-*).

[aashish24]

@mbertrand should we close this?