Application plugins not being loaded after entry point refactor about girder_worker

[zachmullen]

These were previously loaded in __init__.py at module import time.

[kotfic]

@danlamanna @manthey further investigation of the issue seems to suggest that a fresh checkout is loading the application plugins correctly inside the celery worker. Plugins are loaded here and inside the worker they appear to be importable:

(gwtest) kotfic@minastirith:~/tmp/girder_worker ⇒ girder-worker -l info
pdb is running on 127.0.0.1:4444

(gwtest) kotfic@minastirith:~/tmp/girder_worker ⇒ telnet 127.0.0.1 4444
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
> /home/kotfic/tmp/girder_worker/girder_worker/core/__init__.py(61)<module>()
-> def _resolve_scripts(task):
(Pdb) l
 56  	_paths = [p for p in _paths if p.strip()]
 57  	_paths.append(os.path.join(PACKAGE_DIR, 'plugins'))
 58  	utils.load_plugins(_plugins, _paths, quiet=True)
 59  	import rpdb; rpdb.set_trace()
 60  	
 61  ->	def _resolve_scripts(task):
 62  	    if task.get('mode') != 'workflow':
 63  	        if 'script_uri' in task and 'script' not in task:
 64  	            task['script'] = io.fetch({
 65  	                'url': task['script_uri']
 66  	            })
(Pdb) import girder_worker.plugins.girder_io
(Pdb) print girder_worker.plugins.girder_io
<module 'girder_worker.plugins.girder_io' from '/home/kotfic/tmp/girder_worker/girder_worker/plugins/girder_io/__init__.py'>
(Pdb) from girder_worker.plugins.girder_io import load
(Pdb) print load
<function load at 0x7fa58e0e11b8>
(Pdb) 

girder_worker/core/__init__.py is included when girder_worker/tasks.py imports core:

 
(Pdb) w
  /home/kotfic/.venvs/gwtest/bin/girder-worker(11)<module>()
-> load_entry_point('girder-worker', 'console_scripts', 'girder-worker')()
  /home/kotfic/tmp/girder_worker/girder_worker/__main__.py(56)main()
-> app.worker_main()
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/app/base.py(206)worker_main()
-> app=self).execute_from_commandline(argv)
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/bin/base.py(311)execute_from_commandline()
-> return self.handle_argv(self.prog_name, argv[1:])
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/bin/base.py(377)handle_argv()
-> return self(*args, **options)
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/bin/base.py(274)__call__()
-> ret = self.run(*args, **kwargs)
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/bin/worker.py(212)run()
-> state_db=self.node_format(state_db, hostname), **kwargs
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/worker/__init__.py(95)__init__()
-> self.app.loader.init_worker()
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/loaders/base.py(128)init_worker()
-> self.import_default_modules()
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/loaders/base.py(121)import_default_modules()
-> tuple(maybe_list(self.app.conf.CELERY_INCLUDE))
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/loaders/base.py(103)import_task_module()
-> return self.import_from_cwd(module)
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/loaders/base.py(112)import_from_cwd()
-> package=package,
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/utils/imports.py(101)import_from_cwd()
-> return imp(module, package=package)
  /home/kotfic/.venvs/gwtest/lib/python2.7/site-packages/celery/loaders/base.py(106)import_module()
-> return importlib.import_module(module, package=package)
  /usr/lib64/python2.7/importlib/__init__.py(37)import_module()
-> __import__(name)
  /home/kotfic/tmp/girder_worker/girder_worker/tasks.py(1)<module>()
-> import core
> /home/kotfic/tmp/girder_worker/girder_worker/core/__init__.py(61)<module>()
-> def _resolve_scripts(task):

Which is triggered through the celery workers task import mechanisms whenever core_tasks is true in the configuration (the default). The code that manages that functionality is here

Could you elaborate on the problem that raised this issue? Thanks!

[manthey]

Using a job created for large image, I get the following error:

[2016-11-29 09:03:35,761: ERROR/MainProcess] Received unregistered task of type 'girder_worker.run'.
The message has been ignored and discarded.

Did you remember to import the module containing this task?
Or maybe you are using relative imports?
Please see http://bit.ly/gLye1c for more information.

The full contents of the message body was:
{'utc': True, 'chord': None, 'args': [], 'retries': 0, 'expires': None, 'task': 'girder_worker.run', 'callbacks': None, 'errbacks': None, 'timelimit': (None, None), 'taskset': None, 'kwargs': {'inputs': {'out_filename': {'type': 'string', 'data': u'ar300200.tiff', 'mode': 'inline', 'format': 'text'}, 'tile_size': {'type': 'number', 'data': 256, 'mode': 'inline', 'format': 'number'}, 'in_path': {'name': u'ar300200.tif', 'format': 'text', 'token': u'TzV0kiLejMmBNxNF7aIipSyafFyOSr6PMzgKRmPj65uQDRHPBBnkAboqnkionnOn', 'fetch_parent': False, 'mode': 'girder', 'type': 'string', 'id': '56bcb55d1d41c806366a9af1', 'resource_type': 'file', 'api_url': 'http://127.0.0.1:8080/api/v1'}, 'quality': {'type': 'number', 'data': 90, 'mode': 'inline', 'format': 'number'}}, 'task': {'inputs': [{'target': 'filepath', 'type': 'string', 'id': 'in_path', 'format': 'text'}, {'type': 'string', 'id': 'out_filename', 'format': 'text'}, {'type': 'number', 'id': 'tile_size', 'format': 'number'}, {'type': 'number', 'id': 'quality',... (2940b)
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/worker/consumer.py", line 456, in on_task_received
    strategies[name](message, body,
KeyError: 'girder_worker.run'

[zachmullen]

I'm seeing the same error as @manthey , can't run anything through the worker on latest master when I run python -m girder-worker. However, when I run girder-worker, things work. Something is different running via the console entry point as opposed to invoking __main__.py directly, not sure why that is.

[manthey]

As @kotfic pointed out (on the girder mailing list, I think), you now have to install using setup.py and not just the requirements.txt files. You can pip install ., too, which does the equivalent of python setup.y install.

[zachmullen]

If there really isn't a way around that, we need to at least update this page or our docs so that it no longer mentions python -m girder_worker, and we should explicitly say that it only works using the girder-worker executable.

[manthey]

python -m girder_worker still works once it has been installed.

[zachmullen]

Ah, you're right, that's fine then. My vote would still be to update those docs and change python -m girder_worker --> girder-worker.