Document job event binding and output reference kwargs. about girder_worker

[subdavis]

Converting this issue to a docs update issue. I'll make a PR sometime after I've used it a bit more.

[manthey]

I assume we want to pass some small information to the endpoint, such as the job's completion status and the job ID, but not large data, like an output file. Further, I assume the hook could have some auth token to send to the endpoint to show that it is a trusted source.

Do all status hooks get triggered after all result hooks, as the success of processing result hooks is part of the final status of the job? Or would you want a flag indicating where in the job's life cycle the status hook should be called?

[subdavis]

I assume we want to pass some small information to the endpoint, such as the job's completion status and the job ID, but not large data, like an output file

Yes.

Further, I assume the hook could have some auth token to send to the endpoint to show that it is a trusted source.

Also yes.

Do all status hooks get triggered after all result hooks.

That's how I imagine it would work.

Or would you want a flag indicating where in the job's life cycle the status hook should be called?

I don't know. I've never had a use case where I cared about a status other than completion. We could definitely have hooks that trigger for specific statuses, like queued, started, completed, but I've never needed any of that, so I think I'd prefer to do option 1 I really don't know.

[zachmullen]

The way I've always handled this is to attach special metadata fields to the job, then hook into the job update event on the server and check if it's a status update, and if so take action on whatever piece of data I linked into the job.

# create some job
job = ...
# link it to some piece of data
job['relatedThingId'] = item['_id']
Job().save(job)

def load(self, info):
    events.bind('jobs.job.update', 'my_plugin', jobUpdated)

Here's an example

Maybe you already knew about that and were making a case that the webhook approach is superior, however unless we actually have a use case to talk to something other than Girder I'm not convinced. What makes this preferable to using the existing job update endpoint to listen to job updates? If we just want more specificity of events, we could add those as more granular server-side events, such as jobs.jobs.update.status instead of just jobs.job.update, for example.

[subdavis]

Maybe you already knew about that and were making a case that the webhook approach is superior.

No, I didn't know this was possible. It would be great to add this to the docs. I'll try it out. Thanks!

[zachmullen]

Cool. FWIW, another common need is to come from the other direction when linking data -- that is, linking file(s) produced by a worker job back to some existing data in Girder. The way to accomplish that is using the reference feature on the output, like so.

[zachmullen]

Thanks. Our docs for the new girder_worker architecture are abysmal. We've heretofore done everything via tribal knowledge and reading source code.