Function hooks when executing cells / notebooks? about nbclient

[MSeal]

I think papermill is a more established place for adding execution hooks in managed execution patterns, yeah. Keeping nbclient simple and low level makes it easier to understand and less opinionated about how you execute notebook code.

Want me to transfer this issue to papermill to continue the discussion about making it easier to add these particular hook patterns?

[MSeal]

I think you could do this in papermill by extending https://github.com/nteract/papermill/blob/master/papermill/engines.py#L231-L285 and calling a post-execution hook. Course you'd need to run code in the kernel to support it properly. Maybe a hook that saves the final line of each cell as a scrap on the notebook?

def cell_complete(self, cell, cell_index=None, **kwargs):
    post_cell = """
    result = _
    try:
      import scrapbook as sb
    except ImportError:
      print("scrapbook not installed, no outcomes saved")
    else:
      try:
        if result:
          sb.glue(f"__result_{cell_index}", result)
      except sb.ScrapbookException:
        print("Could not save result of cell {cell_index}")
    """
    self.execute_cell(post_cell, -1)
    self.super(cell, cell_index, **kwargs)

I think we'd want to expose call-backs for cell-complete and other events to the engine in papermill so you can just register an engine that supported it.

[choldgraf]

yes! that is the kind of pattern that I had in mind. You think papermill would be a better home for this kind of functionality than nbclient?

Agreed that the nicer way to handle this for our use-case (since I am still not sure what the "right" UX is here) is to just make it possible to extend the back that papermill uses along w/ access to the kernel

[choldgraf]

happy to do so, thanks for the explanation