API Discussion about grappling-hook

[creynders]

So, I've been thinking this through. I think the problem I have is that it's still not clear enough, which is why I wasn't satisfied with "my" API either. I'm going to ditch the thematic hooky method names and try to find better, maybe more generic, but hopefully more understandable names.

That said, I'd REALLY (really) like to wrap this up.

Also, I think the word "hook" is used in many meanings: as a noun, as a verb, one time it's the function that gets executed on an event and the other time it's the event itself, I'm trying to find a consistent use for it.
I'll annotate any changes I made from your proposition

There's two things I try to keep in mind:

1. Actually the API consists out of 2 API's: a consumer- and an emitter-facing API
2. Automatic vs manual setup

The parallel execution thing is something I have the most trouble with and will tackle that last.

Consumer facing

Since pre/post are fixed, I'd really try to keep these all of these API methods to one word to be consistent.

Adding middleware

sender.pre("save", fn);

sender.post("save", fn);
//equals
sender.hook("post:save", fn); //CHANGE: new convenience method

Async middleware is automatically detected by the presence of done, next or callback in the argument names.

Removing middleware

sender.unhook("post:save", fn);

Emitter facing

Here I'm going for clarity.

Creating instances

// set up a class for hook emissions
grappling.attach(MyClass, {
    wrapMethods : true, //CHANGE: new option
    strict: true
});

// set up an existing object for hook emissions
grappling.mixin(instance, {
    wrapMethods : ["save", "post:destroy"]
});

//creates vanilla object with hooking methods
instance = grappling.create(); //CHANGE: extra convenience method

When wrapMethods is true, it will iterate over all methods and wrap them with pre and post calls, or you can pass it an array of methodnames or event:methodname, to wrap specific methods with hook calling. See also the wrapMethods method, you'd use one or the other.

strict determines whether an error is thrown if a consumer tries to hook into a unregistered event. (Meant for delegated dynamic hooking, this way consumers can register hooks before a broadcaster takes control of the delegate to emit hooks)

Registering hooks

Manual

// registers 'pre' and 'post'
this.allowHooks("save"); //CHANGE: renamed 'hookable'

// only allows 'pre
this.allowHooks("pre:save"); '

Automatic

// wraps "save" with pre
this.wrapMethods("pre:save"); //CHANGE: renamed "hookup"

// wraps "save" with pre:save and post:save
this.wrapMethods("save"); 

// wraps all methods
this.wrapMethods(); //CHANGE: extra functionality

// wraps fn with pre:save
this.wrapMethods("pre:save", fn); 

// wraps fn with pre:save and post:save
this.wrapMethods("save", fn); 

Calling hooks

this.callHooks("pre:save", ...args); //CHANGE: renamed "hooks"
this.callHooks("pre:save", ...args, callback);
this.callHooks(context, "pre:save", ...args, callback); //CHANGE: extra argument `context` to allow hooks be called in another context than the emitter. (Pretty sure we need this)

Introspecting hooks

this.getHooks() // { pre: { save: [fn] } }
this.getHooks('save') // { pre: [fn] }
this.getHooks('pre:save') // [fn]
this.hasHooks('pre:save') // shortcut for this.getHooks('pre:save').length ? true : false

Removing middleware

The consumer facing API already mentioned unhook, I split it up, just because conceptually this rather belongs to the emitter API.

this.unhook("post:save");
this.unhook("save");
this.unhook();

The only slight problem I have with unhook is that it seems inconsistent with get*H*ooks, call*H*ooks, ... but on the other hand I wanted to keep it consistent as the negative version of hook. And you might argue that hook and unhook are verbs, while in getHooks etc. "hooks" are nouns.

Parallel vs serial

Then, the whole parallel thing. My main trouble with how you proposed it, is I think people will find it confusing and am also struggling with imagining how to get that working technically.

E.g.

var parallelFn = function(...args, next, done) {
  next() // middleware continues
  doSomethingAsync(function() {
    done() // hook is complete, save can continue
  })
};

emitter.pre("save", parallelFn, parallelFn, parallelFn);

I need to somehow establish when all of them are done, so I'll need to maintain a list of those, all of these could call their callbacks with errors etc. No rocket science, but it does complicate things more than I like. It seems beyond the scope of what we set out to create.

Also, I'm not entirely sure of the implications of this:

emitter.pre("save", parallelFn, serialFn, parallelFn, serialFn, serialFn);

It seems that if we get the only-parallel flow working then it won't be a lot of trouble to get a mixed one working, but I'm not entirely sure.

Maybe I'm just overcomplicating things in my head :)

On the other hand I do like the inversion-of-control, i.e. the middleware controls whether it's successors are called in parallel or not.

I'd really want to stick to an existing library like async for this kind of stuff, but AFAICT it doesn't allow the above mentioned middleware based control flow.

[JedWatson]

Looks good. Just a few tweaks.

I'm personally not keen on the "wrap everything" option. I think there's too much scope for unexpected behaviour; if someone adds a method to a class from another area of the system without realising everything gets hooks added, it's not great. The API isn't overly verbose or hard to use so let's force authors to be explicit.

On a similar topic, let's reduce the number of ways of doing things as much as possible. Adding alternate "convenience" syntax is a trap mongoose falls into way too often, leading to lack of clarity about whether there is functional difference between implementations. TBH I think I have been guilty of this in some of Keystone's early design too and would like to reign it in a bit in upcoming versions.

Only including modified sections, my proposals are:

Creating instances

// set up a class for hook emissions
grappling.attach(MyClass, {
    // no general wrap, each method should be wrapped explicitly
    strict: true
});

// set up an existing object for hook emissions
grappling.mixin(instance, {
    // no general wrap, each method should be wrapped explicitly
});

It's possible to get a similar syntax to the wrapMethods option by just chaining:

grappling.mixin(instance).addHooks(["save", "post:destroy"]); // new syntax, see below

Registering hooks

Manual

// registers 'pre' and 'post'
this.allowHooks("save");

// only allows 'pre
this.allowHooks("pre:save");

Automatic

// CHANGE; renamed
// wraps "save" with pre
this.addHooks("pre:save");

// wraps "save" with pre:save and post:save
this.addHooks("save");

// CHANGE; I don't think we should support a wrap-all method
// this.addHooks();

// wraps fn with pre:save, fn is added as this.save()
this.addHooks("pre:save", fn, ...args);

// wraps fn with pre:save and post:save, fn is added as this.save()
this.addHooks("save", fn, ...args);

Two error conditions here:

• If fn is provided but a property already exists with the given name
• If fn is not provided and no method exists with the given name

If we want to provide a shorthand syntax for registering multiple hooks, let's do it with an array or object:

// wrap multiple methods
this.addHooks(["pre:save", "remove"]);

// attach multiple methods
this.addHooks({
  "pre:save": fn1,
  "remove": [fn2, ...args]
});

Parallel vs serial

I borrowed this concept from the hooks library that mongoose uses, and can see a use for it. The power it has over async is that you may not specify all the hooks in one place, so using any other library may not be an option. Also you're right async doesn't support it.

However, it does increase implementation complexity and I'd be happy to leave it out for the first version. We can always add it later, it's not going to be a breaking change if/when we support it.

EDIT if we only implement one flow (not parallel + serial) we should implement serial, as there are times parallel may not be safe.

[creynders]

Let's

[JedWatson]

👍

[creynders]

@JedWatson updated JedWatson/asyncdi#1 JedWatson/asyncdi#2 JedWatson/asyncdi#3

• Cleaned up code style
• Added tests if missing.
• rebased them

I need all 3 of them to get the above working.

[creynders]

A few adjustments I'd like to propose based on my attempts to find a fixed terminology for everything.
I need this for consistency in both documentation and code.

• a "hook" is an intercept in a process of a service that allows you to modify said process through a "callback"
• a "qualified hook identifier" exists out of a "qualifier" and an "action" separated by a colon, e.g. "pre:save" I.e most methods accept a combination of qualified hook identifiers and/or actions.
• "callback" is a function which is called by the service right before or after a particular action/process occurs, subdivided into:
  • "middleware" is a "callback" which on its turn accepts a next (or done) "callback" (i.e. async)
  • "simple callback" is a non-middleware "callback", i.e. (sync)

So, keeping this in mind, I realised we use "hook" ambiguously: to denote the interception point and to label the callbacks. So, I would like to propose the following:

• getHooks renamed to getMiddleware (or getCallbacks, but I prefer the first)
• callHooks renamed to callHook, since it only calls exactly one, qualified, hook.

Then, I'd like to restrict getHooks (or getMiddleware) to accept only qualified hook identifiers, since currently the output is mixed: sometimes it's an array, sometimes it's an object. Which leads me to:
hasHooks doesn't really make sense with a getHooks that can return an array or an object.
If we restrict getHooks to arrays, it makes sense again.
However.
The meaning of "hasHooks" is again ambiguous. ATM it means "does this service have middleware registered for this specific hook?".
We could keep hasHooks but change what it does to "does this service allow before/after interception for this specific action?" I.e. if you do

service.allowHooks("save")
  .hasHooks("pre:save")// returns true.

It simply allows you to query whether a service really does implement a certain hook. This however, won't play nice with strict:false mode and I have no idea how to solve that.

Then, I'd add a hasMiddleware method (which does what hasHooks does now) and restrict it to qualified hook identifiers only.

[creynders]

I'm also narrowing down the accepted addHooks parameters. Otherwise I have to write all kinds of crazy argument parsing, and TBH it could get pretty confusing, so what it boils down to is: you can pass strings (i.e. existing method name) or objects with hook/function pairs.

E.g.

//wrap existing methods
instance.addHooks('save', 'pre:remove');

instance.addHooks({
   "save": instance._upload,
   "pre:remove": function(){
     //...
   }
});

These can be mixed however:

instance.addHooks("save", {
   "pre:remove": function(){
     //...
   }
});

[creynders]

Getting there. I pushed to v1. Already using the new API method names as described above.
Need, to simplify addHooks though.

Phew, 77.5% code coverage.

[creynders]

Updated the v1 branch again. Refactored, optimized, added tests. And started on the docs.

I'm at 100% statement coverage and 92% branch coverage, however the latter is due to fall-throughs, i.e. parts where nothing should happen if a conditional fails. Seems a bit silly to test do-nothings.

We're nearing completion, only the documentation should be finalized. Feel free to take a look @JedWatson and modify the readme as you see fit. My English faulty sometimes can be 😉

[creynders]

There's one (non-blocking !) thing I'd like to solve:

ATM if you pass a callback to a wrapped method it will automatically use it as a final callback, i.e. it's called after the post hooks have all finished, e.g.

instance.save = function(callback){
  console.log("SAVE");
  setTimeout(function(){
    console.log("SAVED");
    callback && callback();
  }, 1000);
};
instance.addHooks("save");
instance.pre("save", function(){
  console.log("PRE");
});
instance.post("save", function(){
  console.log("POST");
});
instance.save(function(){
  console.log("CALLBACK");
});

# outputs
PRE
SAVE
SAVED
POST
CALLBACK

However. First of all is this correct? Or should the callback be called before the POST middleware is run, which does seem logical too.
And. There's a problem, the above runs as expected, but if you call save w/o passing it a callback, then the order gets messed up.

instance.save();

# outputs
PRE
SAVE
POST
SAVED
CALLBACK

It seems like an odd discrepancy and it's due to the limitation on function introspection. If no callback is passed to save, I have no way of knowing whether the method expects a callback or not (unless we use asyncdi, but that would mean we have to limit to specific names, which seems like a really bad idea)
The question remains, is a callback passed to a hooked method supposed to be called before or after the post middleware?

[creynders]

Pandora's
never ending
box
of
evil incarnate
😉

[creynders]

Two other options:

1. require the wrapped method to accept a callback as the last parameter. period.
2. allow configuration of sync/async through addHooks, but then we'll need to use a configuration object.

[JedWatson]

Closing, API has landed. Awesome work @creynders 😄

To wrap up the final discussion for anyone following, we went with mandatory callbacks (1) and the logs output as described above.