Review #3 about post--building-blocks

[arvind]

Thank you for the in depth and thoughtful review. We're glad that you found our work exciting! We've responded to points inline below.

One comment on this: while using visual examples definitely gives a finer degree of granularity than lumping several abstractions together under one description -- e.g. “floppy ear detectors” -- it seems likely that any visual concept that the network reliably learns that is unfamiliar to humans is likely to still be overlooked, even using canonical examples. (This would be an example of the human scale issue). One way to study this further (probably in future work) might be to extract semantic concepts that are repeatedly learned, and thus identify these important concepts that aren’t immediately human recognizable.

We agree! The factorization approach we used is only one from a larger family of techniques, and we’re excited about future research that explores the impact of using these other techniques. We’ve made this clearer with 790f4c9.

I would have really liked to see a simple equation or two (maybe even as an aside), so that the mathematically minded readers can have a (simplified) mathematical description.

We want to keep the focus of the article on how the various interpretability building blocks come together to form rich user interfaces. The challenge is that digging into the details of each of the individual building blocks involves many moving parts (e.g., with feature visualization, we would need to unpack initialization, parameterization, optimization objective, procedure, etc.; with matrix factorization, it’s unclear to us how more formal mathematic description would not essentially reduce to redefining NMF).

With that said, we’ve improved our prose description of our approaches in 3c4312b, and added a link to our previous feature visualization article (which discusses our specific approach at length) in 90979c4.

Is this kind of influence visualization robust to other attribution methods?

Our interfaces reify attribution methods, so we believe that they will be as robust as the attribution methods they rely on. We can use the same interface but swap out the underlying attribution method used, and we think doing so (and evaluating the resultant interfaces) is a promising direction for future work.

It is good to see the human scale problem being addressed but the results here appear to be preliminary, and there is likely much more work to be done in the future.

We agree, and have clarified this as of 790f4c9.

Have the authors considered methods to automatically cluster together similar concepts? (E.g. first collecting a set of canonical examples corresponding to a certain class, and then applying clustering to see what natural groups they fall into?)

Yes! This is one of the things we are actively researching, but unfortunately don’t have conclusive results yet to include in the article :)

Without having read other distill articles, it’s not immediately obvious that the little orange box selecting a certain part of the image can actually be moved around, and it might be worthwhile noting this somewhere.

We’ve improved our annotations and controls for all diagrams (e.g., with 5f9cebb, we’ve added new zooming controls for this particular interface).