API doc format (JSDoc vs. dojodoc) about delite

[wkeese]

See @lbod's conversion experiment for option using the JSDoc parser directly (option (3) above) at https://github.com/lbod/dojo/tree/dojo-jsdoc.

See also https://groups.google.com/forum/#!topic/jsdoc-users/U2DJcGph6Wg.

[wkeese]

https://npmjs.org/package/amd-doc will apparently help too.

[cjolif]

Might be worth looking at as well: https://github.com/Libertymutual/Dojo-Jsdoc-Sample

[asudoh]

Took some look at this topic. In summary, what I have found so far are:

1. JSDoc supports multiple inheritance out-of-the-box, though there is a bug. We can get it around by half-dozen lines of code in JSDoc template (Shown below)
2. JSDoc output for @module looks as though it was a CommonJS module instead of an AMD module. We may live with it, or we can get around it by some code in JSDoc template (Shown below)
3. JSDoc does not provide a way to auto-parse AMD dependencies. We haven't found a viable third-party solution for this, even in the tools mentioned in this ticket
4. JSDoc does not provide a way to auto-parse DCL class heritage. We haven't found a viable third-party solution for this, even in the tools mentioned in this ticket

And the crux seem the last two: Should we still pursue creating and maintaining tools to auto-parse AMD/DCL? I feel I can live with manually writing @module and @augments in my experience writing liaison code, but other libraries may have bigger cumbersomeness with it.

Here's the JSDoc template code to work around the issue with multiple inheritance, that should be applied to the 1st line in exports.publish() in templates/xxx/publish.js:

var table = {};
data = taffy(taffyData().get().reduceRight(function (data, entry) {
    if (!entry.memberof) {
        data.unshift(entry);
    } else if (!table[entry.longname]) {
        table[entry.longname] = 1;
        data.unshift(entry);
    }
    return data;
}, []));

Here's the JSDoc template code to work around the issue that modules shown in CommonJS format instead of AMD format, the 1st one should be applied to attachModuleSymbols() in templates/xxx/publish.js and the 2nd one should be applied to the first <h4> in templates/xxx/tmpl/method.tmpl:

var name = module.module.name.replace(/^module:/, ""),
    index = name.lastIndexOf("/");
module.module.name = name.substr(index >= 0 ? (index + 1) : 0);
module.module.amdSyntax = "require([\"" + name + "\"], function (" + module.module.name + ") { $1; })";

<h4 class="name" id="<?js= id ?>"><?js= (function () { var outName = data.attribs + (kind === 'class' ? 'new ' : '') + name + (kind !== 'event' ? data.signature : ''); return data.amdSyntax ? data.amdSyntax.replace(/\$1/g, outName) : outName; })() ?></h4>

[AdrianVasiliu]

I feel I can live with manually writing @module and @Augments in my experience writing liaison code, but other libraries may have bigger cumbersomeness with it.

After experimenting with a (very) partial migration to jsdoc in delite/deliteful, I also think we can live with this manual writing. It only represents a small fraction of the total time for writing the doc (or for converting from the dojo doc format).

That said, there would be a number of concerns - some minor, others more annoying. I hope most if not all can be addressed with a custom jsdoc template. A rough list:

• The info about the inheritance tree is absent (it doesn't say what are the parent classes).
• The old Dojo API doc provides nice navigation features such as compact list of properties, methods, and events, while the JSDoc output is a rough list with no summary.
• The old Dojo API doc has a checkbox to include/exclude the APIs inherited from parent classes. No such choice in the JSDoc output, the doc always shows the inherited APIs, thus potentially gives a huge list (without summary).
• For modules such as mixins, the jsdoc output is unfortunate because it starts with a generated usage example including "new Mixin()"... We do have a solution to avoid that in the old doc tool, would be a pity to regress on that by introducing a misleading usage example.
• For modules for which we do want to tell the user that they can do new Module(), it lacks the declarative counterpart (for that, we'd need to instruct JSDoc about the custom tag name).
• For optional function arguments, the "?" suffix for the type (say, String?) nicely adds the word "Optional" to the description of the parameter in the old Dojo API doc, while in JSDoc it adds a column "Argument" to the parameters table which contains nothing for a non-optional parameter and "" for an optional parameter. Not that bad, but less nice. An example:
  
• We used to have, and still have in delite/deliteful, API doc relying on the feature of the Dojo doc parser allowing to embed code in /* ===== ... ===== */ comments. One motivation why such comments still exist is to avoid the extra bits of declaring a variable just for the sake of doc. Personally I think it's not a big loss if we get rid of these comments (unless there's an equivalent solution in jsdoc).
• More generally, the concern would be whether JSDoc with our custom template will allow producing something at least as nice and usable as http://dojotoolkit.org/api/. At worst, we'd just target something good enough...

For your reference, here's the output I got using JSDoc 3.3.0-alpha4 tweaked using asudoh's workarounds (from the comment above):

• delite: http://adrianvasiliu.github.io/delite/delitejsdocout/index.html (7 modules partially migrated)
• deliteful: http://adrianvasiliu.github.io/deliteful/delitefuljsdocout/index.html (1 module partially migrated)

The corresponding input (migration of doc comments) is here:

• delite: AdrianVasiliu@5659cc8
• deliteful: AdrianVasiliu/deliteful@9193e59

[lbod]

It's good to see a few of us have tried this out now, I think we (@asudoh @AdrianVasiliu and others) all agree this is perfectly do-able.
ISTM when I looked at this a few months ago and now, the issue is knowing what tags to use and when; it would be good if between the 3 of us (or more) we could come to an agreement on this for the best approach.

What I highlighted to @wkeese

the current maintainer @hegemonic of JSDOC is an extremely helpful maintainer and willing to support current JSDOC3 AMD capabilities as well as documentation around this. Posting questions on the JSDOC users google group is a must and we should pursue this (considering Dojo/delite/deliteful would be the largest AMD projects to move to JSDOC, it will help AMD adoption for JSDOC too).

What I worry about

We're all trying to figure out the best way to migrate to JSDOC3 on our own with different understandings of the best approach.
I'd like to see a small team manage this and provide an agreed guidance document/wiki or w/e : "Dojo to JSDOC3 migration best practice"
There are a number of ways we can achieve the same results using different tags, so having a single approach of what tags to use and when is essential e.g.

• multiple inheritance
  • how to show inherited members
  • how to we link to inherited members
• documenting inheritance vs mixins
• how to show 'pseudo classes'
• how to link to modules or classes

etc etc

For now, I think we can ignore the theming work, we're all capable JS developers so it should be seen as business as usual how we present the documentation. Though I do think theming documentation in JSDOC3 is sparse.

It's unfortunate we need to spend time understanding how AMD should be documented but consider that end users will also need to adopt this, we should provide guidance, if this feed's back to JSDOC3 itself then it's more than worthwhile

[AdrianVasiliu]

Also noticed a cross-repository issue: to take an example, in the standalone doc generated for deliteful, the doc of a widget such as deliteful/ScrollableContainer does not contain any API inherited from its parents from delite, despite the @Augments tags for its delite parents. As long as we plan to deliver per-repository documentation, this looks as something we would need to dig into.

[asudoh]

The list from @AdrianVasiliu looks very nice and comprehensive (thanks!). I guess we may use this as the starting point to prioritize/cost them, as well as to let JSDoc team know so they can address some as @lbod pointed out. Some things I can say about this list are:

• If you do @param {int} [insertIndex] instead of @param {int?} insertIndex, JSDoc output indicates the parameter is "optional" (instead of "nullable")
• Though you cannot write pseudo code with JSDoc, you can define pseudo classes/methods, etc. Some examples can be seen in liaison/Handle.js and in liaison/BindingSource.js.

Another point @lbod raised (JSDoc comment standard) makes sense, too. JSDoc has many ways to achieve one thing, and using different set of tags in different ibm-js project will be confusing.

[AdrianVasiliu]

@asudoh wrote:

If you do @param {int} [insertIndex] instead of @param {int?} insertIndex, JSDoc output indicates the parameter is "optional" (instead of "nullable")

Indeed, and also adds this:

Although the column "Argument" in the table looks overkill to me, it's nicer. So that's the way to go (one item for our future JSDoc comment standard...).

@asudoh wrote:

Though you cannot write pseudo code with JSDoc, you can define pseudo classes/methods, etc. Some examples can be seen in liaison/Handle.js and in liaison/BindingSource.js.

Indeed, this does the trick, great. Now, your examples use @method. For real code, I used so far @type for properties, so I tried "@type {DomNode} module:delite/Widget#containerNode" for a pseudo-property, but this gives me a JSDoc error ("Unable to create a Tag object"). Instead, it worked fine with @var, but I realized @member might be the way to go (although in my case they produce strictly the same result) even for real properties, not only pseudo-properties (so I replaced everywhere @type by @member in my migration test).

Here are my updates to the doc comments: AdrianVasiliu@f0cc903

I wrote:

Also noticed a cross-repository issue: to take an example, in the standalone doc generated for deliteful, the doc of a widget such as deliteful/ScrollableContainer does not contain any API inherited from its parents from delite, despite the @Augments tags for its delite parents. As long as we plan to deliver per-repository documentation, this looks as something we would need to dig into.

In my eyes, it is a no-go to provide the doc of deliteful widgets while the APIs coming from delite are completely hidden. (The same holds for any other project which extends classes from another project.) Hence I gave a try to the generation of a merged doc by running JSDoc over both delite and deliteful at once. The merged output is here: http://adrianvasiliu.github.io/deliteful/delite-and-deliteful-jsdoc-out/index.html.

This brings back the "hidden" APIs, good, but it makes even more painful the lack of nice navigation capabilities of the generated doc: the global index is a flat list of all modules from all projects:

Hopefully, getting something better than that is "just" a matter of creating a more capable JSDoc template.

[cjolif]

Ok. Thanks everyone for all those details. I think we are now at a point where we either need an action plan to move to jsdoc or abandon the idea. As everyone seems to say we are not far from a solution I would propose an action plan.

So we need:

1. to define the exact list of our issues (basically what @AdrianVasiliu listed)
2. to bring our issues to jsdoc community (that's good for long term but depending on how fast they react I would not like us to "just" count on them fixing the issues)
3. to define a "standard" way of using jsdoc tags (i.e. so we all follow the same practices) as @lbod suggested
4. work on our own template that solves problems that are not solved by jsdoc community (should probably be started in parallel to reporting the issues so we don't get stuck if no solution is found at jsdoc level) probably based on what @asudoh started.

The real question is now who has time to take over one or several of those items ;)

[AdrianVasiliu]

A few updates:

I wrote:

For modules such as mixins, the jsdoc output is unfortunate because it starts with a generated usage example including "new Mixin()"...

In the process of collecting the issues encountered so far, I found the JSDoc tag @mixin, which does perfectly the trick. When using it, the output doesn't contain anymore the misleading usage example:

That said, once I use @mixin for some of the modules, for some reason the Index contains a new section "Mixins" but this is empty, and the mixin is still listed under the category "Modules":

I also tried to combine the usages of @mixin with @mixes, but didn't work better (and this issue: jsdoc/jsdoc#379 seems to hurt too).

Anyway, I've updated my migration test to use @mixin: b22976e.
The updated output (for delite+deliteful): http://adrianvasiliu.github.io/deliteful/delite-and-deliteful-jsdoc-out/Scrollable.html

Also, for some of the issues I previously reported (and for some we didn't encounter yet...), I found issues already registered at https://github.com/jsdoc3/jsdoc/issues. I note them apart (as agreed among us).

[cjolif]

Ok. Looks like we settle to make the effort to move to jsdoc. Do we keep that issue open until all API doc has been moved to jsdoc or do we close that one considering the decision has been taken?

[hegemonic]

Hi, I'm the maintainer of JSDoc. If an existing JSDoc issue is blocking your efforts, please feel free to add a comment to the issue. And, of course, please file new issues as you run into them!

It's very likely that you will want to create a heavily customized version of the JSDoc template. The JSDoc README now links to some third-party templates that might be a better starting point than the current default template.

[wkeese]

Thanks @hegemonic!

@cjolif, let's close this ticket since it's gotten pretty long. Since @asudoh and @AdrianVasiliu are working on the actual conversion to JSDoc they can open a new ticket if they like (or a pull request).