Documentation generation with chapter support about torchnet

[gforge]

Just in case you are interested I've just finished updating the torch-dataframe and in this release I've also fixed the docs - it requires though a bit of code and complexity.

[lvdmaaten]

The documentation is actually automatically generated from the comments in the code by running -largcheck.dump -e "require 'torchnet'. So next time we would re-generate the documentation, the changes would be overwritten. It may be worth looking into supporting chapters in argcheck.dump though.

[Atcold]

Agreed. Right now it's quite unreadable.
Thank you for the heads up about the overwriting phew 😌

[gforge]

We use in the torch-dataframe a similar argcheck structure and use a simple doc.lua for auto-generating the doc-folder. I think it could easily be adapted to the torchnet-package. Adding a deeper function index shouldn't be that hard - we use the same anchors as you <a name="ConcatDataset"> but we haven't made any good use of them yet. Also the current implementation looks for the first line in order to retrieve the file title/chapter that is added to the README.md - you will need to add a doc[[]] at the top if you don't want to use the filenames as

Btw - thanks for this awesome package. We've just finished adding compatibility and it should play nicely with the tnt.Dataset classes. Have you thought of adding a Wiki that could contain a list of compatible packages?

[gforge]

Here's a doc-script that I did for getting all the docs into separate files according to folders for easier navigation. It doesn't work for the main README file but otherwise it does separate the documentation nicely (you can find it here):

local argdoc = require 'argcheck.doc'
local paths = require 'paths'
local tnt = require 'torchnet'

-- This is a workaround for reloading already loaded class
torch.class_org = torch.class
torch.class = function(a,b,c)
  if (tnt[a:gsub("tnt.","")]) then
    return tnt[a:gsub("tnt.","")]
  end
  return torch.class_org(a,b,c)
end

local base_path = paths.thisfile():gsub("doc.lua$", "?.lua")

-- Fails due to an argcheck for makepackageserializable
if (false) then
  -- README file
  local readmefile = io.open("README.md", "w")

  argdoc.record()
  local init_file = base_path:gsub("?", "init")
  assert(loadfile(init_file))()
  local content = argdoc.stop()

  readmefile:write(content)
  readmefile:close()
end

if (not paths.dirp("doc")) then
  paths.mkdir("doc")
end

function getFileDoc(full_path, file_name, header, content)
  -- Get documentation
  argdoc.record()
  assert(loadfile(full_path))(tnt)
  local file_content = argdoc.stop()
  file_content = file_content:gsub("^%s*(.-)%s*$", "%1")

  local rows = file_content:split("\n")
  local title
  for _,row in ipairs(rows) do
    if (row:match("^#")) then
      title = row:gsub("#", ""):gsub("^%s*(.-)%s*$", "%1")
      break
    end
  end

  -- If title not found
  if (not title) then
    title = file_name:sub(1,1):upper() .. file_name:sub(2)
  end

  if (file_content:len() > 0) then
    local anchor = "__" .. file_name:gsub(".lua$", "") .. "__"

    header = header ..
    ("\n- [%s](#%s)"):format(title, anchor)
    content = content ..
    ("\n\n<a name=\"%s\"></a>\n%s"):format(anchor, file_content)
  end

  return header, content
end

local doc_readmefile = io.open("doc/README.md", "w")
doc_readmefile:write("# The torchnet API\n")

for _,type in pairs({"dataset", "engine", "log", "meter", "utils"}) do
  -- Load all extensions, i.e. .lua files in extensions directory
  local tnt_path = base_path:gsub("[^/]+$", "") ..  type .. "/"
  if (paths.dirp(tnt_path)) then
    local header = "# Documentation for " .. type .. "\n"
    local content = ""

    -- We want the init file if it exists first
    local init_file = tnt_path .. "init.lua"
    if (paths.filep(init_file)) then
      header, content = getFileDoc(init_file, "init.lua", header, content)
    end

    for file_name in paths.files(tnt_path) do
      if (file_name ~= "init.lua" and
          file_name:match("[.]lua$")) then
        local full_path = tnt_path .. file_name
        header, content = getFileDoc(full_path, file_name, header, content)
      end
    end

    if (content:len() > 0) then
      local doc_file = io.open("doc/" .. type .. ".md", "w")
      doc_file:write(("%s\n\n%s"):format(header, content))
      doc_file:close()
      doc_readmefile:write(("\n - [Docs for %s](%s)"):format(type, type ..".md"))
    end
  end
end

doc_readmefile:close()

There is some inconsistency in the formatting of the docs but otherwise it should do the job of providing a more complete API-documentation. Due to the sundown conversion it is important to run with /dev/null, i.e. th doc.lua > /dev/null or you'll have bash color markup in your output.

[lvdmaaten]

This looks very nice, thanks! Let me test this and look into where to put this code (I'm thinking in argcheck?).

[gforge]

Thanks. Glad you found it useful.

We have the code in a separate doc.lua in the root folder as it requires the script to be run with a > /dev/null - it's a little manual but works. Not sure what you mean about putting it in argcheck.

As you may have noticed there are some minor annoyances:

• Iterator and ParallelIterator don't end up next to each other. Furthermore the iterator has a different main heading structure "Dataset Iterators" while the other has "tnt.ParallelDatasetIterator(self[, init], closure, nthread[, perm][, filter][, transform][, ordered])". I think this could benefit from deepening the folder structure together with an init.lua.
• Some of the classes have only one doc element at the __init function while others have on every function. I prefer the latter as I believe that having the documentation closer to the actual code reduces the risk for documentation issues.
• The code is already prepared with custom anchor tags that should probably be used in the contents. This is trivial to implement and I have an example set up here but the doc.lua is now somewhat more advanced and also more sensitive to doc errors, e.g. if a coder forgets the anchor tag is forgotten that function won't end up in the docs.
• I think there is a doc-error in the meter section, see my StackOverflow question for details.

[lvdmaaten]

Thanks @gforge. I will look into this.

[gforge]

I see that this issue has been open for a while - any decision on if we should implement the doc-system?

[Atcold]

@gforge, that would be great! There's plenty of good explanations, well mixed up.

[lvdmaaten]

It would certainly be nice to have --- we would greatly appreciate a PR implementing this if you know how to do it!