Try if citations work about nbsphinx

[mgeier]

Thanks for researching this heading level stuff, but I think it isn't as simple as the doconce manual suggests. Actually, it's perfectly fine to follow a # header with a ### header without having a ## header in between.

The actual limitation is described in http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#sections:

All section title styles need not be used, nor need any specific section title style be used. However, a document must be consistent in its use of section titles: once a hierarchy of title styles is established, sections must use that hierarchy.

I'm having a hard time wrapping my head around that ... but I think I get it now ...
The strange thing with Markdown cells converted to Sphinx is that the actual number of # signs doesn't determine the heading level at all!
Whatever sequence of #-groups is used in the top of the notebook, defines the heading levels from the highest towards lower levels.
I.e., even if you start your notebook with #### My Title, this will become a h1 title in the HTML output.
In your notebooks, you are often using big jumps between # and ### or even ####, which may look as expected in the live notebook, but they will look quite differently in Sphinx's output!

The shortest sequence of titles I could come up with, that raises the above-mentioned warning, is:

# a
## b
# c
### d

---

About the citation stuff: It would be great if you could make a PR with a notebook and the corresponding .bib file, that works with nbconvert as shown here (and which will be documentation as well as test case for this feature). You can probably even run the actual nbconvert command in a %%bash code cell that does the conversion (of itself) to LaTeX. In a Markdown cell below, you can then create a link to the resulting .tex file. You can probably even run latexmk -pdf on it, since the necessary packages seem to be installed on readthedocs.org.
Don't worry about Sphinx at this point. Once you have something that makes sense in the nbconvert universe, I can take over and try to make it work in Sphinx-land, too.

[jhconning]

Have you had a chance to explore the use of citations? Judging from the discussion at that repo (see issue #11 ) that solution is broken as of nbconvert 4.0.

This prompts me to ask a related question and maybe a suggestion. From what I can see nbsphinx uses nbconvert to translate the .ipynb notebooks into an interim RST which is then passed onto Sphinx for HTML rendering. But no RST files are left behind in the conversion process. Is there a way to instruct nbsphinx to leave a copies of the RST files?

This might be useful for debugging purposes and possibly for using sphinx extensions (e.g. sphinxcontrib-bibtex to add citations to the RST files) to the RST. Not the best solution I know (since the whole purpose is to avoid such two-step procedures) but it could be useful for those situations where nothing else seems to work.

Having the RST files might also be useful for figuring out the cause of sphinx rendering problems. On simple examples I can get the sphinx table of contents to work with jupyter headers by correctly indenting based on header level (e.g. # for top level, ## for next level). But on a repo with a larger collection of notebooks here I am for some reason getting just a flat table of contents with no indents based on the headers. I can't figure out why. If I could look at the intermediary RST files I might be able to pin down where things might be going wrong.

Thanks again for this excellent resource.

[mgeier]

No, I haven't tried that yet.

It would be very helpful if you could create a PR that adds a new notebook to the docs with an example of how to use citations (even if that doesn't actually work yet).

Is there a way to instruct nbsphinx to leave a copies of the RST files?

Not exactly. And I don't think I want to do that. I wouldn't even know where to put those. I don't want to leave any garbage in the source directory, so it would have to go somewhere into the build directory, and then nobody would find those files ...

But I probably have something helpful after all. Just yesterday I added a commit (224a8ce) to master with some refactoring that gives you a nbsphinx.Exporter with which you can directly obtain the RST string from a given .ipynb file. It should work like this:

import nbsphinx
rst, res = nbsphinx.Exporter().from_filename('my-notebook.ipynb')
print(rst)

Would this be helpful in your case?

If you want to add stuff to the RST string, you can for now try to subclass nbsphinx.Exporter, override the method from_notebook_node() and monkey-patch that back into nbsphinx.Exporter. I haven't actually tried if that would work, though.

I was thinking about adding rst_header and rst_footer (or similar) configuration options to simply add content at the beginning or end of the RST string. But until now I didn't have a concrete use case.
Feel free to submit a PR or an issue if you would need such a feature.

I don't know how the sphinxcontrib-bibtex extension is helpful/harmful to the handling of citation used by nbconvert, but I'd sure like to support one of them or both in nbsphinx. Any additional information (including a PR) would be highly appreciated.

[mgeier]

I had a look at the problem with the headers you were mentioning.
I've already seen warnings like this before:

.../Dev-II/notebooks/FarmHousehold.ipynb:482: ERROR: Unknown interpreted text role "raw-latex".

I admit those are quite obscure, but they come from pandoc having problems knowing where your equations start and end.
It is important that you don't use leading or trailing spaces within the $ markers!

http://pandoc.org/README.html#math tells us:

The opening $ must have a non-space character immediately to its right, while the closing $ must have a non-space character immediately to its left, and must not be followed immediately by a digit.

[mgeier]

But I guess the problem you're mentioning is related to this:

/Dev-II/notebooks/jupyter_notebooks.ipynb:231: SEVERE: Title level inconsistent:

Learn more about jupyter and scientific python
----------------------------------------------

TBH, I don't understand those restrictions for title levels ...

[jhconning]

Thank you as always for fast and useful responses.

With your clues and this documentation page for a package that uses Sphinx I've learned:

Title level inconsistent
reST does not like jumps in the levels of headings. For example, you cannot have a === (paragraph) heading after a ======= (section) heading without a ==== (subsection) heading in between.
Subsection headings appear at the level of sections in the table of contents
This is a problem that arises if you split a Sphinx document at the subsection level. Then the subsection is the highest level in that part, and it will then move up to the section level in the table of contents ...

Mapping from this reST explanation to the jupyter markdown with which we write the problem seems to be that one cannot follow a # header with a ### header without having a ## header in between. It seems that messing that up in just one notebook can throw off the table of contents rendering for the entire collection. I'll go through the documents carefully.

The nbsphinx.Exporter sounds very useful and indeed a better alternative than dumping the RST files. I'll look into it.

It would be very helpful if you could create a PR that adds a new notebook to the docs with an example of how to use citations (even if that doesn't actually work yet).

I'm still rather new to Sphinx but I will dig deeper into alternative ways to include citations and generating bibliographies. I will first explore sphinxcontrib-bibtex within sphinx RST files and then think of ways to use that in a notebook context (and submit a sample notebook if I can).

But the key question will be how to write in jupyter markdown something that gets translated into sphinx peppered RST. The sphinxcontrib-bibtex allows you to add citations mid-sentence and later generate a bibliography in a RST document like this:

See :cite:`1987:nelson` for an introduction to non-standard analysis.

.. bibliography:: refs.bib

But how to write something that gets translated into that using jupyter markdown? I see that you have ways to use cell metadata to insert sphinx directives. That would work for the bibliography directive but how to insert the mid-sentence cite: entries? When you invited me to submit a PR with a sample notebook, were you suggesting that I suggest such a method?