.tex file has no document content about ipypublish

[chrisjsewell]

Ok, so this is the basic premise for the output formats:

1. For latex_standard_article the format rules are; to create a pdf with all notebook cells represented (simplistically) in the final tex/pdf, as per the original nbconvert article format. Essentially, running jupyter nbconvert --to pdf Untitled.ipynb and nbpublish -pdf Untitled.ipynb should give you almost exactly the same tex/pdf.

2. For latex_ipypublish_main (the default), the formatting rules are:

   a. Output all markdown cells, unless they have the latex_doc.ignore metadata tag
   b. Output no code cells or their output, unless they have a suitable latex_doc metadata tag (see here)

   my use case for this (and why I started building the package!) is that I have a notebook with lots of exploratory code and plots, which I don't want output in the final pdf. So I want to specifically select what cells are output.

So, firstly, try creating Test.ipynb (in a text editor) with the following text. Open it in Jupyter and run all the cells, then save it. (note you don't need to explicitly put %matplotlib inline; import matplotlib.pyplot as plt, since this is done in ipynb_latex_setup)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ipypublish.scripts.ipynb_latex_setup import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chart 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.plot([1,2,3], [4,2,7]);"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

Now run nbpublish -f latex_standard_article Test.ipynb and you should get:

	\begin{document}

\title{Notebook}
\maketitle

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}1}]:} \PY{k+kn}{from} \PY{n+nn}{ipypublish}\PY{n+nn}{.}\PY{n+nn}{scripts}\PY{n+nn}{.}\PY{n+nn}{ipynb\PYZus{}latex\PYZus{}setup} \PY{k}{import} \PY{o}{*}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}2}]:} \PY{k}{for} \PY{n}{i} \PY{o+ow}{in} \PY{n+nb}{range}\PY{p}{(}\PY{l+m+mi}{10}\PY{p}{)}\PY{p}{:}
            \PY{n+nb}{print}\PY{p}{(}\PY{n}{i}\PY{p}{)}
\end{Verbatim}

    \begin{Verbatim}[commandchars=\\\{\}]
0
1
2
3
4
5
6
7
8
9

    \end{Verbatim}

    Chart 2.

    \begin{Verbatim}[commandchars=\\\{\}]
{\color{incolor}In [{\color{incolor}3}]:} \PY{n}{plt}\PY{o}{.}\PY{n}{plot}\PY{p}{(}\PY{p}{[}\PY{l+m+mi}{1}\PY{p}{,}\PY{l+m+mi}{2}\PY{p}{,}\PY{l+m+mi}{3}\PY{p}{]}\PY{p}{,} \PY{p}{[}\PY{l+m+mi}{4}\PY{p}{,}\PY{l+m+mi}{2}\PY{p}{,}\PY{l+m+mi}{7}\PY{p}{]}\PY{p}{)}\PY{p}{;}
\end{Verbatim}

\texttt{\color{outcolor}Out[{\color{outcolor}3}]:}\begin{center}
    \adjustimage{max size={0.9\linewidth}{0.9\paperheight}}{Test_files/output_3_0.pdf}
    \end{center}
    { \hspace*{\fill} \\}

	\end{document}

and run nbpublish Test.ipynb and you shoud get

	\begin{document}

Chart 2.

	\end{document}

[angeloklin]

Hello Chris,
I believe I was misled by my cursory reading of the documentation.

My brain got that the output would be out regardless (unless stated otherwise). Probably because that was what I was looking for. My brain workflow was to have a working version with the actual content and just skip the source code, as the target audience would not be interested in it. So the ipynb here would be a publishable (!? publishable !?) quasi-print quality, without any extras.

But I totally get your point of having test code all around so you would focus on pushing what you do want to come out.

Maybe an idea for your apreciation would be to have a 3rd template that would put the output automatically out, for the lazy people that don't want to mess with metadata cell by cell. Honestly, it would be easier to just use the one line solution with the -f option pointing to an alternative template.

I will check the metadata content again to force the output. It then should work.
After I test it tomorrow, I can close this ticket as well.

Sorry for the confusion.

Thanks!

[chrisjsewell]

Yeh, again thanks its good to get a user perspective.

So yes, as you mention, there are two routes to go;

1. using latex_ipypublish_main with meta tags in the notebook. If you didn't know, you do View/Cell Toolbar/Edit Metadata to access these (in #1 I talk about the goal to have a nicer way to do this).

2. creating a new converter plugin to convert the notebook according to a different set of rules (e.g. including code and/or plots by default). My "dream" (lol) is that, as a community we can create a library of the export plugins for different use cases / output formats.

• As the default then, I could think about making a latex_ipypublish_draft plugin, with all output put in the pdf by default. The only downside of such a document is that you wouldn't have a nice caption under the plot or label (to reference the plot in other parts of the document)
• it might be good to have a specific check for .tex files output with nothing between the \begin{document} and \end{document} and output a specific warning in the log.
• also in the log, it would be good for it to give a summary/verbose output of what cells it is outputting to the document and which it is ignoring, to avoid this kind of confusion.
• put in the documentation about accessing metadata (View/Cell Toolbar/Edit Metadata)

[chrisjsewell]

FYI, for your test notebook, try this :)

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ipypublish.scripts.ipynb_latex_setup import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(10):\n",
    "    print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chart 2\n",
    "\n",
    "with a nice reference to it \\cref{fig:chart2}."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "latex_doc": {
     "figure": {
      "caption": "the caption for chart 2",
      "label": "fig:chart2",
      "placement": "H"
     }
    }
   },
   "outputs": [],
   "source": [
    "plt.plot([1,2,3], [4,2,7]);"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}

[angeloklin]

Hello Chris,
I think you are with a very nice to do list in your hands.
The log options are useful for some people, but most probably won't use it. Maybe medium priority for those.
I like the draft idea, as it would solve a simpler problem which is just to publish the output but not the input. My guess is that most people wouldn't mind as the main intent is just to hide the code.
Your default serves a more ambitious goal to have a professional look for serious publications (very good and well done!). More configuration but it is a small price for the quality of the final product.

Again, I will just check your instructions tomorrow and will close the ticket if it all goes as planned.
Looking forward to seeing what the version 0.5.1 will bring. The 0.5.0 always has a minor typo or something that need to be fixed. :)

Thanks for your support.

[chrisjsewell]

haha cheers, yeh just keep the feedback coming :)

My last top tip is, if you don't want all the extra files (produced by latexmk) in the converted folder, use nbpublish -pdf -ptemp Test.ipynb. The only drawback, is that latexmk can use these files to work out what changes have been made to the document since it last run and only process those, rather than make the entire document from scratch (particularly handy for making many small changes to long/complex documents).

Also, if you haven't noticed I've also created a new command nbpresent Test.ipnb which will convert and open your notebook as a slideshow.

• This isn't related to nbpresent (which I've just found out about from you comments) so maybe I should change the command name to avoid conflict.
• I've just noticed from the Test.ipynb that it outputs "output_type": "stream" in a weird position

[chrisjsewell]

Moved the action points to #28, so I'll close this ticket now

[chrisjsewell]

oops actually I'll wait for you to check it tomorrow ta

[angeloklin]

Hello Chris,

Sorry. It just went over my head. :(

I thought I would be able to do it but it still escapes me. I believe I saw the solution in one way and you probably envision in a different one.
I was trying to find a tag to force the output of code, but not the code itself. I went to the code and definitions and only found code, table, figure. If I use code I get the code, but still no output.
I tried also to create a new format, but could not use it with the -f option.

To give you some context on what I would like to do, it is as follows:

• I have some data from a database, which I converted to a flat file;
• I am loading the flat file and doing some initial EDA (Exploratory Data Analysis)
• I have a loop for each column and am I doing some simple investigation, which in essence looks like:

df = pd.read_csv()

for c in df.columns:
  c.describe()
  c.plot()

There is more to the code, but you get the idea. From that, the output would be a repetition of some text and a chart.

I could use the standard output, but it is quite simple. I did like the ipypublish latex style, but it leans more to an academic paper (which is why you created it, so I get it).
So goes back to the suggestion of having a format that by default would print out the markdown and all output without the code and/or have a module template like result analogous to code or content in the standard.

As I said I tried to do it, but requires learning a lot and takes quite some time. I have to go back to my analysis and modelling.

It would help me if you could digest it in smaller bits in a how to kind of format, on what metadata to use, it would be great. If it requires creating a whole new template/style, I would leave that to you to improve your code when you see fit.

In any case, I have learned quite a few new things and tricks in trying to find a solution for my need. Also nice to see how people can help each other. So thanks for that too.

Thanks!

[chrisjsewell]

Ah you're so close!

1. a plot is just a subcategory of a figure (this is a latex term for basically any graphic that you want to add a caption to)
2. the metadata acts on the entire cell, so both the input (i.e. the code) and the output (i.e. the plot)

So literally replace code with figure, in that same place, and you will get the plot instead of the code. Or leave the code and add figure below it, and you will get both the code and the plot, e.g.

{
"latex_doc": {
  "code": {},
  "figure": {
    "caption": "Figure caption.",
    "label": "fig:flabel",
    "placement": "H",
    }
}
}

Does that explain it?

[chrisjsewell]

[angeloklin]

Hello Chris,
Didn't expect to get you awake at this time.

The figure was working before.
My issue now is the 0, 1, 2, 3,... output without the code.

Do you have a trick for that one?

Thanks

[chrisjsewell]

haha, bit of a night owl. So just to confirm; the figure is working now and, if I put the gif (above) in the documentation, and altered the following line:
For figures (i.e. any graphics output by the code), enter in cell metadata:
do you think that would make it clearer?

For the 0,1,2,3,.., no I didn't think to add a text meta type for my use case, so I will do that

[angeloklin]

Hello Chris,
Yes, the image worked since you sent the details the first time. Thanks.

For me, the confusion was that I was looking for the output of the text as well.
The visual clue is nice, but it might make the page really long. Personally, I would be fine with something like: "for this: do that, where that is a more step-by-step instructions' list".
The catch here for you is that you have the big picture and are familiar with the code and the processes. For someone looking for a quick fix one (myself) would prefer to be lead by example (literally) :).

Nevertheless, you have done a great job.

PS: Feel free to close the ticket if you want.

[chrisjsewell]

ta