Installing problems about ipypublish

[chrisjsewell]

Hey Angelo, thanks for the interest and letting me know. It was just literally because I'd forgot to put the setup.cfg file in the repo! So it wasn't doing a universal build. I've just pushed the update to pypi and tested it myself, and it looks to be working now (you should get v0.4.1). So give it a go and let me know how you get on!

[angeloklin]

Hello Chris,
Thanks for your prompt reply and fix.
I am off my business hours here in Australia, but since you were so efficient, I decided to test it today and give you a feed back.

As I was installing/running and writing the reply, I thought it would be all good and that you could close this one. Maybe not so fast. Although it can still be novice mistake on my part.
The install went just great.
The run, not quite. I tried both a simple code and the Example from the package, and both fail at the end.
I then run with -log debug, but I don't see how to upload the resulting files to give you some hints.

I might be missing something.

• Any metadata I should use/force?
• Should I run the from ipypublish.ipynb_latex_setup import * cell? It still gives me errors if I do:

ImportErrorTraceback (most recent call last)
<ipython-input-1-6f0c5a516ba9> in <module>()
----> 1 from ipypublish.ipynb_latex_setup import *
ImportError: No module named ipynb_latex_setup

Install

$ pip install ipypublish
Collecting ipypublish
  Downloading ipypublish-0.4.1-py2.py3-none-any.whl (73kB)
    100% |████████████████████████████████| 81kB 761kB/s 
Requirement already satisfied: jinja2 in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: nbconvert in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: tornado in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: bibtexparser in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: pathlib in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: traitlets in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: nbformat in ./anaconda2/lib/python2.7/site-packages (from ipypublish)
Requirement already satisfied: MarkupSafe>=0.23 in ./anaconda2/lib/python2.7/site-packages (from jinja2->ipypublish)
Requirement already satisfied: mistune!=0.6 in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: entrypoints>=0.2.2 in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: pandocfilters>=1.4.1 in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: pygments in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: jupyter-core in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: bleach in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: testpath in ./anaconda2/lib/python2.7/site-packages (from nbconvert->ipypublish)
Requirement already satisfied: configparser>=3.5; python_version == "2.7" in ./anaconda2/lib/python2.7/site-packages (from entrypoints>=0.2.2->nbconvert->ipypublish)
Requirement already satisfied: html5lib>=0.99999999 in ./anaconda2/lib/python2.7/site-packages (from bleach->nbconvert->ipypublish)
Requirement already satisfied: six in ./anaconda2/lib/python2.7/site-packages (from bleach->nbconvert->ipypublish)
Requirement already satisfied: setuptools>=18.5 in ./anaconda2/lib/python2.7/site-packages (from html5lib>=0.99999999->bleach->nbconvert->ipypublish)
Requirement already satisfied: webencodings in ./anaconda2/lib/python2.7/site-packages (from html5lib>=0.99999999->bleach->nbconvert->ipypublish)
Installing collected packages: ipypublish
Successfully installed ipypublish-0.4.1

$ conda list | grep ipypublish
ipypublish                0.4.1                     <pip>

$ pip list | grep ipypublish
ipypublish (0.4.1)

Test

$ nbpublish -pdf Untitled.ipynb 
INFO:main:started ipypublish at Tue Jul 18 21:02:31 2017
INFO:main:logging to: /Users/user/Documents/Untitled.nbpub.log
INFO:main:running for ipynb(s) at: Untitled.ipynb
INFO:main:with conversion: latex_ipypublish_main
INFO:nbmerge:Reading notebook
INFO:main:getting output format from exporter plugin
INFO:nbexport:running nbconvert
INFO:latex_doc:resolving external file paths in latex_doc metadata to: Untitled.ipynb
INFO:main:outputting converted file to: /Users/user/Documents/Untitled.tex
INFO:main:dumping external files to: /Users/user/Documents/Untitled_files
INFO:main:running pdf conversion
INFO:pdfexport:running: latexmk -bibtex -pdf --interaction=batchmode <outpath>
INFO:pdfexport:latexmk: Latexmk: This is Latexmk, John Collins, 19 Jan. 2017, version: 4.52c.
INFO:pdfexport:latexmk: Rule 'pdflatex': Rules & subrules not known to be previously run:
INFO:pdfexport:latexmk: pdflatex
INFO:pdfexport:latexmk: Rule 'pdflatex': The following rules & subrules became out-of-date:
INFO:pdfexport:latexmk: 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Run number 1 of rule 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Running 'pdflatex  --interaction=batchmode -recorder  "/Users/user/Documents/Untitled.tex"'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Latexmk: applying rule 'pdflatex'...
INFO:pdfexport:latexmk: This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
INFO:pdfexport:latexmk: restricted \write18 enabled.
INFO:pdfexport:latexmk: entering extended mode
INFO:pdfexport:latexmk: Latexmk: References changed.
INFO:pdfexport:latexmk: Latexmk: Log file says no output from latex
INFO:pdfexport:latexmk: Latexmk: For rule 'pdflatex', no output was made
INFO:pdfexport:latexmk: Rule 'pdflatex': File changes, etc:
INFO:pdfexport:latexmk: Changed files, or newly in use since previous run(s):
INFO:pdfexport:latexmk: 'Untitled.aux'
INFO:pdfexport:latexmk: 'Untitled.out'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Run number 2 of rule 'pdflatex'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: Running 'pdflatex  --interaction=batchmode -recorder  "/Users/user/Documents/Untitled.tex"'
INFO:pdfexport:latexmk: ------------
INFO:pdfexport:latexmk: === TeX engine is 'pdfTeX'
INFO:pdfexport:latexmk: Latexmk: applying rule 'pdflatex'...
INFO:pdfexport:latexmk: This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017) (preloaded format=pdflatex)
INFO:pdfexport:latexmk: restricted \write18 enabled.
INFO:pdfexport:latexmk: entering extended mode
INFO:pdfexport:latexmk: Latexmk: Log file says no output from latex
INFO:pdfexport:latexmk: Latexmk: For rule 'pdflatex', no output was made
INFO:pdfexport:latexmk: Failure to make 'Untitled.pdf'
INFO:pdfexport:latexmk: Latexmk: Use the -f option to force complete processing,
INFO:pdfexport:latexmk: unless error was exceeding maximum runs of latex/pdflatex.
INFO:pdfexport:latexmk: === TeX engine is 'pdfTeX'
INFO:pdfexport:latexmk: Latexmk: Errors, so I did not complete making targets
ERROR:pdfexport:pdf conversion failed: Try running with pdf_debug=True
ERROR:main:pdf export returned false, try running with pdf_debug=True
Traceback (most recent call last):
  File "/Users/user/anaconda2/bin/nbpublish", line 131, in <module>
    nbpublish(filepath, **options)   
  File "/Users/user/anaconda2/bin/nbpublish", line 64, in nbpublish
    create_pdf=create_pdf, pdf_in_temp=pdf_in_temp, pdf_debug=pdf_debug) 
  File "/Users/user/anaconda2/lib/python2.7/site-packages/ipypublish/main.py", line 148, in publish
    raise RuntimeError('the pdf export failed, try running with pdf_debug=True')
RuntimeError: the pdf export failed, try running with pdf_debug=True

Untitled.ipynb

[1] (Metadata first)
{
  "init_cell": true,
  "slideshow": {
    "slide_type": "skip"
  },
  "trusted": true
}
# - - - - - - - - -
from ipypublish.ipynb_latex_setup import *

[2]
for i in range(10):
    print(i)

2nd run

Success, BUT no .pdf

I excluded the slide part on the metadata.

• Do I need to force or edit the metadata for every cell?

$ nbpublish -pdf Untitled.ipynb 
INFO:main:started ipypublish at Tue Jul 18 21:55:16 2017
INFO:main:logging to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/Untitled.nbpub.log
INFO:main:running for ipynb(s) at: Untitled.ipynb
INFO:main:with conversion: latex_ipypublish_main
INFO:nbmerge:Reading notebook
INFO:main:getting output format from exporter plugin
INFO:nbexport:running nbconvert
INFO:latex_doc:resolving external file paths in latex_doc metadata to: Untitled.ipynb
INFO:main:outputting converted file to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/Untitled.tex
INFO:main:dumping external files to: /Users/angeloklin/Documents/Immersive/JLL/Transaction Analytics/code/converted/Untitled_files
INFO:main:running pdf conversion
INFO:pdfexport:running: latexmk -bibtex -pdf --interaction=batchmode <outpath>
INFO:pdfexport:latexmk: Latexmk: This is Latexmk, John Collins, 19 Jan. 2017, version: 4.52c.
INFO:pdfexport:latexmk: Latexmk: All targets (Untitled.pdf) are up-to-date
INFO:pdfexport:pdf conversion complete
INFO:main:process finished successfully

Config

$ which latexmk
/Library/TeX/texbin/latexmk

$ latexmk --version
Latexmk, John Collins, 19 Jan. 2017. Version 4.52c

$ pandoc --version
pandoc 1.19.2.1
Compiled with pandoc-types 1.17.0.5, texmath 0.9, skylighting 0.1.1.4

[chrisjsewell]

Cheers for the feedbck, ok lets see if we can work this out, so:

1. it should be from ipypublish.scripts.ipynb_latex_setup import *
   (the documentation has recently changed to say this but it looks like I didn't update the example, sorry). This import doesn't have anything to do with the actual conversion (which doesn't actually run any of the notebook code), it just sets up some convenience functions and packages like matplotlib to output better looking graphics, if your starting a new notebook (try making a plot before and after running the import).
2. There is a separate debug flag for pdf (--pdf-debug), try running nbpublish -pdf --pdf-debug Untitled.ipynb and see what it says
3. The default formatter in nbpublish (latex_ipypublish_main) only outputs input markdown cells as default, then input code cells (and their output) must have relevant metadata tags to export (this is geared towards making an article where you want to pick exactly what goes in it and don't generally want to show all the code). To output everything by default run with the latex_standard_article: nbpublish -f latex_standard_article -pdf --pdf-debug Untitled.ipynb. (running nbpublish -h shows all the currently available output formats at the bottom)
4. For the 2nd run, because you still have all the intermediate output files from the 1st run in the converted folder, latexmk has looked at them, decided there is no changes in the output and so has not to run the conversion (INFO:pdfexport:latexmk: Latexmk: All targets (Untitled.pdf) are up-to-date)

Give that a go and let me know ta

[angeloklin]

Hello Chris,

Thanks again for the reply.
I will test it tomorrow as it is getting late around here.
If the debug show something interesting I will let you know. //

I think I saw the hint for the --pdf-debug in the output and tried it, but it failed.
Checking the message there is this bit of code:

File "/Users/user/anaconda2/lib/python2.7/site-packages/ipypublish/main.py", line 148, in publish
    raise RuntimeError('the pdf export failed, try running with pdf_debug=True')
RuntimeError: the pdf export failed, try running with pdf_debug=True

[chrisjsewell]

yeh no worries, if you delete the converted folder, then run with --pdf-debug, it will literally pause the program at the point of error, or just run latexmk -bibtex -pdf Example.tex directly. I'm guessing that maybe you are missing one of the tex styles, did you install mactex?

[angeloklin]

Hello Chris,
I was hoping that your suspicion was right, but tex is not missing any styles.
I added a markdown and a matplotlib chart cell, but only the markdown comes out in the pdf. The chart does not.
I run the Example and it works just fine on a quick visual inspection.
It feels like nbconvert is missing or not doing something. I noticed that I had a copy of nbconvert from anaconda (5.1.1) and ipypublish installed a new one (pip 5.2.1).
If I run with the -f latex_standard_article it runs fine, aside from some minor formatting issues.

In any case, I think you could close the ticket as it was opened in regards to installation, and that part is fixed. (Thanks!)

conda / pip environments (FYI)

$ conda list | grep -E "(nb|pub)"
ipypublish                0.4.1                     <pip>
nb_anacondacloud          1.2.0                    py27_0  
nb_conda                  2.0.0                    py27_0  
nb_conda_kernels          2.0.0                    py27_0  
nbconvert                 5.1.1                    py27_0  
nbconvert                 5.2.1                     <pip>
nbformat                  4.3.0                     <pip>
nbformat                  4.3.0                    py27_0  
nbpresent                 3.0.2                    py27_0  
widgetsnbextension        2.0.0                    py27_0  

$ pip list | grep -E "(nb|pub)"
ipypublish (0.4.1)
nb-anacondacloud (1.2.0)
nb-conda (2.0.0)
nb-conda-kernels (2.0.0)
nbconvert (5.2.1)
nbformat (4.3.0)
nbpresent (3.0.2)
widgetsnbextension (2.0.0)

[chrisjsewell]

Hey again, yeh thanks for this, its really good to get the perspective of someone who is not as 'close' to the project as me. Obviously there is aspects of the installation/running that I take for granted and don't realise need more explanation/coding. So, to close this issue (and move discussion to #27), can I confirm that:

1. There is now no issues with following the installation steps
2. After following the installation steps and downloading the example from github, you can convert the Example.ipynb (using nbpublish -pdf examples/notebooks/Example.ipynb ) with no errors.
3. If there is anything that could be improved upon in the installation/documentation for this process

Obviously it would be nice to have it as a conda package (as well as pip). I just haven't got round to figuring out how this can be automated with the integration testing (i.e. for the pip package, when I create a new release in Github, it automatically runs all the tests (with Travis CI) and, if these pass, it uploads it to Pypi). Including installation of MacTex/TexLive in the conda package may be more hassle than its worth though, since I would then have to have different releases for different operating systems/versions (and the relevant tests to make sure they were working).

[angeloklin]

Hello Chris,
I think your summary is correct for this ticket.
As a suggestion, MacTex is getting better but is far from being a simple thing for most users. Put as a requirement in the documentation, but leave for people to fight that by themselves. :)

Conda manages/understands pip well. Maybe just investigate if would be possible to avoid the redundancy of nbconvert/nbformat unless it is a concrete requirement.

Anyway, the installation after the setup.cfg was uploaded just worked. So all good.

Thanks again.
PS: I will hit the Close and comment, if it does not close, you can close as Author.