aaren / notedown

Markdown <=> IPython Notebook

Python 2/3 and IPython 4 / Jupyter compatible!

Convert IPython Notebooks to markdown (and back)

notedown is a simple tool to create IPython notebooks from markdown (and r-markdown).

notedown separates your markdown into code and not code. Code blocks (fenced or indented) go into input cells, everything else goes into markdown cells.

Usage:

notedown input.md > output.ipynb

Installation:

pip install notedown

or the latest on github:

pip install https://github.com/aaren/notedown/tarball/master

Conversion to markdown

Convert a notebook into markdown, stripping all outputs:

notedown input.ipynb --to markdown --strip > output.md

Convert a notebook into markdown, with output JSON intact:

notedown input.ipynb --to markdown > output_with_outputs.md

The outputs are placed as JSON in a code-block immediately after the corresponding input code-block. notedown understands this convention as well, so it is possible to convert this markdown-with-json back into a notebook.

This means it is possible to edit markdown, convert to notebook, play around a bit and convert back to markdown.

NB: currently, notebook and cell metadata is not preserved in the conversion.

Strip the output cells from markdown:

notedown with_output_cells.md --to markdown --strip > no_output_cells.md

Running an IPython Notebook

notedown notebook.md --run > executed_notebook.ipynb

Editing in the browser (new!)

You can configure IPython / Jupyter to seamlessly use markdown as its storage format. Add the following to your config file:

c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'

Now you can edit your markdown files in the browser, execute code, create plots - all stored in markdown!

For Jupyter, your config file is jupyter_notebook_config.py in ~/.jupyter. For IPython your config is ipython_notebook_config.py in your ipython profile (probably ~/.ipython/profile_default):

Editing in vim

There is a vim plugin that allows editing notebooks (ipynb files) directly in vim. They will be automatically converted to markdown on opening the file, and converted back to the original json format on writing.

R-markdown

You can use notedown to convert r-markdown as well. We just need to tell notedown to use knitr to convert the r-markdown. This requires that you have R installed with knitr.

Convert r-markdown into markdown:

notedown input.Rmd --to markdown --knit > output.md

Convert r-markdown into an IPython notebook:

notedown input.Rmd --knit > output.ipynb

• --rmagic will add %load_ext rpy2.ipython at the start of the notebook, allowing you to execute code cells using the rmagic extension (requires rpy2). notedown does the appropriate %R cell magic automatically.

Magic

Fenced code blocks annotated with a language other than python are read into cells using IPython's %% cell magic.

You can disable this with --nomagic.

• --pre lets you add arbitrary code to the start of the notebook. e.g. notedown file.md --pre '%matplotlib inline' 'import numpy as np'

How do I put a literal code block in my markdown?

By using the --match argument. notedown defaults to converting all code-blocks into code-cells. This behaviour can be changed by giving a different argument to --match:

• --match=all: convert all code blocks (the default)

• --match=fenced: only convert fenced code blocks

• --match=language: only convert fenced code blocks with 'language' as the syntax specifier (or any member of the block attributes)

• --match=strict: only convert code blocks with Pandoc style attributes containing 'python' and 'input' as classes. i.e. code blocks must look like

    ```{.python .input}
    code
    ```
  

This isn't very interactive!

Try editing the markdown in the IPython Notebook using the NotedownContentsManager (see above).

You can get an interactive ipython session in vim by using vim-ipython, which allows you to connect to a running ipython kernel. You can send code from vim to ipython and get code completion from the running kernel. Try it!

Where's my syntax highlighting?!

Try using either vim-markdown or vim-pandoc. Both are clever enough to highlight code in markdown.

Rendering outputs in markdown

This is experimental!

Convert a notebook into markdown, rendering cell outputs as native markdown elements:

notedown input.ipynb --render

This means that e.g. png outputs become ![](data-uri) images and that text is placed in the document.

Of course, you can use this in conjuntion with runipy to produce markdown-with-code-and-figures from markdown-with-code:

notedown input.md --run --render > output.md

Not a notebook in sight!

The --render flag forces the output format to markdown.

TODO

• Python 3 support
• unicode support
• IPython 3 support
• IPython 4 (Jupyter) support
• Allow kernel specification

notedown's Issues

Option --match not-fenced

It would be nice to have a "--match indented". In other words, the opposite of "--match fenced".

Benefits are:

1. Easy to indicate syntax of non-executable code (using fencing) so IPYN syntax highlighting can work.
2. Easy to distinguish executable vs non-executable code.

Broken urls when converting ipynb to markdown

I have a ipynb (here: https://github.com/HiSPARC/infopakket/blob/d15cde4747efe42b129482c1a70c00ee47788bd3/notebooks/3_zenith_afhankelijkheid.ipynb ) which contains a few links. When converted to markdown using notedown (v1.5) the links are broken when the markdown is viewed on GitHub or when using the QLMarkdown Quick Look generator. In those cases the links contain the new line characters which are in the link because they are broken over multiple lines by notedown. See the output markdown here (from a slightly modified ipynb): https://github.com/HiSPARC/infopakket/blob/6d7803cbed7a6c2190ba671ed0d181d83053e7cc/notebooks/3_zenith_afhankelijkheid.md

add overwrite option

The -o or --output flag is used to specify an output filename.

When supplied in the absence of a filename this should specify overwriting or
changing the file extension, depending on the output type:

notedown test.md -o                 --> test.ipynb
notedown test.md --to notebook -o   --> test.ipynb
notedown test.md --to markdown -o   --> test.md

Encoding error when loading notebook written in markdown in Jupyter

Hi,

I've created a markdown document jose.md containing some accented characters like:

## Sobre mí

José de España.

When I tried to load it in Jupyter after activating markdown format by using this:

printf "c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'" >> ~/.jupyter/jupyter_notebook_config.py

I get an encoding error:

Unreadable Notebook: /home/researcher/wwc/_friends/jose.md UnicodeDecodeError('ascii', b'---\ntitle: Jos\xc3\xa9\n---\n\n## Sobre m\xc3\xad\n\nJos\xc3\xa9 de Espa\xc3\xb1\n', 14, 15, 'ordinal not in range(128)')

However, I've I try to create a new notebook within Jupyter, and I add the same text in a markdown cell does not happing anything.

I get a similar error when trying to add accented characters in an already existing and running markdown notebook, if I try to save.

Unexpected error while saving file: wwc/_friends/jose5.md 'ascii' codec can't encode character '\xe9' In position 218: ordinal not in range(128)

I think it is something that has to do with notedown. Since I can add UTF-8 characters in markdown cells in native jupyter notebooks. And run/load/save without any issue.

Any hints?

converting from notebook or markdown to Rmarkdown

Hi Aaron,
thank you for this wonderful package. Is it planed or would be possible to get the option of converting markdown files or notebooks to Rmarkdown? I think that for those (like me) how use RStudio as editor would be very helpful to be able to convert from a notebook or markdown file to Rmarkdown ;-)
Cheers,
jmm

IPython.html.widgets` has moved to `ipywidgets`.", ShimWarning

Please updata this package.
error: ShimWarning: The IPython.html package has been deprecated. You should import from notebook instead. IPython.html.widgets has moved to ipywidgets.
"IPython.html.widgets has moved to ipywidgets.", ShimWarning)

Figure embedding

We can write in markdown and convert to a notebook.

We can convert a notebook to markdown (exported markdown).

Notebooks can cache computed output (figures...).

Exported markdown stores output figures as markdown image links
to an outputs directory.

We would like to be able to do a round trip through notedown and
ipython, where the original markdown is preserved.

Workflow

• Write a notebook in markdown.
• Use notedown to convert to ipython notebook
• Play around in web console. Change code, embed figures.
• Convert back to markdown. Figures are now embedded.
• Edit more. Convert to ipython notebook. Embedded figures are
  pulled in to the notebook (if using inline matplotlib).

If you're using vim-ipython:

• Write notebook in markdown.
• Execute contents of a cell.
• Resulting output gets embedded in the markdown.
• Notedown to ipython notebook at any point and the result is just
  as if you had created and executed the cells in the web notebook.

How to see the ipython notebook equivalent of the markdown:

either

• Use pandoc to convert markdown to html and view this in a browser.
• Notedown to ipynb periodically and view the ipynb with a local
  nbviewer server.
• Notedown to ipynb periodically and view the ipynb as an
  interactive web notebook that reads directly from disk - is this
  possible??

Ideas

New ideas:

1. Make the output directory available on github so that we can link
   directly on the web (as raw links to github).
2. Allow inclusion of stream output in the markdown. i.e. include
   the png base64 in the markdown.
3. Create notebooks with output. i.e. for a markdown file with
   code blocks and image links, put the code block as the input and
   the source of the image link as the output.

Notebooks with output

Need to specify what is output in the markdown. Ideas:

• first image after code block
• first image without newline
• anything inside an output tag

An output tag could be something like

```python
![image](path/to/image)
```

problem with using a fenced code blocks is what happens when we
pandoc the markdown?

Convert link extensions

I just tried notedown for converting google's tensorflow docs from .md to .ipynb.

It worked great.

The only thing that could have been nicer is the inks between the .md files, I had to reprocess the files so that the notebooks linked to each other.

It could be nice if there was a command line option that swapped the link extension during conversion.

But I'm happy either way. Thanks.

wrap text on markdown output

Text is not wrapped when we output to markdown. Wrap it to 80 chars.

error after adding c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'

In order to read md file directly in Jupyter Notebook, I install notedown.

But after adding c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager' at the end of my config file 'jupyter_notebook_config.py'
or
using jupyter notebook --NotebookApp.contents_manager_class='notedown.NotedownContentsManager'

I get the following error:

Bad config encountered during initialization: The 'contents_manager_class' trait of a NotebookApp instance must be a subclass of 'notebook.services.contents.manager.ContentsManager', but a value of class 'traitlets.traitlets.MetaHasTraits' (i.e. <class 'notedown.contentsmanager.NotedownContentsManager'>) was specified.

Any idea what cause this error? When I delete jupyter notebook --NotebookApp.contents_manager_class='notedown.NotedownContentsManager' the notebook works fine.

match argument handling

I'm having trouble setting the match argument to a specific language. For example

notedown --nomagic --match=stata StataIntro.md > StataIntro.ipynb
Traceback (most recent call last):
  File "/home/izahn/.local/bin/notedown", line 11, in <module>
    sys.exit(app())
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/main.py", line 312, in app
    main(args, help=parser.format_help())
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/main.py", line 275, in main
    notebook = reader.read(ip, as_version=4)
  File "/usr/lib/python3.5/site-packages/nbformat/v4/rwbase.py", line 89, in read
    return self.reads(nbs, **kwargs)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 376, in reads
    return self.to_notebook(s, **kwargs)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 368, in to_notebook
    cells = self.create_cells(blocks)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 347, in create_cells
    markdown_cell = self.create_markdown_cell(block)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 323, in create_markdown_cell
    markdown_cell = nbbase.new_markdown_cell(**kwargs)
  File "/usr/lib/python3.5/site-packages/nbformat/v4/nbbase.py", line 112, in new_markdown_cell
    validate(cell, 'markdown_cell')
  File "/usr/lib/python3.5/site-packages/nbformat/v4/nbbase.py", line 23, in validate
    return validate(node, ref=ref, version=nbformat)
  File "/usr/lib/python3.5/site-packages/nbformat/validator.py", line 156, in validate
    raise better_validation_error(e, version, version_minor)
  File "/usr/lib/python3.5/site-packages/nbformat/validator.py", line 152, in validate
    return validator.validate(nbjson, {'$ref' : '#/definitions/%s' % ref})
  File "/usr/lib/python3.5/site-packages/jsonschema/validators.py", line 123, in validate
    raise error
jsonschema.exceptions.ValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['source']:
    {'oneOf': [{'type': 'string'},
               {'items': {'type': 'string'}, 'type': 'array'}]}

On instance['source']:
    None

Any help will be appreciated.

ShimWarning: The `IPython.html` package has been deprecated since IPython 4.0

I'm not sure if the warning in the subject is itself a problem, but I can't seem to run your example. I have a new installation of Anaconda with Python3 and have installed notedown by pip install https://github.com/aaren/notedown/tarball/master. Following is from the r-examples directory:

$ notedown r-example.Rmd --knit --rmagic > r-example-me.ipynb
/anaconda3/lib/python3.6/site-packages/IPython/html.py:14: ShimWarning: The `IPython.html` package has been deprecated since IPython 4.0. You should import from `notebook` instead. `IPython.html.widgets` has moved to `ipywidgets`.
  "`IPython.html.widgets` has moved to `ipywidgets`.", ShimWarning)
$ 

And I don't get a complete .ipynb output file:

$ cat r-example-me.ipynb 
{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction to ggplot2\n",
    "\n",
    "This is a short demo on how to convert an R Markdown Notebook into an IPython Notebook using knitr and notedown.\n",
    "\n",
    "Adding a Python Chunk"
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 2
}

prompt number not set if first cell has no output

Handling of indented fenced input/output code blocks (e.g. within an itemize)

Markdown supports fenced code blocks within itemize or other markdown construct denoted by indentation, as in the following example:

# Test file for notedown's handling of indented fenced code blocks

Those may typically occur within an itemize or so.

Normal fenced input/output code blocks:
```{.python .input}
1+1
```
```{.json .output}
[{"execution_count": 1, "output_type": "execute_result", "data": {"text/plain": "2"}, "metadata": {}}]
```

1.  Start of an itemize  
    Indented fenced input/output code blocks

    ```{.python .input}
    1+1
    ```
    ```{.json .output}
    [{"execution_count": 1, "output_type": "execute_result", "data": {"text/plain": "2"}, "metadata": {}}]
    ```

    Some more:

    ```{.python .input}
    1+1
    ```
    ```{.json .output}
    [{"execution_count": 1, "output_type": "execute_result", "data": {"text/plain": "2"}, "metadata": {}}]
    ```

1.  Next item (should be numbered 2.)

Would it be possible to have notedown support them as well? In particular, is there a way to translate such code blocks in the Jupyter notebook model?

Currently notedown gets confused and treat the the fenced code blocks and everything indented after as a single raw codeblock. I tried the different values of --match without success.

Convert to markdown

It might be useful to be able to convert a notebook back into markdown.

We can already do this using nbconvert:

ipython nbconvert notebook.ipynb --to markdown

There are two issues with this:

1. code cells are indented, no option for fence
2. outputs are converted to figures

Solution:

Provide a simple markdown template with notedown that allows

1. switching to fenced code blocks
2. stripping outputs

We then have a possible workflow of

edit markdown... 
convert to notebook... 
play with / edit notebook... 
convert to markdown... 
edit markdown

Basically, would let us roundtrip through the notebook format and preserve edits made to the text / code of notebooks.

Simple test:

notedown input.md > input.ipynb
notedown input.ipynb > output.md
diff input.md output.md

test strip outputs:

notedown input.md > input.ipynb
runipy input.ipynb output.ipynb
notedown output.ipynb > output.md
diff input.md output.md

support ipython 3 and nbformat v4

Presently notedown requires ipython 1 or 2 to work. Version 3 uses a new
notebook format (the v4 notebook) that I haven't got around to implementing.

Extract code from markdown file

Is there a way of extracting the code chunks directly from notedown?

--precode does not work with --match=python

$ notedown --match=python --precode='%matplotlib inline' lessons/0_images_are_arrays.md > book/lessons/0_images_are_arrays.ipynb
Traceback (most recent call last):
  File "/home/stefan/envs/py3/bin/notedown", line 11, in <module>
    sys.exit(app())
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/main.py", line 312, in app
    main(args, help=parser.format_help())
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/main.py", line 275, in main
    notebook = reader.read(ip, as_version=4)
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/nbformat/v4/rwbase.py", line 89, in read
    return self.reads(nbs, **kwargs)
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/notedown.py", line 376, in reads
    return self.to_notebook(s, **kwargs)
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/notedown.py", line 366, in to_notebook
    blocks = [self.process_code_block(block) for block in all_blocks]
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/notedown.py", line 366, in <listcomp>
    blocks = [self.process_code_block(block) for block in all_blocks]
  File "/home/stefan/envs/py3/lib/python3.5/site-packages/notedown/notedown.py", line 214, in process_code_block
    return self.new_text_block(content=block['raw'])
KeyError: 'raw'

$ notedown --match=python lessons/0_images_are_arrays.md > book/lessons/0_images_are_arrays.ipynb
$ notedown --precode='%matplotlib inline' lessons/0_images_are_arrays.md > book/lessons/0_images_are_arrays.ipynb

argument `--precode` causing display of help when `-o` is also specified

As the label says. Using a pipe instead of -o appears to work though.

Fenced code blocks: Compatibility with pandoc

When converting Rst to markdown, and writing a fenced code block,
pandoc insert a space between the triple quotes and the options:

mistral-/tmp>cat bla.rst 
.. code:: python

    1+1

mistral-/tmp>pandoc bla.rst -t markdown
``` {.sourceCode .python}
1+1
```

However, when converting this further to a notebook, notedown seems
to get confused by this additional space, and does not produce a block
as desired.

Sorry, that later part is not super specific. I'll provide a complete
example later on, but right now I broke notedown on my machine while
hacking around ...

Would it be possible to tweak notedown's regexp to tolerate this
additional space?

Btw: thanks a lot for notedown! That's going to be super useful in my
Rst -> notebook workflow!

How to select code chunks and Jupyter kernel

Suppose I have a multikernel markdown document (written with Stitch). Then I convert it from markdown to makdown and now have two kinds of code chunks: code chunks of the particular language that should be converted to jupyter code cells. And code chunks of other languages or dummy code chunks that should be inside jupyter markdown code cells.

How can I specify target language (it can be any jupyter kernel) and marks chunks that should be converted to jupyter code cells?

Note that stitch md -> md conversion makes all code chunks to be fenced.

Regex Help for Custom Code Blocks

I have a regex related question. I am interested in identifying code blocks that take the following form

```{r key = value}
some code
```

I was trying to modify your fenced_regex to account for the extra brace between the language, but was not successful. Any thoughts on what would be the easiest way to account for an option {?

Code attributes

Notebook code cells can contain metadata, which can be anything that
we can express as JSON.

Presently we just allow a single code attribute, the language, which
comes immediately after the fenced code block start:

```python
this is some(code)
```

In pandoc markdown we can have code attributes:

```{.python #identifier .thing .numberLines startfrom="100" blah=100}
this is some(code)
```

The idea is to take the code attributes and set them as metadata on
the code cell. Then you can think of the code attributes as a way to
edit cell metadata.

Issues:

1. Blindly pass all attributes as metadata, or sanitise them to
   known attributes? Should notedown do particular things with
   special attributes?
2. Format of the code attributes.

Formatting

Two considerations:

1. Not creating another markdown dialect
2. Rendering on github

Examples of different styles:

Regular

```python
for i in range(10):
    print i
```

for i in range(10):
    print i

Attributes 1

```{.python #mycode1 .thing .numberLines startfrom="100" blah=100}
for i in range(10):
    print i
```

for i in range(10):
    print i

Attributes 2

```{.python #mycode2}
for i in range(10):
    print i
```

for i in range(10):
    print i

Attributes 3

(not github)

```{#mycode3 .python}
for i in range(10):
    print i
```

for i in range(10):
    print i

Attributes 4

(not pandoc)

```{python #mycode4}
for i in range(10):
    print i
```

for i in range(10):
    print i

Attributes 5

(not pandoc)

```{python #mycode1 .thing .numberLines startfrom="100" blah=100}
for i in range(10):
    print i
```

for i in range(10):
    print i

r-markdown

(not pandoc)

```{r label, a=FALSE, b=2}
some(r(code))
```

some(r(code))

A solution to this problem would be to support all of these formats
and make it the authors problem what kind of markdown they make.
This lifts the responsibility of deciding a markdown type, but adds
a problem in that we have to support the syntax.

Personally, my favourite syntax is the r-markdown. However I feel
that something supported by pandoc is the right format to choose if
we don't support everything.

Notebook stored in markdown cannot be trusted

I'm using notedown 1.5.0 and jupyter 4.2.1

I have foo.md produced by notedown foo.ipynb --to markdown > foo.md. I open the file with jupyter notebook foo.md and it shows up fine, except one of my extensions (python-markdown) isn't working. I see a warning in the terminal "Notebook foo.md is not trusted". I select the "trust notebook" option in the menu, the file saves, and once again I see "Notebook foo.md is not trusted."

support of cell metadata in markdown

I am interested in supporting notebook cell metadata in the markdown file.This would have benefits:

• It would maintain metadata like slideshow information for cells and allow one to write a slide show in markdown originally.
• The would allow true round trip fidelity ipynb -> markdown -> ipynb

I am not sure about the best way to do this, but I can imagine abusing markdown links somewhat

each markdown cell would be allowed to have a "comment:"

[//]: # ( "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide"} } )
### The markdown content level three header
content plain text...

The code blocks would be handled as they are in notedown, but would add an optional metadata tag:

```{.python, "metadata" : {"collapsed": false, "slideshow": { "slide_type": "fragment"} } }
import numpy, pandas
y = 3 * 4
```

This may be an answer to #47
I am not married to the idea of using this particular system, but it would seem it would be ignored by a typical common-mark/markdown renderer and would not be too hard to implement.

Support the Jupyter (IPython) R kernel

Project Jupyter now supports a R kernel (https://github.com/IRkernel/IRkernel)
It would be nice to support this in translating Rmd to ipynb by removing the
%%R magics in the cells and then adding the R specific metadata and
language_info such as

"metadata": {
"kernelspec": {
"display_name": "R",
"language": "",
"name": "ir"
},
"language_info": {
"codemirror_mode": "r",
"file_extension": ".r",
"mimetype": "text/x-r-source",
"name": "R",
"pygments_lexer": "r",
"version": "3.1.3"
}

raise error if knitr not installed

Using --knit fails silently if knitr is not installed - the result is an empty notebook.

Ideally an error would be raised instead.

convert Rmd to ipynb with photo

I have example.Rmd and I use command
notedown example.Rmd --knit > example.ipynb but I have only file with this

{ "cells": [], "metadata": {}, "nbformat": 4, "nbformat_minor": 0 }
What the problem?

Writing to sys.stdout fails on Python 3

Just ran into this with @stefanv. This line:

writer.write(notebook, codecs.getwriter('utf-8')(sys.stdout))

aims to wrap sys.stdout in a unicode interface that will encode using UTF-8 before writing. But in Python 3, sys.stdout is already a unicode interface, so it encodes and tries to write bytes to something that's expecting unicode.

We have Python 2 and 3 compatible code for this in nbconvert - feel free to copy and paste our function:

https://github.com/jupyter/jupyter_nbconvert/blob/3a55808265c8b140cb483c374b5c6ccd464373d0/jupyter_nbconvert/utils/io.py#L12

You'd use this as:

writer.write(notebook, unicode_std_stream('stdout'))

Incorrect Conversion from md&ipynb

1. symbol.md

The contents in symbol.md (see attachment symbol.md) is like below. When I run "notedown symbol.md --to markdown --strip" (see chapter 1 and attachment symbol.md), then it becomes like chapter 2.

---

ex = c.eval(ctx = mx.cpu(), a = mx.nd.ones([2,3]), b = mx.nd.ones([2,3]))
print('number of outputs = %d\nthe first output = \n%s' % (
            len(ex), ex[0].asnumpy()))

For neural nets, a more commonly used pattern is simple_bind, which
creates all of the argument arrays for you. Then you can call forward,
and backward (if the gradient is needed) to get the gradient.

Load and Save

Logically symbols correspond to ndarrays. They both represent a tensor. They both
are inputs/outputs of operators. We can either serialize a Symbol object by
using pickle, or by using save and load methods directly as we discussed in
NDArray tutorial.

---

2. symbol_result.md

---

ex = c.eval(ctx = mx.cpu(), a = mx.nd.ones([2,3]), b = mx.nd.ones([2,3]))
print('number of outputs = %d\nthe first output = \n%s' % (
            len(ex), ex[0].asnumpy()))

For neural nets, a more commonly used pattern is simple_bind, which
creates all of the argument arrays for you. Then you can call forward,
and
backward (if the gradient is needed) to get the gradient.

2. Load and

Save

Logically symbols correspond to ndarrays. They both represent a tensor.
They both
are inputs/outputs of operators. We can either serialize a Symbol
object by
using pickle, or by using save and load methods directly as we
discussed in
[NDArray tutorial](http://mxnet.io/tutorials/basic/ndarray.html
#serialize-from-to-distributed-filesystems).

---

3. Attachment

symbol.zip

Support GitHub flavored markdown

GitHub supports, e.g., fenced codeblocks which are fantastically useful.

Make running notebooks more prominent in readme

@stefanv, notedown can't run notebooks itself but runipy can and can be chained with notedown in a single command.

I'll make this more prominent in the readme.

Code and Text blocks may not always be alternate

I expect

Code and Text block to be subsequent

Instead

The implementation forces code and text blocks to be alternate

Error in travis.yml: r-base-core (>= 3.2.5-1precise0) but it is not going to be installed

travis.yml can't install r-base-core

r-base-dev : Depends: r-base-core (>= 3.2.5-1precise0) but it is not going to be installed

Allow overriding of cell timeout setting

When executing a long-running cell, notedown times out.

notedown --match fenced --run markdown/ch7.markdown --output build_ipynb/ch7.ipynb
ERROR:root:Timeout waiting for execute reply
ERROR:root:failed to run cell: Empty()
ERROR:root:# dm6.fa.gz can be downloaded from ftp://hgdownload.cse.ucsc.edu/goldenPath/dm6/bigZips/
# Unzip before using: gzip -d dm6.fa.gz
dm = 'data/dm6.fa'
model = tz.pipe(dm, genome, markov)
Traceback (most recent call last):
  File "/Users/hdashnow/anaconda/envs/elegant/bin/notedown", line 9, in <module>
    load_entry_point('notedown==1.4.4', 'console_scripts', 'notedown')()
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/notedown/main.py", line 275, in app
    main(args, help=parser.format_help())
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/notedown/main.py", line 241, in main
    run(notebook)
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/notedown/notedown.py", line 43, in run
    notebook, resources = executor.preprocess(notebook, resources={})
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/IPython/nbconvert/preprocessors/execute.py", line 39, in preprocess
    nb, resources = super(ExecutePreprocessor, self).preprocess(nb, resources)
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/IPython/nbconvert/preprocessors/base.py", line 70, in preprocess
    nb.cells[index], resources = self.preprocess_cell(cell, resources, index)
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/IPython/nbconvert/preprocessors/execute.py", line 49, in preprocess_cell
    outputs = self.run_cell(cell)
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/IPython/nbconvert/preprocessors/execute.py", line 63, in run_cell
    msg = self.kc.shell_channel.get_msg(timeout=self.timeout)
  File "/Users/hdashnow/anaconda/envs/elegant/lib/python3.4/site-packages/IPython/kernel/blocking/channels.py", line 57, in get_msg
    raise Empty
queue.Empty

It looks like the ExecutePreprocessor has a settable timeout (defaulting to 30 seconds) but I don't understand how to set it...

Compatibility with ipython/jupyter 4.0

I am unable to get --render to work. This seems to be because of backward incompatible changes in ipython/jupyter.

lists with paragraphs

Related to #40, notedown now works with arbitrary language blocks, which is really great! I have noticed that the conversion in that case is more fragile however. For example, this markdown file (saved as example.md)

A markdown file
=================

This markdown file has lists with newlines between the items

-   Item 1
    -  item 1a

    -  item 1b

converts just fine with notedown example.md, but notedown --match=stata example.md produces

Traceback (most recent call last):
  File "/home/izahn/.local/bin/notedown", line 9, in <module>
    load_entry_point('notedown==1.5.0', 'console_scripts', 'notedown')()
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/main.py", line 312, in app
    main(args, help=parser.format_help())
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/main.py", line 275, in main
    notebook = reader.read(ip, as_version=4)
  File "/usr/lib/python3.5/site-packages/nbformat/v4/rwbase.py", line 89, in read
    return self.reads(nbs, **kwargs)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 380, in reads
    return self.to_notebook(s, **kwargs)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 372, in to_notebook
    cells = self.create_cells(blocks)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 351, in create_cells
    markdown_cell = self.create_markdown_cell(block)
  File "/home/izahn/.local/lib/python3.5/site-packages/notedown/notedown.py", line 326, in create_markdown_cell
    markdown_cell = nbbase.new_markdown_cell(**kwargs)
  File "/usr/lib/python3.5/site-packages/nbformat/v4/nbbase.py", line 112, in new_markdown_cell
    validate(cell, 'markdown_cell')
  File "/usr/lib/python3.5/site-packages/nbformat/v4/nbbase.py", line 23, in validate
    return validate(node, ref=ref, version=nbformat)
  File "/usr/lib/python3.5/site-packages/nbformat/validator.py", line 156, in validate
    raise better_validation_error(e, version, version_minor)
  File "/usr/lib/python3.5/site-packages/nbformat/validator.py", line 152, in validate
    return validator.validate(nbjson, {'$ref' : '#/definitions/%s' % ref})
  File "/usr/lib/python3.5/site-packages/jsonschema/validators.py", line 123, in validate
    raise error
jsonschema.exceptions.ValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['source']:
    {'oneOf': [{'type': 'string'},
               {'items': {'type': 'string'}, 'type': 'array'}]}

On instance['source']:
    None

optionally strip outputs when saving markdown notebook from ipython

Not sure of the ideal mechanism for setting this, but would be nice to be able to save a markdown without outputs from within ipython, allowing for seamless use of git without polluting repo with output noise (text encoded images etc).

Allow markdown -> ipynb -> Reveal.js slideshow

I want to write a presentation in vim, preferably using markdown.

I have tried converting notebooks to Reveal.js slideshows using jupyter nbconvert --to slides practical_python.ipynb.

This works great if the CellToolbar Slideshow is used and the Slide Type is specified for each notebook cell as appropriate.

This feature could be supported from markdown if slide type could somehow be annotated in the markdown files.

With this feature, I think this would make for an incredible presentation tool (markdown to both a notebook and slideshow presentations).

No unicode support

If I nbconvert on a markdown file that contains unicode, I obtain the following error:

Traceback (most recent call last):
  File "/Users/scott/anaconda/bin/notedown", line 11, in <module>
    sys.exit(cli())
  File "/Users/scott/anaconda/lib/python2.7/site-packages/notedown/main.py", line 261, in cli
    writer.write(notebook, op)
  File "/Users/scott/anaconda/lib/python2.7/site-packages/IPython/nbformat/__init__.py", line 161, in write
    fp.write(s)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in position 95: ordinal not in range(128)

I believe this is a simple fix (after you find the right place).

notedown: command not found

I'm on OSX 10.10.3 (Yosemite)

• Python version / location

~> python -V
Python 2.7.10 :: Anaconda 2.3.0 (x86_64)
~> which python
//anaconda/bin/python

• iPython version / location

~> which ipython
//anaconda/bin/ipython
~> ipython -V
3.2.0

• notedown

pip install note down

Appears to work fine, and installs at:

/anaconda/lib/python2.7/site-packages

import notedown

Works fine within ipython

But when I try to call notedown I get an error:

-bash: notedown: command not found

Any ideas on how to fix this? Deep thanks!

notedown doesn't work with Jupter Notebook 5.1.0

Whenever I run notedown it returns with this error

$ notedown 
Traceback (most recent call last):
  File "/home/adonese/anaconda3/bin/notedown", line 7, in <module>
    from notedown.main import app
  File "/home/adonese/anaconda3/lib/python3.5/site-packages/notedown/__init__.py", line 8, in <module>
    from .contentsmanager import NotedownContentsManager
  File "/home/adonese/anaconda3/lib/python3.5/site-packages/notedown/contentsmanager.py", line 8, in <module>
    from notebook.services.contents.filemanager import FileContentsManager
  File "/home/adonese/anaconda3/lib/python3.5/site-packages/notebook/services/contents/filemanager.py", line 21, in <module>
    from .manager import ContentsManager
  File "/home/adonese/anaconda3/lib/python3.5/site-packages/notebook/services/contents/manager.py", line 37, in <module>
    class ContentsManager(LoggingConfigurable):
  File "/home/adonese/anaconda3/lib/python3.5/site-packages/notebook/services/contents/manager.py", line 69, in ContentsManager
    untitled_notebook = Unicode(_("Untitled"), config=True,
NameError: name '_' is not defined

The same applied when I try to import it from python shell.

ipymd functionality: feedback wanted

@jni, @stefanv maybe you can test this out and give some feedback?

I've just pushed a change to master that lets you use markdown files as the on disk notebook storage, editing them with the web notebook. This was inspired by ipymd which lets you do a similar thing.

This isn't documented yet but I've released it with 1.4.4.

To try it out, add the following to ipython_notebook_config.py in your ipython profile (probably ~/.ipython/profile_default/):

c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'

and then try opening up, editing and saving some markdown in the browser.

Unlike ipymd, notedown preserves prompt numbers and outputs (using json code cells) in the markdown. This isn't very pretty but I get around it by collapsing json cells in my editor (and turning wrap off!).

Documentation for match argument misleading

--match=language: only convert fenced code blocks with 'language' as the syntax specifier (or any member of the block attributes)

I was confused by the above. It appears in a list where other options given are specified literally, but then this is a parameter--"language" should become "python", or other languages. As such, my Makefile contained notedown --match language, hoping it would catch any language and convert to the appropriate code block.

Cannot run .md file in jupyter: None not found.

Editing and running markdown files is a great idea!
Unfortunately for me it did not work out. When opening a markdown file in jupyter notebook a red box appears next to the 'Kernel' which says None not found. I am unable to manually restart or choose a different kernel.

Steps to reproduce:

1. I adjusted the configuration file as described to be able to open .md files in jupyter notebook. This works well.

2. I created a minimal test.md with the following content:

   ```{.python .input}              
   print('Hello World') 
   \``` 
   

(sorry, don't know how to escape the fences properly...)

3. I open this file in jupyter notebook (v4.2.1 on Anaconda python 3.5.2)

UnicodeDecodeError when directly opening *.md files downloaded from Dropbox Paper

I am teaching a programming course where we collaborate through Dropbox Paper. In order to get the lecture slides displayed more intuitively, I rely on Jupyter notebook to get me the execution cells. So far, notedown has been working well converting *.md files downloaded from Dropbox Paper.

When I got excited to learn that notedown can enable the handling of *.md* files directly, I was overjoyed and put the following line in my configuration file (c:\users\user_name.jupyter\jupyter_notebook_config.py`)

c.NotebookApp.contents_manager_class = 'notedown.NotedownContentsManager'

However, here is what I got:

1. Jupyer can now see the *.md files in the home page;
2. Yet, I shall see the following error message when trying to open such *.md through Jupyter.

---

Error loading notebook

Unreadable Notebook: c:\users\llinfeng\SI\GSI\bug.md UnicodeDecodeError('ascii', ',\n " \"The accumulation pattern isn\xe2\x80\x99t a new Python type, syntax, or concept. It\xe2\x80\x99s an approach to solve many different types of problems:\\n\",\n"', 45, 46, 'ordinal not in range(128)')

---

Here is the link to bug.md that I tried to open through Jupyter: https://www.dropbox.com/s/qqcczat5473eouf/20170127_bug.md?dl=0

Is there a converter that could help me solve this Unicode error?

Thank you!

All the best,

-Linfeng

Distinguish input / ouput based on >>>

(Following from /r/ipython)

It is common to use >>> at the start of a line in a code block to indicate a line entered into an interactive prompt. Lines without >>> are interpreted as output from the prompt. For example:

>>> a = 1
>>> b = 2
>>> a + b
3
>>> a * b
2

Presently, notedown does not interpret these sorts of code blocks intelligently. The entire block is used to create an input code cell. The desired behaviour would be for only the lines starting with >>> to be used in the code cell. Lines that don't start with >>> can either be ignored or put in an output cell.

I would lean towards ignoring lines that don't start with >>>, as including output is a big can of worms. For example, we have no way of knowing whether this is valid output, or how it will render in ipython.

Load Magic Extensions

This is an awesome tool and exactly what I was looking for as I try to work with R and IPython. I was able to successfully convert a markdown file into an IPython notebook, but one glitch I noticed was that I had to manually load the rmagic extension before running my notebook. Since, your code automatically detects magics, I was wondering if it was possible to auto load these magics?

%load_ext rmagic

unicode python 2 / R

Tests fail with python 2, R installed, knitr not installed:

(thesis)see4-206:~/code/notedown (master=)$ nosetests
................E
======================================================================
ERROR: Check that the R notebook generated from Rmd looks the same
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nfs/see-fs-02_users/eeaol/src/anaconda/envs/thesis/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/nfs/see-fs-02_users/eeaol/code/notedown/tests.py", line 346, in test_R
    knitr = notedown.Knitr()
  File "/nfs/see-fs-02_users/eeaol/code/notedown/notedown/notedown.py", line 569, in __init__
    stderr = stderr.decode()  # cast to unicode (python 3 compatible)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 166: ordinal not in range(128)

----------------------------------------------------------------------
Ran 17 tests in 3.268s

FAILED (errors=1)

Support for multi paragraph itemize / crash with --match=strict

With the following example, the second paragraph "Contrairement ..." gets interpreted by notedown as a code block instead of a second paragraph of the first item:

1.  Donner un exemple de drapeau infini décroissant; c'est-à-dire une suite infinie ... de sous-modules strictement emboîtés.

    Contrairement aux espaces vectoriels, deux sous-modules emboîtés de même dimension ne sont pas forcément égaux.

2.  Existe-t'il des drapeaux croissants infinis?

Since all my code blocks are fenced, I tried to work around this issue using match=strict, but this led to a crash:

mistral-/tmp>notedown --match=strict algebre_lineaire_anneaux.md >! algebre_lineaire_anneaux.ipynb
Traceback (most recent call last):
  File "/opt/miniconda3/lib/python3.6/site-packages/nbformat/validator.py", line 256, in validate
    return validator.validate(nbjson, {'$ref' : '#/definitions/%s' % ref})
  File "/opt/miniconda3/lib/python3.6/site-packages/jsonschema/validators.py", line 130, in validate
    raise error
jsonschema.exceptions.ValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in schema['properties']['source']:
    {'oneOf': [{'type': 'string'},
               {'items': {'type': 'string'}, 'type': 'array'}]}

On instance['source']:
    None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/miniconda3/bin/notedown", line 11, in <module>
    sys.exit(app())
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/main.py", line 312, in app
    main(args, help=parser.format_help())
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/main.py", line 275, in main
    notebook = reader.read(ip, as_version=4)
  File "/opt/miniconda3/lib/python3.6/site-packages/nbformat/v4/rwbase.py", line 114, in read
    return self.reads(nbs, **kwargs)
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/notedown.py", line 376, in reads
    return self.to_notebook(s, **kwargs)
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/notedown.py", line 368, in to_notebook
    cells = self.create_cells(blocks)
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/notedown.py", line 347, in create_cells
    markdown_cell = self.create_markdown_cell(block)
  File "/opt/miniconda3/lib/python3.6/site-packages/notedown/notedown.py", line 323, in create_markdown_cell
    markdown_cell = nbbase.new_markdown_cell(**kwargs)
  File "/opt/miniconda3/lib/python3.6/site-packages/nbformat/v4/nbbase.py", line 112, in new_markdown_cell
    validate(cell, 'markdown_cell')
  File "/opt/miniconda3/lib/python3.6/site-packages/nbformat/v4/nbbase.py", line 23, in validate
    return validate(node, ref=ref, version=nbformat)
  File "/opt/miniconda3/lib/python3.6/site-packages/nbformat/validator.py", line 260, in validate
    raise better_validation_error(e, version, version_minor)
nbformat.validator.NotebookValidationError: None is not valid under any of the given schemas

Failed validating 'oneOf' in notebook['properties']['source']:

On instance['source']:
None

Thanks!