yoavram / markx

Markdown editor for scientific writing. Batteries included.

Markx

Markdown editor for scientific writing. Batteries included.

Why Markx?

Markdown has recently become popular among academics as a way to produce scientific documents. When paired with document conversion engines such as pandoc, it provides an easy and powerful way to write. Being a simple plain-text markup language, Markdown is easy to learn, and can be handled by version control systems. We decided to build Markx as to make markdown writing a collaborative effort that could be hosted as web service. For additional discussion, see these blog posts:

• Markdown and the future of collaborative academic writing
• Thoughts on a preprint server
• How to ditch Word

Features

1. Free as in free speech - promotes open science, pre-publication review, collaboration.

2. Free as in free beer - uses open-source, free software and released under CC-NC-BY 3.0.

3. Easy to learn - Markdown is not LaTex, for better and worse.

4. Integration with GitHub - free hosted version control, ideal for backup, versioning and collaboration.

5. Pandoc integration - multiple Markdown extensions, multiple output formats.

6. Real-time Markdown preview - WYSIWYM / WYSIWYG.

7. Display math with LaTeX formatting - $x^2+y^2=r^2$.

8. Citations - BibTeX: [@Drake1991]

9. Code highlighting:

    if __name__ == '__main__':
        # Bind to PORT if defined, otherwise default to 5000.
        port = int(os.environ.get('PORT', 5000))
        app.run(host='0.0.0.0', port=port, debug=app.debug)
   

10. Local host option with a limited number of dependencies - for offline editing on the train and in the airplane

11. Remote host option at http://markx.herokuapp.com/ - no installation required

12. By scientists, for scientists

Development Status

Markx is currently in development but can be used locally (see Installation instructions below) or remotely at http://markx.herokuapp.com.

It is still experimental, so take care of your data!

We would love to get feedback from anyone using it - Please open an issue with any bug or suggestion. If you can't open an issue please contact Yoav on twitter (you can use the #markx hashtag).

Technology

• Server side (this is what you need to install if you run it on localhost):
  • Python with Flask and requests
  • Pandoc - optional, for conversion to PDF, DOCX etc.
• Client side (no installation required):
  • PageDown - Stack Overflow's Markdown editor
  • Google Code Prettifier - Code highlighting
  • MathJax - Rendering of $LaTeX$ equations
  • BibTeX-js - Processing BibTeX citations
  • Twitter Bootstrap with [Glyphicons Free]
  • Github.js - integration with GitHub
• 3rd Party services:
  • GitHub - version control hosting
  • Docverter - online conversion of Markdown to multiple formats (currently, citations are not supported)
  • LaTeX-Online - online compilation of LaTeX to PDF

Install locally

1. Clone this repository
2. Install Python - developed with version 2.7
3. Install Flask and requests: pip install Flask requests
4. Optionally install Pandoc
5. Run Markx by calling python server.py and pointing your browser at http://localhost:5000.
6. Load a BibTeX file with your citation library by choosing Load a Bibliography File from the Bibliography menu

More help can be found in the help file.

License

markx's Issues

Pandoc and PDFs

Check if pdflatex exists, if not convert to PDF with the online tool.

Just checking whether this is still an active project?

I LOVE this idea and I REALLY dont understand why it has not become the de facto project to get scholarly markdown developed and adopted.

I dont write javascript but do python a lot. I would love to help if I could.

Gus

Convert using Pandoc is not working

Commit e168dd2

The button to toggle from pandoc to docverter seems fixed in docverter thus pandoc output is not generated.

KaTeX support

It would be really nice if you could get some KaTeX support, for instant fast making!

PDF to XML converter

One great thing to enhance scholarly writing would be to convert this to semantic markup. This tool http://pdfx.cs.man.ac.uk/ might be super handy for us because we could first export to PDF, then programmatically convert to xml. I'll leave it here as a placeholder.

no mathjax result in the exported files

As we all know that math formula support by mathjax is just js effect. So if someone export a pdf or docx file from the button, we cannot see any math formula result there.

I will consider it a bug. Hope some smart developer guys can fix it.

And at least there should be some warning.

PS: I found a tricky way, in case anyone need it. Click "change display", and print the page using chrome. It's a "better than never" way to export a "right" pdf file.

img tag is not supported by PageDown

I realized that if I have img tag in my document for displaying images, the preview using PageDown is not coming up properly. I am forced to use ![image](http://url/to/image.png) to display the image.

Exporting to odt

Hi i wondered if it was possible to add an option to export the document in the ODT-Fileformat?
Seems kind of logical to add this if there is an option for docx.

Thanx

Size of editor

As the text in the editor accumulates, the preview pane grows, but the editor stays at the same size.

Although the editor text area can be re-sized manually, it would be nice if it automatically grew together with the preview panel.

Problem with BibTeX file configuration

I don't understand the step 4 of the readme. I don't know what the BibTeX .bib file is.
So i get this error:

$ python server.py 
 * Running in debug mode
Traceback (most recent call last):
  File "server.py", line 35, in <module>
    bib = bibi.parse_file(app.config['BIB_FILE'])
  File "/home/cristobal/markx/bibi.py", line 6, in parse_file
    fin = open(filename)
IOError: [Errno 2] No such file or directory: '/path/to/bib/file'

Obviously i should replace /path/to/bib/file with the right path.

Citation features

Currently the citation feature is the conversion of BibTeX citation keys - [@Drake1998] - in the text to a bibliography below the preview pane. This feature is activated by a Citations button.

Additional citations features:

• Conversion of the keys in the editor panel to citations in the preview panel. For example, converting [@Drake1998] to (Drake et al., 1998).
• Tooltip on the converted citations in the preview panel showing the full citation.
• Citation Style Language support, possibly via citeproc-js.
• Warning message when a citation key was not found - above the preview panel? by highlighting the unconverted key in the preview pane? highlighting the key in the editor?

Reload BibTeX file

There is no OK button in the file choice modal, the behavior is that a change of file loads the file, but if you want to reload it this forces you to open the file dialog, choose a different file, open it again... OR reload the entire page.
We need a Reload button, and maybe also a OK button.

File buttons

• Change file name after upload
• Save after uploading
• Allow loading from the files already saved
• PDF/DOCX/EPUB generation should do 'update citations' and 'save .bib' before running
• Feedback when clicking Save .md and Save .bib

A button to add references

It would be nice a button in the edit bar to add the reference skeleton [@ ]

\\(..\\) and \\[..\\] not supported in LaTeX export?

This might belong under #43 but that seems to discuss HTML/PDF exports.

In https://markx.herokuapp.com/ preview both $, $$ math and \\(, \\[ math gets rendered (but not \(, \[). So far good, as documented.

- dollar mode $e=mc^2$
- NON Inline mode \(e=mc^2\)
- Inline mode \\(e=mc^2\\)
- double dollar mode: $$\frac{df(x)}{dt}=lim_{x \to 0}{\frac{f(x+h)-f(x)}{h}}$$
- NON Display mode: \[\frac{df(x)}{dt}=lim_{x \to 0}{\frac{f(x+h)-f(x)}{h}}\]
- Display mode: \\[\frac{df(x)}{dt}=lim_{x \to 0}{\frac{f(x+h)-f(x)}{h}}\\]

But when I download exported LaTeX, both in D and P modes, I'm seeing only dollars forms exported as math; all backslash forms are treated as text (this portion exactly same in D and P outputs):

\item
  dollar mode $e=mc^2$
\item
  NON Inline mode (e=mc\^{}2)
\item
  Inline mode \textbackslash{}(e=mc\^{}2\textbackslash{})
\item
  double dollar mode:
  \[\frac{df(x)}{dt}=lim_{x \to 0}{\frac{f(x+h)-f(x)}{h}}\]
\item
  NON Display mode: {[}\frac{df(x)}{dt}=lim\_\{x
  \to 0\}\{\frac{f(x+h)-f(x)}{h}\}{]}
\item
  Display mode: \textbackslash{}{[}\frac{df(x)}{dt}=lim\_\{x
  \to 0\}\{\frac{f(x+h)-f(x)}{h}\}\textbackslash{}{]}

For P mode it should be enough to pass -f markdown+tex_math_double_backslash to pandoc.

Docverter's markdown only alludes to math support but doesn't document the exact syntax. It doesn't seem to pass tex_math_single_backslash nor tex_math_double_backslash to pandoc so I guess it uses pandoc default which is only dollars...

GitHub button

GitHub button on GitHub toolbar functionality - I was thinking of a modal with some help and some links to GitHub - commit log for this branch, issues, etc.
This is the last task from the issue #13 which was closed.

github favoured markdown?

Is it possible to support gfm?

So I can write codes without (stupid) indent, and I can generate a <br/> more conveniently.

Why Marx

We should have some explanation on why we are developing Markx. Maybe some references to @karthikram 's blog posts?

pdf to xml

Loading bib file also from github

You can load the manuscript from github but you may need also the bib file that should be on github too. Also can be assumed that the bib file has the same name as the manuscript with bib extension.

Independent position of the right panel

Editing a long document the edit panel and the preview panel get shifted, so if you want to check your changes you have to scroll down, if something is wrong you have to scroll up, edit, then scroll down again an so on. It would be nice if the right and left panels keep independent positions.

Change HTML title

When loading a repo/branch/file change the title of the HTML to represent the loaded resource

Conversion to pdf not working in markx

Conversion to pdf in http://markx.herokuapp.com is not working either with pandoc or with docverter, Loading the bibliography or not gives the same error.

When the P/D button is pressed the browser address change to

http://latex.aslushnikov.com/compile?url=undefined/view/markx.latex

Error: fetching url undefined/view/markx.latex
Failed to fetch document

This is what I found in the error console:
"Fecha: 22/02/13 14:33:06
Error: La codificación de caracteres del doucmento de texto plano no fue declarada. El documento se verá con basura en el texto en algunas configuraciones de navegadores si contiene caracteres fuera del rango US-ASCII. La codificación de caracteres necesita ser declarada en el protocolo de transferencia o el archivo necesita usar una marca de orden de byte como firma de codificación.
Archivo Fuente: http://latex.aslushnikov.com/compile?url=undefined/view/markx.latex
Línea: 0

In the case of local markx either with P or D the pdf is generated under /files but is not downloader by the browser.

I am using firefox 19.0 under linux ubuntu 12.04

Syntax Highlight Only First Code Block

Currently, only the first code block is syntax highlighted.

This is because the current preprocessing of code blocks does not use the global flag for JS string replacing... the following code fixes the issue (markx.js):

function processGooglePrettifierPreBlocks(text) {
    text = text.replace(new RegExp("<pre><code>", "g"), '<pre class="prettyprint">');
    text = text.replace(new RegExp("\n</code></pre>", "g"), '\n</pre>');
    return text;
}

private git repositories are not listed

If I sign in to my GitHub account, only public repositories can be selected/opened.

Private repositories are commonly used by our group to share paper drafts. Hence, it would be great to have access to them as well.

Markdown conversion

Consider using Docverter - see example and links here https://gist.github.com/4356539.
This will bypass installing and wrapping Pandoc.

General TODO

This is an issue for feature features

• table of contents option
• Help/About page
• Interface to Mendeley?
• Python/R interfcace (knitr/Rmd/ipython notebook)?
• Replace Python/Flask with node.js?

Syntax highlighting in code blocks

When using the web page it highlights the syntax of the first code block, but it doesn't in any of the subsequent ones. This happens in a Windows 7 machine (32 bit), using Mozilla Firefox 20.01.

The following is an example:

First code block (nice colors):

plot(3:5, main = "Some text in greenish color")

Second code block (no colors):

plot(2:4, main = "Everything is just black")

Word count

Add a word count and/or line count and/or character count.
There are some jqeury plugins for it.

server.py error in commit 53a0757d5c5f7c29142b237e2a14e934f8c0b11f

I did a pull to actualize markx and then run the server:

leonardo@lypMini:~/markx$ python server.py

Traceback (most recent call last):
 File "server.py", line 11, in <module> 
 import requests
 ImportError: No module named requests

Textarea is covered when the viewport is under 1079px width

In firefox using a 1024x760 resolution the div id="right-panel" covers textarea id="wmd-input-second".
Using firefox responsive design view I found that the problem is present under 1079 pixels width.

use marked instead of pagedown

Similar to ipython you could switch from pagedown to marked (https://github.com/chjj/marked) as later supports gfm and tables (which can be processed by pandoc) out of the box.

Would solve a lot of these missing feature "bugs".
Anyway, really nice tool!

cheers

Conversion with pandoc is not working after loading bibliography

In the local version works, in the hosted version don't. Tested with chromium 24 and firefox 19.0

Fix editor preview ratio

I got my bib issues resolved (I like the link to view online). One thing that would be great is to alter the ratio of the editor/preview so neither of them feel constrained. One way might be to use tabs to separate the two (so they could be wider, say 10 column boostrap each).

Herko & GitHub integration

I deployed Markx on Heroku (http://markx.herokuapp.com/) but there is a problem - GitHub integration doesn't work, I get this message on the JS console:

XMLHttpRequest cannot load https://api.github.com/users/yoavram/repos?type=all&per_page=1000&sort=updated&1360445203928. Origin http://markx.herokuapp.com is not allowed by Access-Control-Allow-Origin. 

There is already an open issue on Github.js (github-tools/github#35) about this.
There is also a closed issue (github-tools/github#8) with some hints on how to solve this.

the preview panel is not updated

The rendered markdown or preview panel does not change until you edit something, that's ok, but when you pull and load a new document it should be automatically updated.

how to run in suburl instead of root (/)

How can I configure server.py to provide all resources below a suburl instead of web root (/)?

I'm running markx on a separate server behind an Apache Reverse Proxy and at the suburl domain.tld/marx. The html is correctly returned, but no grafics and style sheets.

Scholarly Markdown

1. What about figures? Committing them as well? Probably need a button to upload images.
2. What about tables? Perhaps committing CSV files? Or importing CSV to the markdown?
3. Equations are easy with LaTeX and MathJax, but what about numbered equations? Any clue how these work?

BibTeX

Currently we use a bibtex file that resides on the server machine and a python script bibi.py that reads from it and creates a dict.
On the client, the PageDown converter, once finished converting, calls a js function that searches for citation keys ([@Drake1998]) and when found queries the server using AJAX for the citation key (via the URL /bibtex).
The server returns an answer with the full bibtex entry for the citation key.
The client js then adds the citation entry to a hidden textarea.
The plugin bibtex-js is called, and it processes the entries in the textarea, formats them to a bibliography, and puts the rendered HTML in a div element in the preview panel.

Missing features:

• Upload bibtex files
• Using other citation formats, such as RIS, EndNote, XML

One alternative to the above workflow could be using Bib2x

DCSV integration

If using git, we can use GitPython for the python side API.

Pandocs title block

Pandoc's title block is really nice and works good when converting to DOCX and PDF, and even HTML if using the -s option.

But it is not supported by the PageDown Markdown-to-HTML converter we are currently using.

This can be supported by adding a preConversion hook (see PageDown docs) that checks for a title block and replaces it with Markdown syntax (# / ## ?) or a postConverion hook that replaces it with HTML syntax (<h1> / <h2>?)

For an example of a Markdown document with a title block see this gist: https://gist.github.com/4305736/

working offline with math

I was trying to use markx offline, math is not parsed, can the MathJax script be included in local code?

\\ for linebreak in LaTeX math environment (not \\\\)

To break a line in a LaTeX environment I need to use \\\\ instead of \\.

I guess here \ plays a role of escape character. However, it is inconsistent with other places in formulas where \ does not need to be escaped.

E.g.:

\begin{align}
c_{ij} &= - \langle \psi | \sigma^x_i \sigma^x_j  | \psi \rangle \\\\
\sum_{i \neq j } c_{ij} &= 1
\end{align}

works but

\begin{align}
c_{ij} &= - \langle \psi | \sigma^x_i \sigma^x_j  | \psi \rangle \\
\sum_{i \neq j } c_{ij} &= 1
\end{align}

not (i.e. there is no line break).

Highlight.js

Highlight.js is a much nicer, and lighter weight syntax highlighter. It has multiple css options (instead of the current single dark one) that we could allow users to set on the options.

automerge on commit

If someone else has changed a file in GitHub, markx should merge these changes on commit automatically.

Currently, it either gives an error message or overrides the last commit(s). The later shouldn't be the case.

Github sign out

When clicking the sign out button verify that the user really wants to sign out.

Use git branches to allow private/independent edits

Just a thought: Sometimes you wish to send a manuscript for feedback and don't want editors to view each other's comments/edits immediately, so you can compare independent feedback. I suggest that the git interface allow this by a "private share" button that creates a new branch and sends the editor a link to the editor on that branch.

Warning: Fonts transferred with MIME type application/octet-stream

In the Chrome Developer Tools log I saw this warning for all .woff files:
Resource interpreted as Font but transferred with MIME type application/octet-stream: "http://localhost:5000/markx/static/js/mathjax/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff".

It's probably not an issue.

no safety window on clicking markx icon

When loading a new file the "safety" window asking if you really want to overwrite/remove/lose current work is a nice feature. Unfortunately the same doesn't happen if you click on the markx logo in the top left, and your work is lost.

Pandoc flavored Markdown superscript and subscript are not displayed in the html preview

Superscript (^letter^) and subscript (letter) as defined by Pandocs Markdown flavour http://johnmacfarlane.net/pandoc/README.html#superscripts-and-subscripts are not displayed in the HTML preview.