fletcher / multimarkdown-6

Lightweight markup processor to produce HTML, LaTeX, and more.

About

Title:	libMultiMarkdown
Author:	Fletcher T. Penney
Date:	2023-06-10
Copyright:	Copyright © 2016 - 2023 Fletcher T. Penney.
Version:	6.7.0

master branch:
develop branch:

An Announcement!

MultiMarkdown v6 is finally here! If you want more information about testing, see DevelopmentNotes.

If you want to more know about the differences from v5, see the QuickStart guide.

Obtaining MultiMarkdown

You can download the latest installer for MacOS or Windows at Github:

https://github.com/fletcher/MultiMarkdown-6/releases

To build from source, download from Github. Then:

make release
(OR)
make debug

cd build
make

You can optionally test using the test suite:

ctest

Xcode

In order to use libMultiMarkdown in your Xcode project:

1. cd into the root of your Xcode project folder (where the .xcodeproj file resides).

2. Add this project as a git submodule:

   git submodule add https://github.com/fletcher/MultiMarkdown-6 MultiMarkdown-6
   

3. Compile:

   cd MultiMarkdown-6
   make xcode
   

4. Drag the build-xcode/MultiMarkdown.xcodeproj file to the root of your Xcode project as a subproject.

5. Select the MultiMarkdown subproject, select the libMultiMarkdown target, and in Build Phases > Copy Files select Products Directory from the Destination popup menu.

6. Select your root project, select your target, add libMultiMarkdown under Target Depencies and libMultiMarkdown.framework/libMultiMarkdown and libcurl.tdb under Link Binary with Libraries.

Warning: if you move the project on disk or update the MultiMarkdown source files, you need to rerun step 3 above.

You can now #import <libMultiMarkdown/libMultiMarkdown.h>. To get you started, here is (untested) demonstration code that converts a NSString to HTML:

token_pool_init(); // needs to be done once per app lifecycle - PLEASE READ token.h!!!!!!!!!!!

NSString *input = @"Test *string* for **demonstration**.";
const char *cString = [input cStringUsingEncoding:NSUTF8StringEncoding];
const char *mmd = mmd_string_convert(cString, EXT_SMART | EXT_NOTES | EXT_CRITIC | EXT_TRANSCLUDE, FORMAT_HTML, ENGLISH);
NSString *output = [[NSString alloc] initWithCString:mmd encoding:NSUTF8StringEncoding];

// Cleanup
free(mmd); 
token_pool_drain();		// again, PLEASE READ token.h!!!!!!
token_pool_free();

There are 3 main versions of the primary functions:

• mmd_string...: start from source text in c string
• mmd_d_string...: start from a DString (Useful if you already use DString's for your text)
• mmd_engine...: useful when you are processing the same source multiple times

The C string variants are as follows:

// Convert OPML string to MMD
DString * mmd_string_convert_opml_to_text(const char * source);

// Convert ITMZ string to MMD
DString * mmd_string_convert_itmz_to_text(const char * source);

// Convert MMD text to specified format, with specified extensions, and language
// Returned char * must be freed
char * mmd_string_convert(const char * source, unsigned long extensions, short format, short language);

// Convert MMD text to specified format using DString as a container for block of data
// and length of that block.  Must be used for "complex" output formats such as EPUB.
// Returned DString * must be freed
DString * mmd_string_convert_to_data(const char * source, unsigned long extensions, short format, short language, const char * directory);

// Convert MMD text and write results to specified file -- used for "complex" output formats requiring
// multiple documents (e.g. EPUB)
void mmd_string_convert_to_file(const char * source, unsigned long extensions, short format, short language, const char * directory, const char * filepath);

// Does the text have metadata?
bool mmd_string_has_metadata(char * source, size_t * end);

// Return metadata keys, one per line
// Returned char * must be freed
char * mmd_string_metadata_keys(char * source);

// Extract desired metadata as string value
// Returned char * must be freed
char * mmd_string_metavalue_for_key(char * source, const char * key);

// Insert/replace metadata in string, returning new string
char * mmd_string_update_metavalue_for_key(const char * source, const char * key, const char * value);

// Grab list of all transcluded files, but we need to know directory to search,
// as well as the path to the file
// Returned stack needs to be freed
struct stack * mmd_string_transclusion_manifest(const char * source, const char * search_path, const char * source_path);

The following enums can be used for the parameters language, format and extensions:

enum smart_quotes_language {
	ENGLISH = 0,
	DUTCH,
	FRENCH,
	GERMAN,
	GERMANGUILL,
	SPANISH,
	SWEDISH,
};

enum output_format {
	FORMAT_HTML,
	FORMAT_EPUB,
	FORMAT_LATEX,
	FORMAT_BEAMER,
	FORMAT_MEMOIR,
	FORMAT_FODT,
	FORMAT_ODT,
	FORMAT_TEXTBUNDLE,
	FORMAT_TEXTBUNDLE_COMPRESSED,
	FORMAT_OPML,
	FORMAT_ITMZ,
	FORMAT_MMD,
};

enum parser_extensions {
	EXT_COMPATIBILITY       = 1 << 0,    //!< Markdown compatibility mode
	EXT_COMPLETE            = 1 << 1,    //!< Create complete document
	EXT_SNIPPET             = 1 << 2,    //!< Create snippet only
	EXT_SMART               = 1 << 3,    //!< Enable Smart quotes
	EXT_NOTES               = 1 << 4,    //!< Enable Footnotes
	EXT_NO_LABELS           = 1 << 5,    //!< Don't add anchors to headers, etc.
	EXT_PROCESS_HTML        = 1 << 6,    //!< Process Markdown inside HTML
	EXT_NO_METADATA         = 1 << 7,    //!< Don't parse Metadata
	EXT_OBFUSCATE           = 1 << 8,    //!< Mask email addresses
	EXT_CRITIC              = 1 << 9,    //!< Critic Markup Support
	EXT_CRITIC_ACCEPT       = 1 << 10,   //!< Accept all proposed changes
	EXT_CRITIC_REJECT       = 1 << 11,   //!< Reject all proposed changes
	EXT_RANDOM_FOOT         = 1 << 12,   //!< Use random numbers for footnote links
	EXT_TRANSCLUDE          = 1 << 13,   //!< Perform transclusion(s)
	EXT_PARSE_OPML          = 1 << 14,   //!< Convert from OPML before processing source text
	EXT_PARSE_ITMZ			= 1 << 15,   //!< Convert from ITMZ (iThoughts) before processing source text
	EXT_RANDOM_LABELS		= 1 << 16,   //!< Use random numbers for header labels (unless manually defined)
	EXT_FAKE                = 1 << 31,   //!< 31 is highest number allowed
};

Differences in the MultiMarkdown Syntax

MultiMarkdown v6 is mostly about making a better MMD parser, but it involves a few changes to the MultiMarkdown syntax itself.

1. Setext headers can consist of more than one line to be included in the header:

   This is
   a header
   ========
   

2. Whitespace is not allowed between the text brackets and label brackets in reference links, images, footnotes, etc. For example [foo] [bar] will no longer be the same as [foo][bar].

3. Link and image titles can be quoted with 'foo', "foo", or (foo). Link attributes can be used in both reference and inline links/images.

4. HTML elements are handled slightly differently. There is no longer a markdown="1" feature. Instead, HTML elements that are on a line by themselves will open an HTML block that will cause the rest of the "paragraph" to be treated as HTML such that Markdown will not be parsed in side of it. HTML block-level tags are even "stronger" at starting an HTML block. It is not quite as complex as the approach used in CommonMark, but is similar under most circumstances. Leaving a blank line after the opening tag will allow MultiMarkdown parsing inside of the HTML block.

   For example, this would not be parsed:

   <div>
   *foo*
   </div>
   

   But this would be:

   <div>
   
   *foo*
   
   </div>
   

5. "Malformed" reference link definitions are handled slightly differently. For example, the test suite file Reference Footnotes.text is parsed differently in compatibility mode than MMD-5. This started as a side-effect of the parsing algorithm, but I actually think it makes sense. This may or may not change in the future.

6. Table captions in MMD-6 must come immediately after the table, not before it.

7. Escaped linebreaks (\ preceding a line break) will be interpreted as <br /> (even in compatibility mode). This was previously an optional feature in MMD, but I don't see a problem with just making it default behavior.

8. Escaped spaces (\ ) will be interpreted as a non-breaking space, if the output format supports it.

9. CriticMarkup, Abbreviations, Glossary Terms, and Citations are handled slightly differently. See the QuickStart guide for more information.

10. Fenced code blocks can use leading/trailing "fences" of 3, 4, or 5 backticks in length. That should be sufficient for complex documents without requiring a more complex parser. If there is no trailing fence, then the fenced block is considered to go through the end of the document.

11. Emph and Strong parsing is conceptually the same, but the implementation is different. It is designed for speed, accuracy, and consistency. In general, it seems to handle edge cases much more reliably, but there are still a couple of situations that I would like to take into account, if possible. These are not situations that should occur often in "real life."

12. EPUB 3 output is supported without need of any external tools.

13. Internationalization support for HTML phrases, such as "see footnote". See Github for more information.

License

The `MultiMarkdown 6` project is released under the MIT License..

GLibFacade.c and GLibFacade.h are from the MultiMarkdown v4 project:

	https://github.com/fletcher/MultiMarkdown-4/

MMD 4 is released under both the MIT License and GPL.


CuTest is released under the zlib/libpng license. See CuTest.c for the
text of the license.

uthash library:
	Copyright (c) 2005-2016, Troy D. Hanson

	Licensed under Revised BSD license

miniz library:
	Copyright 2013-2014 RAD Game Tools and Valve Software
	Copyright 2010-2014 Rich Geldreich and Tenacious Software LLC

	Licensed under the MIT license

argtable3 library:
	Copyright (C) 1998-2001,2003-2011,2013 Stewart Heitmann
	<[email protected]>
	All rights reserved.

	Licensed under the Revised BSD License


## The MIT License ##

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


## Revised BSD License ##

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.
    * Redistributions in binary form must reproduce the above
      copyright notice, this list of conditions and the following
      disclaimer in the documentation and/or other materials provided
      with the distribution.
    * Neither the name of the <organization> nor the
      names of its contributors may be used to endorse or promote
      products derived from this software without specific prior
      written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL <COPYRIGHT
HOLDER> BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE, DATA, OR
PROFITS OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

multimarkdown-6's Issues

Support memoir output

Go back through code to fix any memory leaks

ctest - 3/8 Test #3: mmd-6-latex ......................***Failed

Hi Fletcher,

I am kind of new in this topic., but I could compile successfully on Fedora 25. Still ctest
fails on mmd-6-latex test.

I had installed previously:
glibc-static
glibc-utils
Then, after test failure
3/8 Test #3: mmd-6-latex ......................***Failed

I installed
latex-mk
latex2html
latexmk

Any thing to be worried about. Thx for this endeavour.

ODF Style "Source 20 Text" needs to be left justified

Review logic around `print_localized` function -- may be obsolete now

Support LyX output?

Fix issue with HTML comments with empty line

An HTML comment containing an empty line is not treated as a comment, but the initial line(s) is/are still treated as HTML for block parsing. Doesn't affect HTML output, but does affect LaTeX and others.

CriticMarkup and blank lines

CriticMarkup is a powerful concept, but does
require a bit of thought.

1. The syntax is orthogonal to that of HTML, and therefore
   Markdown/MultiMarkdown. It should really be thought of as a separate "layer"
   of markup -- one that needs to be processed before parsing the underlying
   MultiMarkdown.

2. This leads to additional complexity, as many users would prefer to use
   CriticMarkup as an additional feature of MultiMarkdown, that allows displaying
   the added/deleted/etc. text in the HTML output(using <ins>, <del>, etc.).
   In this instance, CriticMarkup needs to be carefully wrapped around, or
   inside, the underlying Markdown -- not "crossing the streams". For example:

    This is {++**good**++}.
   
    This is **{++not**++}.
   

3. Similarly, CriticMarkup becomes difficult to process when used around
   newlines. For example, using CriticMarkup to describe the act of splitting an
   existing paragraph into two new paragraphs is tricky, as it requires that MMD
   "understands" that the opening {++ in the first paragraph connects to the
   closing ++} in the second paragraph, which violates the normal rules of
   Markdown:

    This is one para.{++
   
    ++} That is now two.
   

4. An alternative could be to require that such CriticMarkup start and end
   with the opening/closing tags on their own separate lines. This could allow
   the MMD parser to recognize a CriticMarkup "chunk" and treat it as it's own
   block:

    This is one para.
    {++
   
    ++}
    That is now two.
   

5. Another alternative is to create a separate CriticMarkup parsing step that
   finds all CriticMarkup and handles accepting or rejecting it, prior to the
   regular MMD parsing. This would be similar to what happened in MMD version 5.

For now, CriticMarkup doesn't play well "across" paragraphs in MMD. I will
consider ways of addressing this in the future, and welcome input!

Latex Config file

This is a requested enhancement.

I am super excited about the new way MMD6 is dealing with Latex config files. I was accomplishing this in a very convoluted way before this. I am just not sure people understand (I didn't for a while) that one can organize everything inside subdirectories. I was just wondering if whenever you release MMD6, you could include a README.md and organize everything in subfolders to the benefits of those trying to get a hang of MMD2TEX. That way one can see a subdirectory that has "article" "letter" "handout" and keep everything more compartmentalized.

Create Windows installer

support fenced code block language specifiers

HTML Images with width or height in units other than pixels must be style

RTF export?

If someone is interested in providing support for an RTF export format, I'm happy to include it in the main repo. It's not something I need, and the RTF format is frustrating to work with.

The recommended way to convert MultiMarkdown into RTF is via the ODF format, and use LibreOffice to convert to RTF. It produces much better results!

Create OS X Installer

Support manual labels for headers

support subtitle metadata

Rewrite URL matching code to use re2c scanner instead of token parsing

Support LaTeX output

Update from 5.4 to 6.0 broke Scrivener MMD -> Latex output

Under 6.0, Scrivener is no longer generating the start of its .tex file correctly.

Correct version produced under 5.4:

\input{mmd-scrivcustom-header}
\def\mytitle{My Great Novel}
\def\myauthor{Me}
\input{mmd-scrivcustom-begin-doc}

6.0 is picking up the \defs from the Scrivener metadata, but omitting the \inputs.

Downgrading back to 5.4 solved the problem.

I am not sure I would call this an issue, but a question about Latex metadata

I have been using MMD for several years and I have created a series of templates for LATEX that create reports, letters, books, etc that I use at least once a week. I noticed that now one can select a latex config file by doing the following,

LaTeX Config: tufte-handout

But I cannot find how that is being changed into what I see on the Latex documents created by MMD6. Where is that latex config taking place? I am not sure what I am missing. I am very interested in understanding this process since this is very important for me.

Add glossary and abbreviation to integrated test file

Create linux installer

Consider approach for CriticMarkup blocks similar to that used for HTML to allow empty lines?

Reconsider MMD/LaTeX metadata approach

MMD-6 metadata for LaTeX is likely to work slightly differently. Currently it is broken up into three distinct pieces -- latex leader (which is inserted before all metadata), latex begin (which comes after metadata and before the body of the document), and latex footer (which comes after the body).

This could be configured such that a single metadata field (or command line argument) can be used to specify the filenames that are included. The latex files would simply have to be named in a consistent manner. For example, if the files are named mmd-beamer-leader, mmd-beamer-begin-doc, and mmd-beamer-footer, they could be automatically configured with the string beamer.

Additionally, the metadata latex header allows custom information to be inserted on a per document basis.

I welcome other thoughts on this as I try to improve beyond the old approach.

Add support for random footnote numbers

This was a feature in MMD 5 that needs to be completed for v6

New syntax for raw source

Since the early days of MMD, HTML comments could be used to pass raw LaTeX (or whatever) to the exported file. This was a bit of a hack, and while useful, it's time to rethink it.

I think the best approach now would be a new syntax that allows passing raw text to the output, based on which output format is selected. This would allow customizing the output based on the chosen format. This can already be done using wildcard transclusion, but in that case the text would still be processed as MultiMarkdown.

Support for abbreviations not implemented

Abbreviation support is not available yet. I'm not sure the best way to implement it, as it will need a slightly different approach than in MMD-5

Content of code blocks with characters <,>,&,",' is structurally wrong and not escaped

Before I start to open more issues, is it OK to file bugs at this early stage or should I wait for a stable release?

    ````
    <p a="a" b="'" c="&" d="'">lorem</p>
    ````

gives

<pre><code></code></pre>

<p a="a" b="'" c="&" d="'">lorem</p>

<pre><code>


</code></pre>

Proposal for quotation marks

Assuming that quotes language will also be implemented in MMD a similar problem with translations exists. Correct quotation marks seem to be quite diverse, as can be found here. Implementing all the different styles is maybe a bit too much for MMD so I propose a similar approach like in #37 :

mmd-lq-primary:        „
mmd-rq-primary:        “
mmd-lq-secondary:      ‚
mmd-rq-secondary:      ‘

Respectively:

lang:                     fr
mmd-lq-fr-primary:        « 
mmd-rq-fr-primary:         »
mmd-lq-fr-secondary:      “
mmd-rq-fr-secondary:      ”
mmd-lq-de-primary:        „
mmd-rq-de-primary:        “
mmd-lq-de-secondary:      ‚
mmd-rq-de-secondary:      ‘

In the example above MMD would use the mmd-lq-fr-*and mmd-rq-fr-* metadata entries to generate the quotation marks. But maybe like in #37 it's a bit out of scope since it could also be achieved with transclusion.

What isn't clearly visible in the example above (but it is really in the code) is that the value of mmd-lq-fr-primary is followed by a half-em non-breaking space (U+202F,  ) and the value of mmd-lq-fr-primary starts with the same space as suggested here. So MMD should not treat these spaces as ordinary whitespace.

Support mmd output format

The mmd format is used to output the modified source, after performing transclusion

Add support for tables/images as targets for cross-reference links

MMD localization

For the most part, the language used in MultiMarkdown documents don't matter. Whatever language you write in is the same text used in the output document.

An exception to this, however, is that there are a few phrases used in the HTML source, such as:

return to body
see footnote
see citation
see glossary

MMD-6 includes the i18n.h file to allow automatic translation of these phrases to other languages, without a performance hit.

The default language is English. Support is built in for German (missing 1 translation). Template strings are in place for Spanish.

Additional languages can be added easily.

1. kNumberOfLanguages needs to be incremented.
2. lc_lookup() needs to be modified -- a new language should be added to that function following the template for English/Spanish/German. The order is CRUCIALLY IMPORTANT.
3. lc_languages() needs to be updated. The language added must be at the end of the list (in other words, the order must match that of lc_lookup(). The two letter code should be the official one for the new language.
4. i18n_language_from_string() needs to be updated for the new language as well, using the lower case two letter code. Again, follow the template for German and Spanish.

Adding a new string is a bit more complicated, and should not generally need to be done. The TranslateTest() function is useful when doing so, however.

If you want to add/improve the translation functionality, please submit a pull request with the proposed changes. (Please use the develop branch for pull requests so I can properly merge and test.)

Thanks!!

Support citation "locators"

Support beamer output

Add support for file transclusion

File transclusion not implemented yet. Will need to consider best way to implement to minimize performance impact and multiple parsing passes.

Support ODF output

Support OPML output

Support keywords metadata

multimarkdown -l <language> doesn't write lang attribute of tag <html>

As the title says, no matter what is given on the commandline the document always contains <html xmlns="http://www.w3.org/1999/xhtml" lang="en">.

A further observation: In MMD 5 it was possible to define the language of the document in metadata with, for example lang: de. This seems to be ignored now.

"Not cited" syntax for citations

Glossaries not implemented

Need to implement glossary support.

Reimplement escaped syntax for non-breaking space

Support smart quotes in other languages

Support Epub Output

Proper support for epub with equivalents to Latex functionality like citations, footnotes, glossaries, abbreviations, tables etc. would make multimarkdown instantly the go to tool for epub creation. (at least for a rough version of a document).
With modern systems the need for LaTeX is declining rapidly while the demand for resizable modern documents rises all the time (think about all the iPads and similar devices)
PDF is a nice document format for printing but it is not really suitable for digital content consumption, especially since nobody uses adobe reader (which is the only tool that actually supports all pdf features).
With epub support one could easily create nice looking documents especially for scientific content. e.g. math via mathjax, code highlighting without issues etc. etc.
I believe mmd already does some basic epub output but a few LaTeX equivalents should be handled too, so that mmd's features get translated into a nice looking default.

Include mmd-6 latex support scripts

Add utility scripts

Revisit parsing approach and thread safety

Ensure that mmd_engine structs are properly used without leaving left over data? Or are these one shot structures?

Also have to revisit token object pool's, as those will not be thread safe.

Are table headers required now?

Both

|------|------|------|
| cell | cell | cell |
| cell | cell | cell |

and

|      |      |      |
|------|------|------|
| cell | cell | cell |
| cell | cell | cell |

no longer produce a table. In MMD 5 they did.

Update homebrew formula

Rethink MMD/LaTeX citations

Over the years there have been various requests to change the way citations work with MMD and LaTeX/BibTeX/natbib/etc. This includes things related to citep, citet, etc.

I don't claim to remember all of the proposals, so now is your chance to bring them up again and argue your case!