mixmark-io / turndown

🛏 An HTML to Markdown converter written in JavaScript

Turndown

Convert HTML into Markdown with JavaScript.

Project Updates

• to-markdown has been renamed to Turndown. See the migration guide for details.
• Turndown repository has changed its URL to https://github.com/mixmark-io/turndown.

Installation

npm:

npm install turndown

Browser:

<script src="https://unpkg.com/turndown/dist/turndown.js"></script>

For usage with RequireJS, UMD versions are located in lib/turndown.umd.js (for Node.js) and lib/turndown.browser.umd.js for browser usage. These files are generated when the npm package is published. To generate them manually, clone this repo and run npm run build.

Usage

// For Node.js
var TurndownService = require('turndown')

var turndownService = new TurndownService()
var markdown = turndownService.turndown('<h1>Hello world!</h1>')

Turndown also accepts DOM nodes as input (either element nodes, document nodes, or document fragment nodes):

var markdown = turndownService.turndown(document.getElementById('content'))

Options

Options can be passed in to the constructor on instantiation. For example:

var turndownService = new TurndownService({ option: 'value' })

Option	Valid values	Default
headingStyle	setext or atx	setext
hr	Any Thematic break	* * *
bulletListMarker	-, +, or *	*
codeBlockStyle	indented or fenced	indented
fence	``` or ~~~	```
emDelimiter	_ or *	_
strongDelimiter	** or __	**
linkStyle	inlined or referenced	inlined
linkReferenceStyle	full, collapsed, or shortcut	full
preformattedCode	false or true	false

Advanced Options

Option	Valid values	Default
blankReplacement	rule replacement function	See Special Rules below
keepReplacement	rule replacement function	See Special Rules below
defaultReplacement	rule replacement function	See Special Rules below

Methods

addRule(key, rule)

The key parameter is a unique name for the rule for easy reference. Example:

turndownService.addRule('strikethrough', {
  filter: ['del', 's', 'strike'],
  replacement: function (content) {
    return '~' + content + '~'
  }
})

addRule returns the TurndownService instance for chaining.

See Extending with Rules below.

keep(filter)

Determines which elements are to be kept and rendered as HTML. By default, Turndown does not keep any elements. The filter parameter works like a rule filter (see section on filters belows). Example:

turndownService.keep(['del', 'ins'])
turndownService.turndown('<p>Hello <del>world</del><ins>World</ins></p>') // 'Hello <del>world</del><ins>World</ins>'

This will render <del> and <ins> elements as HTML when converted.

keep can be called multiple times, with the newly added keep filters taking precedence over older ones. Keep filters will be overridden by the standard CommonMark rules and any added rules. To keep elements that are normally handled by those rules, add a rule with the desired behaviour.

keep returns the TurndownService instance for chaining.

remove(filter)

Determines which elements are to be removed altogether i.e. converted to an empty string. By default, Turndown does not remove any elements. The filter parameter works like a rule filter (see section on filters belows). Example:

turndownService.remove('del')
turndownService.turndown('<p>Hello <del>world</del><ins>World</ins></p>') // 'Hello World'

This will remove <del> elements (and contents).

remove can be called multiple times, with the newly added remove filters taking precedence over older ones. Remove filters will be overridden by the keep filters, standard CommonMark rules, and any added rules. To remove elements that are normally handled by those rules, add a rule with the desired behaviour.

remove returns the TurndownService instance for chaining.

use(plugin|array)

Use a plugin, or an array of plugins. Example:

// Import plugins from turndown-plugin-gfm
var turndownPluginGfm = require('turndown-plugin-gfm')
var gfm = turndownPluginGfm.gfm
var tables = turndownPluginGfm.tables
var strikethrough = turndownPluginGfm.strikethrough

// Use the gfm plugin
turndownService.use(gfm)

// Use the table and strikethrough plugins only
turndownService.use([tables, strikethrough])

use returns the TurndownService instance for chaining.

See Plugins below.

Extending with Rules

Turndown can be extended by adding rules. A rule is a plain JavaScript object with filter and replacement properties. For example, the rule for converting <p> elements is as follows:

{
  filter: 'p',
  replacement: function (content) {
    return '\n\n' + content + '\n\n'
  }
}

The filter selects <p> elements, and the replacement function returns the <p> contents separated by two new lines.

filter String|Array|Function

The filter property determines whether or not an element should be replaced with the rule's replacement. DOM nodes can be selected simply using a tag name or an array of tag names:

• filter: 'p' will select <p> elements
• filter: ['em', 'i'] will select <em> or <i> elements

The tag names in the filter property are expected in lowercase, regardless of their form in the document.

Alternatively, the filter can be a function that returns a boolean depending on whether a given node should be replaced. The function is passed a DOM node as well as the TurndownService options. For example, the following rule selects <a> elements (with an href) when the linkStyle option is inlined:

filter: function (node, options) {
  return (
    options.linkStyle === 'inlined' &&
    node.nodeName === 'A' &&
    node.getAttribute('href')
  )
}

replacement Function

The replacement function determines how an element should be converted. It should return the Markdown string for a given node. The function is passed the node's content, the node itself, and the TurndownService options.

The following rule shows how <em> elements are converted:

rules.emphasis = {
  filter: ['em', 'i'],

  replacement: function (content, node, options) {
    return options.emDelimiter + content + options.emDelimiter
  }
}

Special Rules

Blank rule determines how to handle blank elements. It overrides every rule (even those added via addRule). A node is blank if it only contains whitespace, and it's not an <a>, <td>,<th> or a void element. Its behaviour can be customised using the blankReplacement option.

Keep rules determine how to handle the elements that should not be converted, i.e. rendered as HTML in the Markdown output. By default, no elements are kept. Block-level elements will be separated from surrounding content by blank lines. Its behaviour can be customised using the keepReplacement option.

Remove rules determine which elements to remove altogether. By default, no elements are removed.

Default rule handles nodes which are not recognised by any other rule. By default, it outputs the node's text content (separated by blank lines if it is a block-level element). Its behaviour can be customised with the defaultReplacement option.

Rule Precedence

Turndown iterates over the set of rules, and picks the first one that matches the filter. The following list describes the order of precedence:

1. Blank rule
2. Added rules (optional)
3. Commonmark rules
4. Keep rules
5. Remove rules
6. Default rule

Plugins

The plugin API provides a convenient way for developers to apply multiple extensions. A plugin is just a function that is called with the TurndownService instance.

Escaping Markdown Characters

Turndown uses backslashes (\) to escape Markdown characters in the HTML input. This ensures that these characters are not interpreted as Markdown when the output is compiled back to HTML. For example, the contents of <h1>1. Hello world</h1> needs to be escaped to 1\. Hello world, otherwise it will be interpreted as a list item rather than a heading.

To avoid the complexity and the performance implications of parsing the content of every HTML element as Markdown, Turndown uses a group of regular expressions to escape potential Markdown syntax. As a result, the escaping rules can be quite aggressive.

Overriding TurndownService.prototype.escape

If you are confident in doing so, you may want to customise the escaping behaviour to suit your needs. This can be done by overriding TurndownService.prototype.escape. escape takes the text of each HTML element and should return a version with the Markdown characters escaped.

Note: text in code elements is never passed toescape.

License

turndown is copyright © 2017+ Dom Christie and released under the MIT license.

turndown's Issues

Getting DOMException: Invalid character if an attribute starts with a "."

&lt;bottle.label&gt;<bottle .label=\"\">

yields

node_modules/to-markdown/node_modules/jsdom/lib/jsdom/living/helpers/validate-names.js:10
    throw new core.DOMException(core.DOMException.INVALID_CHARACTER_ERR,
          ^
DOMException: Invalid character: ".label" did not match the Name production: Expected ":", "_", [A-Z], [\u0370-\u037D], [\u037F-\u1FFF], [\u200C-\u200D], [\u2070-\u218F], [\u2C00-\u2FEF], [\u3001-\uD7FF], [\uD800-\uDB7F], [\uF900-\uFDCF], [\uFDF0-\uFFFD], [\xC0-\xD6], [\xD8-\xF6], [\xF8-\u02FF] or [a-z] but "." found.

Background:

I ran into this at blogger2ghost. It looks like a blog post of mine can contain attributes like this and it makes to-markdown, and consequently my library and service, to blow up.

toMarkdown(undefined) crashes

It would be nice if toMarkdown(undefined) and toMarkdown(null) returned undefined and null, respectively.

If helps when map-ing over arrays where properties are not always set.

How to deal with non-markdown HTML elements?

Keep them/strip them?

Hang in list item

This code will cause it to hang

<ol>
  <li> first text
                      some text
  </li>
</ol>

if you remove the space before some text it'll be fine, as you add spaces, you'll notice the performance go down and eventually it'll just freeze.

The space is lost between the image and the text

HTML

<h1><img src="image.png"> Blabla</h1>

Markdown

# ![](image.png)Blabla

Expected markdown

# ![](image.png) Blabla

Strings with more than one list are not converted correctly

Not convert markdown syntax inside html

var html = '<p> *fackbold* _fackitalic_ [fack](link)</p>';
console.log(toMarkdown(html));

I expected to get:\*fackbold\* \_fackitalic\_ \[fack\](link)\n\n
but I get: *fackbold* _fackitalic_ [fack](link)(even no \n\n at the end)

Is it a bug or It's just how it works?

(I do this because I provide a WYSIWYG in browser. When a user happen to write a markdown syntax, I hope it will not become a part of markdown)

not encoding back from HTML codes

Hello, I’m using to-markdown with readabilitySAX project, and I have such code:

var toMarkdown = require('to-markdown').toMarkdown;
var readability = require('readabilitySAX');

var args = process.argv.slice(2);
var url = args[0];

readability.get(url, {}, function (data) {
    var md = toMarkdown(data.html);
    console.log(md);
});

For example, I’m testing it on one of the lastest A List Apart articles:

$ node mdIt.js http://alistapart.com/article/the-era-of-symbol-fonts

In my handcrafted markdown I have

    <div><span class="icon">L</span> Next</div>

but to-markdown makes it like:

    &lt;div>&lt;span class="icon">L&lt;/span> Next&lt;/div>`</pre>

I think it’s all about HTML codes, because markdown don’t need them. Am I right?

Demo doesn't works in Chrome

The demo provided at http://domchristie.github.io/to-markdown/ doesn't work in chrome because of https://code.google.com/p/chromium/issues/detail?id=180007

In console it gives following error.

Refused to execute script from 'https://raw.github.com/domchristie/to-markdown/master/src/to-markdown.js' because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.

<hr></hr> not converted

The HTML:

<hr></hr>

Becomes:

* * *
</hr>

Not Converting emdash

This tool is great, but I noticed it isn't converting the emdash (—) to the markdown equivalent of three hyphens (---).

Whitespace between consecutive inline elements is deleted

html like this

<p>I <a href="http://google.com">need</a> a space</p>

translates into this

I[need](http://google.com) a space

I think that space before the link should not be deleted.

Parsing HTML with a regexp

I see your parsing HTML with regexp. You know this is wrong by definition right?

Related

&quot; not converted to "

HTML " entities are not converted to ":

<pre><code>return shell_exec(&quot;echo $input | $markdown_script&quot;);</code></pre>

Becomes:

    return shell_exec("echo $input | $markdown_script");

Instead of expected:

    return shell_exec("echo $input | $markdown_script");

Element inside <a></a> tag sets the link text to 'undefined'.

On [email protected], the following:

toMarkdown('<a href="http://example.com/"><span>Example</span></a>');

will yield the following markdown:

[undefined](http://example.com/)

I reckon the link text should be something other than "undefined". In fact, even returning nothing would be probably better than "undefined" :)

Newline inside h tag

When converting

<h1>
    Some header text</h1>

the result is

#
    Some header text

which then renders back to

<h1> </h1>
<pre>Some header text</pre>

missing 'new' in instructions for jquery branch

In the browser section.

Fantastically useful module, by the way. Saved me a bunch of time. Thanks!

New lists always start with two newlines

Converting Markdown to HTML, and back to Markdown shows that nested lists have an issue.

*   This is a list item at root level
*   This is another item at root level
    *   This is a nested list item
    *   This is another nested list item
        *   This is a deeply nested list item
        *   This is another deeply nested list item
        *   This is a third deeply nested list item
*   This is a third item at root level

Becomes:

*   This is a list item at root level
*   This is another item at root level

    *   This is a nested list item
    *   This is another nested list item

        *   This is a deeply nested list item
        *   This is another deeply nested list item
        *   This is a third deeply nested list item
*   This is a third item at root level

Test is in my fork.

Uncaught ReferenceError: toMarkdown is not defined

error in subject theme

Cannot update to latest version: No compatible version found: collapse-whitespace

When running npm update I get the following error:

npm http 304 https://registry.npmjs.org/collapse-whitespace
npm ERR! Error: No compatible version found: collapse-whitespace@'domchristie/collapse-whitespace#numerical_node_type'
npm ERR! Valid install targets:
npm ERR! ["1.0.0","1.0.1","1.0.2","1.0.3","1.0.4","1.0.5","1.1.0","1.1.1"]
npm ERR!     at installTargetsError (/usr/share/npm/lib/cache.js:719:10)
npm ERR!     at /usr/share/npm/lib/cache.js:638:10
npm ERR!     at saved (/usr/share/npm/node_modules/npm-registry-client/lib/get.js:142:7)
npm ERR!     at /usr/lib/nodejs/graceful-fs/polyfills.js:133:7
npm ERR!     at Object.oncomplete (fs.js:107:15)

Is the version number correct for collapse-whitespace correct?

Thanks

pre tag parsing is broken on master / RC1

Hi there,

I've been using this library for a long time to extract some HTML from web pages and convert it to markdown and with the last tag (RC1 i think), it looks like all code blocks, pre blocks and such are dropped and don't appear in output...
v0.0.3 tag gives correct output with a ton of   entities but I doubt this is related to your lib.
I'm looking into my code as well to see if somehow you output something different that would make my code drop these blocks.

Thx

[RFC] Moving Forward

The current implementation uses a rather naive approach to parsing HTML. Using regular expressions is “quick and dirty”, but has produced code that is difficult to update and maintain (and the main reason why I have not been very good at staying on top of pull requests).

I have made attempts in the past to refactor the codebase to a more manageable approach, but haven’t quite had the time to complete the work, thoroughly test it, and get it merged.

The most recent work resides one the dom branch, and uses jsdom (node) and document (browser) for parsing HTML. I believe this is a reasonably good start, but needs some work.

Some of the ideas I’m hoping to get done include:

• Only maintain one set of tests that run in both on node and the browser
• Use modules to separate the concerns (I’m thinking about using broccoli and the ES6 transpiler/concatenator?).

and some features:

• Convert more elements (e.g. GFM)
• Options for syntax styles (e.g. * or - for bullets, # or = for <H1>s)
• Option for custom converters

If you have any thoughts on this direction and/or would like to help out let me know or comment below.

crashes on big html chunks

Uncaught TypeError: Cannot read property '1' of null 
75: return '![' + (alt && alt[1] ? alt[1] : '') + ']' + '(' + src[1] + (title && title[1] ? ' "' + title[1] + '"' : '') + ')';

Blockquote

This

<blockquote><p>The price of being a sheep is boredom. The price of being a wolf is loneliness. Choose one or the other with great care.</p></blockquote>
<author>Hugh MacLeod</author>

is converted into this:

http://d.pr/n/zh0J

Don't convert anything inside a tag that is going to be passed through as HTML

E.g.

<table><tr><td>
<i>A</i><b>B</b><strong>C</strong><em>D</em>
</td></tr></table>

Replaces tags with Markdown equivalents.

Entities not decoded

INPUT:

&

EXPECTED OUTPUT

&

ACTUAL OUTPUT

&

REFERENCE:

http://daringfireball.net/projects/markdown/syntax --> AUTOMATIC ESCAPING FOR SPECIAL CHARACTERS

Code blocks incorrect

GitHub Flavored Markdown code blocks are translated incorrectly.

```ruby
puts 'hello'
```

Instead of the above, to-markdown produces this:

  puts 'hello'

Blockquotes within list indentation

`html` parameter in function `replaceEls` may be null

https://github.com/domchristie/to-markdown/blob/master/src/to-markdown.js#L91

If html is null, it will throw exception. It has happened in some tools I'm using with to-markdown. I think we can check it before invoke html.replace

Create command-line executable

I know it's trivial but it would be useful to have a to-markdown command-line tool.

$ npm install -g to-markdown
$ to-markdown file.html > file.md

<div> does not become line break.

Unstyled <div>s imply a new full-width block, or a new line.

Right now this is not being respected by the parser, and a series of divs will be rendered onto the same line.

Is to-markdown no longer in npm?

$ npm search to-markdown
No match found for "to-markdown"

Could your register the module?

Preserve &nbsp; entity

The non-breaking space entity ( ) should not be by a normal space character.
More generally, it may be useful to let the user decide how HTML entities should be dealt with (converted or preserved).

Change behaviour of `converters` option

This issue relates to the discussion on #101.

The converters option currently extends the default converters but when invoking toMarkdown, it appears like it overrides them.

If the converters option were to override the converters, we could expose the default converters publicly (e.g. toMarkdown.defaultConverters, toMarkdown.gfmConverters), which could then be extended as follows:

var myConverters = […]

toMarkdown('…', {
  converters: myConverters.concat(toMarkdown.defaultConverters)
})

Option to escape markdown-like syntax in html

HTML:

<p>
#obamacare is a great idea, 
because blablabla....
</p>

To-markdown result:

#obamacare is a great idea, 
because blablabla....

Which renders in html:

<h1>
obamacare is a great idea,
</h1>
<p>
because blablabla....
</p>

This become a bit problematic when the original html is actually intend to use these markdown syntax characters.

It would be nice to have an option that allows user to escape these lines with a backslash.

Paste doesn't refresh output

When I Paste some text to input box, html output is not refreshed. I have to type some extra character to cause refresh.
Can you check for paste event the same way as for other keyboard entry?
See http://stackoverflow.com/questions/2176861/javascript-get-clipboard-data-on-paste-event-cross-browser/2177059

Tests won't run on node v0.12.0

I can't get the tests to work on NodeJS version 0.12.0 on Windows.

npm WARN engine [email protected]: wanted: {"node":">=0.6.0 < 0.12.0"} (current: {"node":"0.12.0","npm":"2.5.1"})
npm WARN engine [email protected]: wanted: {"node":"0.10.x"} (current: {"node":"0.12.0","npm":"2.5.1"})
npm WARN engine [email protected]: wanted: {"node":"0.10.x"} (current: {"node":"0.12.0","npm":"2.5.1"})

Option to remove all styling and keep just content

I'd like to see an option to remove all fluff such as html, head, style tags, class attributes, empty anchors, spans, pretty much everything.

I wonder how complex is such an enhancement though.

My intent for now is to be able to convert a Google Doc to (HTML to) Markdown.

It hangs when processing 126KB HTML file

Both Chrome and IE are affected. It hangs when doing regex.

nested lists not working

it failed to convert the following to markdown.

<ol>
  <li>item 1</li>
  <li>item 2</li>
  <ol>
     <li>item 2.1</li>
     <li>item 2.2</li>
  </ol>
</ol>

v1.1.1 does't works well with <p><strong><br></strong></p>

<p><strong><br></strong></p> will be converted to **undefined** in v1.1.1.

In v1.0.2, it will be converted to ****.

Error when multiple code blocks are present

This isn't converted correctly to markdown:

<h1>This fails</h1>
<p>First code block:</p>
<pre><code>function x(i) {
        return i++;
}
</code></pre>
<p>Second code block:</p>
<pre><code>
x(5) == 6;
</code></pre>
<p>ups!</p>

The output is:

# This fails

First code block:

    function x(i) {
        return i++;
    }
    `</pre>

    Second code block:

    <pre>`
    x(5) == 6;

ups!

And I would expect:

# This fails

First code block:

    function x(i) {
        return i++;
    }

Second code block:

    x(5) == 6;

ups!

cannot parse tag css style attributes,will be?

Tables

There's no support for converting tables to markdown.

Does not work on <dl><dt>

tags are not converted. Trying to parse to many of them also crashes.

Table Support!

Why this script don't support table

A backslash is added in the markdown before the point when the last word is a number.

HTML

<p>Blabla 167. Blabla</p>

Markdown

Blabla 167\. Blabla

Expected markdown

Blabla 167. Blabla

Accept malformed tag attributes

When a malformed tag is passed it throws a TypeError (Cannot read property '1' of null). It should just return null.

Highlighted code

I am making something like forum. And there is quote button. I click it and message inserted into markdown editor.

Everything works fine.

Bt I have syntax highlight for code, And it is inserted with <span> elements.

Is there a way to strip highlight tags?

Improve table conversion

• Remove cols / colgroups
• Assume ths in first tr in tbody are header cells