accordproject / markdown-transform

Parse and transform markdown text, including TemplateMark markdown templates

Markdown Transform

Introduction

A transformation and parsing framework for converting markdown content to HTML, PDF, DOCX, Slate (for rich-text editing) and other structured document object models (DOMs).

Structure of the Code Repository

Top level repository (markdown-transform), with sub packages. Each sub-package is published as an independent npm module using npm workspaces:

• markdown-cli : command line utilities
• markdown-transform : High-level API to transform files into different formats
• markdown-common : converts between markdown strings and the CommonMark DOM
• markdown-cicero : converts between the CommonMark DOM and the CiceroMark DOM
• markdown-template : converts between JSON data + markdown strings and TemplateMark
• markdown-slate : converts between the CiceroMark DOM and the Slate DOM
• markdown-html : converts between the CiceroMark DOM and HTML
• markdown-pdf : converts between a PDF file and the CiceroMark DOM
• markdown-docx : converts a DOCX file to the CiceroMark DOM
• markdown-it-cicero : markdown-it plugin for contracts and CiceroMark
• markdown-it-template : markdown-it plugin for templates and TemplateMark

CommonMark DOM

The CommonMark DOM is a model for the elements of CommonMark (the specification for markdown text), expressed as a Concerto schema, and serialized as a JSON graph.

The schema is defined here: https://models.accordproject.org/markdown/[email protected]

CiceroMark DOM

CiceroMark defines markdown documents with embedded clauses, where each clause is an instance of a template, specified using TemplateMark.

The CiceroMark DOM extends the CommonMark DOM, defining nodes for Clause, Variable and Formula etc.

The schema is defined here: https://models.accordproject.org/markdown/[email protected]

TemplateMark DOM

TemplateMark defines markdown documents with syntax for embedded variables, optional blocks, formulas etc. It is used to define Accord Project templates.

The TemplateMark DOM extends the CommonMark DOM, defining nodes for ClauseDefinition, VariableDefinition and ForumulaDefinition etc.

The schema is defined here: https://models.accordproject.org/markdown/templatemark.html

Slate DOM

The Slate DOM is defined by the Slate HTML content-editable WYSIWYG editing framework.

Installation

For users

To install the command-line interface:

npm install -g @accordproject/markdown-cli

This will install the markus command-line for markdown transform. To get command-line help:

markus --help

For developers

npm install

Then run:

npm run test

This command uses npm workspaces to run the tests for each package in the monorepo.

---

Accord Project is an open source, non-profit, initiative working to transform contract management and contract automation by digitizing contracts. Accord Project operates under the umbrella of the Linux Foundation. The technical charter for the Accord Project can be found here.

Learn More About Accord Project

Overview

Documentation

Contributing

The Accord Project technology is being developed as open source. All the software packages are being actively maintained on GitHub and we encourage organizations and individuals to contribute requirements, documentation, issues, new templates, and code.

Find out what’s coming on our blog.

Join the Accord Project Technology Working Group Discord channel to get involved!

For code contributions, read our CONTRIBUTING guide and information for DEVELOPERS.

README Badge

Using Accord Project? Add a README badge to let everyone know:

[![accord project](https://img.shields.io/badge/powered%20by-accord%20project-19C6C8.svg)](https://www.accordproject.org/)

License

Accord Project source code files are made available under the Apache License, Version 2.0. Accord Project documentation files are made available under the Creative Commons Attribution 4.0 International License (CC-BY-4.0).

Copyright 2018-2019 Clause, Inc. All trademarks are the property of their respective owners. See LF Projects Trademark Policy.

markdown-transform's Issues

PDF -> CiceroMark

It would be useful to be able to import contracts from existing PDFs.

https://github.com/jzillmann/pdf-to-markdown
https://pandoc.org

This should be integrated with #145

Emphasis around Variables yields null text nodes in Slate DOM

Describe the bug
Using emphasis (e.g., *..* or **...**) around inlines or CiceroMark variables results in nodes with null text in Slate DOM.

This breaks the Slate editor, and the markdown-editor.

To Reproduce
Steps to reproduce the behavior:

1. Download the attached file (which contains markdown with emphasis around inlines or variables)
   stronginline.txt
2. Translate it to the Slate DOM using Markus:

Markus parse --sample strongvariable.txt --slate

3. Notice the null text nodes:

...
        "nodes": [
          {
            "object": "text",
            "text": "On ",
            "marks": []
          }, 
          {
            "object": "text",
            "text": null,
            "marks": [
              {
                "object": "mark",
                "type": "italic",
                "data": {
                }
              }
            ]
          }
        ]
...

Expected behavior
A proper slate node for inline or variables with the corresponding emphasis mark

Empty List Items throw errors in Slate fromMarkdown

Describe the bug
Error occurs when translating markdown to Slate

Additional context
This is happening when taking markdown from a database, which includes an unordered list (ulist) with a final bullet point which is empty.

The error

Uncaught Error: Node Item doesn't have any children!
    at ToSlateVisitor.processChildNodes (VM84894 ToSlateVisitor.js:1809)
    at ToSlateVisitor.visit (VM84894 ToSlateVisitor.js:2091)
    at ValidatedConcept.accept (VM84924 typed.js:57)
    at eval (VM84894 ToSlateVisitor.js:1820)
    at Array.forEach (<anonymous>)
    at ToSlateVisitor.processChildNodes (VM84894 ToSlateVisitor.js:1815)
    at ToSlateVisitor.visit (VM84894 ToSlateVisitor.js:2079)
    at ValidatedConcept.accept (VM84924 typed.js:57)
    at eval (VM84894 ToSlateVisitor.js:1820)
    at Array.forEach (<anonymous>)

From this source code

processChildNodes(thing) {
    cov_2olvdi9vnx.f[4]++;
    const result = (cov_2olvdi9vnx.s[23]++, []);
    cov_2olvdi9vnx.s[24]++;

    if (!thing.nodes) {
      cov_2olvdi9vnx.b[7][0]++;
      cov_2olvdi9vnx.s[25]++;
      throw new Error("Node ".concat(thing.getType(), " doesn't have any children!"));
    } else {
      cov_2olvdi9vnx.b[7][1]++;
    }

When using this markdown

This is the contract editor. You can edit this text!

This is a heading!
====

This is [text](http://clause.io "").

Fragile Goods
----

`\`\`\ <clause src="ap://[email protected]#..." clauseid="49c95009-..."/>
Liquidated Damages for Delayed Delivery.

In the event the EXW delivery date of the Equipment is delayed beyond the delivery schedule as indicated below, solely through the fault of <variable id="seller" value="%22Dan%22"/> (the Seller), and unless the parties mutually agreed to an extension thereto, <variable id="buyer" value="%22Steve%22"/> (the Buyer) is entitled to claim liquidated damages in an amount equivalent to <variable id="lateDeliveryPenalty" value="300%20USD"/>.
Prior to implementing the provisions of Article 16.4 pursuant to this section, Buyer agrees that it shall discuss with Seller alternate remedies in good faith.. . . .

The Equipment to be shipped to the Buyer shall be packed and shipped in accordance with the Specifications and if not specified therein....
Additionally the Equipment should have proper devices on it to record any shock during transportation as any instance of acceleration outside the bounds of <variable id="accelerationMin" value="-0.5"/>g and <variable id="accelerationMax" value="0.5"/>g.
Each shock shall reduce the Contract Price by <variable id="accelerationBreachPenalty" value="5%20USD"/>. Packing containing fragile materials should be so marked in bold stout letters. . . . .

Equipment Description, Contract Price and Delivery Schedule

Contract Price is <variable id="deliveryPrice" value="1000%20USD"/> per unit of Equipment.
Delivery Schedule: no later than <variable id="deliveryLimitDuration" value="10%20seconds"/> after initiation.
`\`\`\



- one
- two
- thre
- 

Acceptance of Delivery
----

`\`\`\ <clause src="ap://[email protected]#..." clauseid="eb77e820-..."/>
Acceptance of Delivery. <variable id="shipper" value="%22Dan%20Selman%22"/> will be deemed to have completed its delivery obligations if in <variable id="receiver" value="%22Adrian%20Fletcher%22"/>'s opinion, the <variable id="deliverable" value="%22Widgets%22"/> satisfies the Acceptance Criteria, and <variable id="receiver" value="%22Adrian%20Fletcher%22"/> notifies <variable id="shipper" value="%22Dan%20Selman%22"/> in writing that it is accepting the <variable id="deliverable" value="%22Widgets%22"/>.

Inspection and Notice. <variable id="receiver" value="%22Adrian%20Fletcher%22"/> will have <variable id="businessDays" value="10"/> Business Days' to inspect and evaluate the <variable id="deliverable" value="%22Widgets%22"/> on the delivery date before notifying <variable id="shipper" value="%22Dan%20Selman%22"/> that it is either accepting or rejecting the <variable id="deliverable" value="%22Widgets%22"/>.

Acceptance Criteria. The "Acceptance Criteria" are the specifications the <variable id="deliverable" value="%22Widgets%22"/> must meet for the <variable id="shipper" value="%22Dan%20Selman%22"/> to comply with its requirements and obligations under this agreement, detailed in <variable id="attachment" value="%22Attachment%20X%22"/>, attached to this agreement.
`\`\`\

Conditional Variable Othertext

Cicero draft will be changed to put all conditional variables in markdown XML

This package should handle this change and put variables in the Slate object, maybe in the data of the inline?

For commonMark to ciceroMark and ciceroMark to Slate:
Adjust to handling these inlines being changed

Related issue comment by Jerome:

Late Delivery and Penalty.
----

In case of delayed delivery<if id="forceMajeure" whenTrue=" except for Force Majeure cases," whenFalse=" even when there is a Force Majeure," value="true"/>
<variable id="seller" value="%22Dan%22"/> (the Seller) shall pay to <variable id="buyer" value="%22Steve%22"/> (the Buyer) for every <variable id="penaltyDuration" value="2%20days"/>
of delay penalty amounting to <variable id="penaltyPercentage" value="10.5"/>% of the total value of the Equipment
whose delivery has been delayed. Any fractional part of a <variable id="fractionalPart" value="days"/> is to be
considered a full <variable id="fractionalPart" value="days"/>. The total amount of penalty shall not however,
exceed <variable id="capPercentage" value="55.0"/>% of the total value of the Equipment involved in late delivery.
If the delay is more than <variable id="termination" value="15%20days"/>, the Buyer is entitled to terminate this Contract.

          {
            "object": "inline",
            "type": "conditional",
            "data": {
              "id": "forceMajeure",
              "whenTrue": " except for Force Majeure cases,",
              "whenFalse": " even when there is a Force Majeure,",
              "value": false
            },
            "nodes": [
              {
                "object": "text",
                "text": " even when there is a Force Majeure,",
                "marks": []
              }
            ]
          }, 

Slate transform for thematic breaks

Describe the bug
The Slate transform turns thematic seems to turn thematic breaks into empty paragraphs.

To Reproduce

Here is a simple test.md:

This is a heading
----

This is a thematic break:

----

This is more paragraph

The Cicero DOM looks fine:

bash-3.2$ markus parse --sample test.md --cicero 
10:38:13 - info:
{
  "$class": "org.accordproject.commonmark.Document",
  "xmlns": "http://commonmark.org/xml/1.0",
  "nodes": [
    {
      "$class": "org.accordproject.commonmark.Heading",
      "level": "2",
      "nodes": [
        {
          "$class": "org.accordproject.commonmark.Text",
          "text": "This is a heading"
        }
      ]
    }, 
    {
      "$class": "org.accordproject.commonmark.Paragraph",
      "nodes": [
        {
          "$class": "org.accordproject.commonmark.Text",
          "text": "This is a thematic break:"
        }
      ]
    }, 
    {
      "$class": "org.accordproject.commonmark.ThematicBreak"
    }, 
    {
      "$class": "org.accordproject.commonmark.Paragraph",
      "nodes": [
        {
          "$class": "org.accordproject.commonmark.Text",
          "text": "This is more paragraph"
        }
      ]
    }
  ]
}

But in the Slate DOM the thematic break has disappeared:

bash-3.2$ markus parse --sample test.md --slate
10:38:37 - info:
{
  "object": "value",
  "document": {
    "object": "document",
    "data": {
    },
    "nodes": [
      {
        "object": "block",
        "type": "heading_two",
        "data": {
        },
        "nodes": [
          {
            "object": "text",
            "text": "This is a heading",
            "marks": []
          }
        ]
      }, 
      {
        "object": "block",
        "type": "paragraph",
        "data": {
        },
        "nodes": [
          {
            "object": "text",
            "text": "This is a thematic break:",
            "marks": []
          }
        ]
      }, 
      {
        "object": "block",
        "type": "paragraph",
        "data": {
        },
        "nodes": []
      }, 
      {
        "object": "block",
        "type": "paragraph",
        "data": {
        },
        "nodes": [
          {
            "object": "text",
            "text": "This is more paragraph",
            "marks": []
          }
        ]
      }
    ]
  }
}

DEVELOPERS file

Tasks

• Create information files for developers

Related Issues:

• Cicero UI Issue 37

Wrapped Variants of List blocks

Is your feature request related to a problem? Please describe.

Provide new CiceroMark nodes for variables corresponding to list blocks in templates.

CiceroMark variables can be either seen as plain text or wrapped (e.g., <variable id="rate" value="3.5"/>).

List blocks in CiceroMark are currently not wrapped.

Variables corresponding to list blocks are not reflected in the Slate AST, which means they cannot easily be edited by the UI components.

Describe the solution you'd like

• A new kind of AST node in CiceroMark: ListVariable.
• Two new kind of AST node in Slate: ulist_variable and olist_variable
• The ability to parse/draft them. Suggested formats would be similar to that of clauses for lists ```<list ...> ```
• Translations to/from markdown, Cicero and Slate.

Additional context
Support for editing of conditional and list variables in Cicero UI:
accordproject/cicero-ui#230

Markus Dingus!

Is your feature request related to a problem? Please describe.
It's a feature request related to a feature. The idea would be to have a deployed Dingus in the style of https://spec.commonmark.org/dingus/ but for CiceroMark.

Describe the solution you'd like
It could be almost exactly like the commonmark dingus but:

1. Left box lets you write CiceroMark
2. Right box shows:
   • Preview
   • HTML
   • CommonMark DOM
   • CiceroMark Dom
   • Slate DOM

Externalize CTO models

The Concerto models for CommonMark and CiceroMark should be published to models.accordproject.org and downloaded by the build scripts.

Template Grammar to Slate DOM with Wrapped Variables

Is your feature request related to a problem? Please describe.
Need a way to go from a template grammar or sample to a Slate DOM with wrapVariables: true

Describe the solution you'd like
This will allow for rendering a formatted clause template without necessarily having the clauseTemplate

Detect Clauses in CommonMark DOM

Users currently have large volumes of existing legal contracts, containing a wide variety of legal clauses. E.g. the contracts may contain 90 day payment terms, acceptance of delivery clauses etc.

It would be very useful to be able to train a model to detect instances of a specific type of clause (say acceptance of delivery) and then to automatically (or with manual guidance) help the user replace the natural language in their contract with an instance of the acceptance of delivery clause, with the correct values extracted from the input contract.

The template detection logic should operate upon the CommonMark DOM (a JSON representation of Markdown formatted text) - replacing paragraph nodes in the DOM with Clause nodes from the CiceroMark DOM.

It would be useful to be able to run a fuzzy match a CommonMark DOM against a set of templates - detecting potential instances of the templates in the CommonMark text (clauses).

1. prepareTemplates : pre-process a set of templates to prepare them for matching (works on a training set?)
2. match : fuzzy match of templates to CommonMark DOM, extract values
3. uniqueMatches : eliminate duplicate matches, eliminate overlapping matches

Variable values are URI encoded in Slate DOM

The values of variables are URI encoded in the Slate DOM.

add image

Hacktoberfest-

add image for clone or fork

or view here

HTML Inlines do not round-trip in Slate transform

Describe the bug
HTML inlines do not roundtrip in Slate transform.

To Reproduce

bash-3.2$ cat inline.md 
On <code/>
bash-3.2$ ../markdown-cli/index.js normalize --sample inline.md --cicero
9:33:31 AM - info: 
On <code/>
bash-3.2$ ../markdown-cli/index.js normalize --sample inline.md --slate
9:33:35 AM - info: 
On 

undefined

add a image

https://i.ibb.co/XL1d3st/download.png

Release Notes (v0.20 Project)

Tasks

• This will track and construct the overall release notes for this repo in relation to the v0.20 project
  • Will cover everything from the inception of this repo

Related Issues:

• N/A

typo in the readme file

Describe the bug
typo in the readme file

To Reproduce
Steps to reproduce the behavior:

Fix typo in the readme file

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Should Markdown Transform depend on Slate at all?

We aren't using Slate in this package, except on one line:

markdown-transform/packages/markdown-slate/src/SlateTransformer.js

Line 54 in aa95442

return Value.fromJSON(result);

Unless we decide to validate against a Slate schema, I see little benefit of that additional dependency. We could make that call rely on the Slate JSON representation and let users load to Slate from it instead.

This would also let us remove of the slate and immutable peer dependencies.

What should be done with linebreaks/softbreaks in Slate transform?

I'm unclear that this is what should be happening?

markdown-transform/packages/markdown-slate/src/ToSlateVisitor.js

Lines 228 to 241 in 90cf0cf

	case 'Linebreak':
	result = {
	'object': 'block',
	'type': 'paragraph',
	'nodes': []
	};
	break;
	case 'Softbreak':
	result = {
	'object': 'block',
	'type': 'paragraph',
	'nodes': []
	};
	break;

problem with test coverage importing markdown-transform package

Describe the bug
same issue as this accordproject/template-archive#401
in markdown-common

Headings following certain kinds of blocks do not roundtrip

Describe the bug
In some cases, headings following certain kinds of blocks are not being rountripped to markdown properly.

bash-3.2$ cat sample.md 
# Heading One
This is text. This is *italic* text. This is **bold** text. This is a [link](https://clause.io). This is \`inline code\`.

This is ***bold and italic*** text

> This is a quote.
## Heading Two
This is more text.

Ordered lists:

1. one
1. two
1. three

Or:

* apples
* pears
* peaches

### Sub heading

Video:

<video/>

Another video:

<video src="https://www.youtube.com/embed/cmmq-JBMbbQ"/>
bash-3.2$ markus parse --sample sample.md --roundtrip 
13:36:50 - info: 
Heading One
====

This is text. This is *italic* text. This is **bold** text. This is a [link](https://clause.io). This is `inline code`.

This is ***bold and italic*** text

> This is a quote.Heading Two
----

This is more text.

Ordered lists:

1. one
2. two
3. three

Or:

- apples
- pears
- peaches### Sub heading

Video:

<video/>

Another video:

<video src="https://www.youtube.com/embed/cmmq-JBMbbQ"/>

CiceroMark -> PDF transformation

Implement a transformation (similar to the markdown-html HtmlTransformer) that generates a PDF document from a CiceroMark DOM.

Describe the solution you'd like
Implement a visitor using the https://pdfkit.org library, generating paragraphs and formatted text.

Describe alternatives you've considered
Tried to fine a lightweight HTML -> PDF library (pure JS) but come up empty.

Additional context
Something small, fast and lightweight that is 100% JS. A bonus would be the ability to customise look and feel etc.

API Refactoring

Simplify the API, improving naming consistency, update README files to explain the internal architecture.

Link Title Skip if Null

Describe the bug
A test in markdown-editor is failing around the way a link is round-tripped.

FAIL  src/SlateAsInputEditor/index.test.js
  ● SlateAsInputEditor component › successful roundtrip

    expect(received).toEqual(expected) // deep equality

    - Expected
    + Received

    @@ -1,8 +1,8 @@
      My Heading

    - This is text. This is *italic* text. This is **bold** text. This is a [link](https://clause.io). This is `inline code`.
    + This is text. This is *italic* text. This is **bold** text. This is a [link](https://clause.io ""). This is `inline code`.

This stems from this code, where the title is set to "" if null.

Expected behavior
Probably need to skip this behavior entirely.

Additional context
This commit

Download CommonMark and CiceroMark models from their published version

CTO models for CommonMark DOM and CiceroMark DOM are copied in the code here:
https://github.com/accordproject/markdown-transform/blob/master/packages/markdown-common/src/Models.js
and here:
https://github.com/accordproject/markdown-transform/blob/master/packages/markdown-cicero/src/Models.js

Those should be consistent with the officially published version at https://models.accordproject.org

Instead of copying those in the code (which is error prone and does not guarantee consistency when the published version changes), we should download them from the https://models.accordproject.org as part of the build process.

This could be done through the new concerto command line (see accordproject/concerto#67)

Issues with Lists in Slate Transform

Describe the bug
The slate transform seems to swallow lists in some cases. Here is a sample test.

list-slate.zip

update Markdown to Slate seems to lose any structure except for the first paragraph:

bash-3.2$ cat list.md 
1. item one
2. item two
   - sublist
   - sublist

bash-3.2$ markus parse --sample list.md --slate --output data.json 
13:16:40 - info: Creating file: data.json
13:16:40 - info:
{
  "object": "value",
  "document": {
    "object": "document",
    "data": {
    },
    "nodes": [
      {
        "object": "block",
        "type": "ol_list",
        "data": {
          "tight": "true",
          "start": "1",
          "delimiter": "period"
        },
        "nodes": [
          {
            "object": "block",
            "type": "list_item",
            "data": {
            },
            "nodes": [
              {
                "object": "text",
                "text": "item one",
                "marks": []
              }
            ]
          }, 
          {
            "object": "block",
            "type": "list_item",
            "data": {
            },
            "nodes": [
              {
                "object": "text",
                "text": "item two",
                "marks": []
              }
            ]
          }
        ]
      }
    ]
  }
}

Which means normalization returns only the top level list:

bash-3.2$ markus draft --data data.json --slate 
13:18:14 - info: 
1. item one
2. item two
bash-3.2$ markus normalize --sample list.md --slate
13:18:31 - info: 
1. item one
2. item two

This is true of anything which isn't pure text from a single paragraph (breaks, nested lists, multiple chunks of text, etc.)

CommonMark to markdown string transform should raise errors on standalone items

Item should always be included in List, so the following case should be turned into an error:

markdown-transform/packages/markdown-common/src/ToStringVisitor.js

Line 202 in fe3f0a6

case 'Item':

Error from `ToMarkdownStringVisitor`'s `escapeCodeBlock`

Describe the bug
Uncaught TypeError: Cannot read property 'replace' of undefined at Function.escapeCodeBlock at ToMarkdownStringVisitor.visit error can be caused when certain documents use the transform

Cannot install Markdown Transform 0.7.0

The latest 0.7.0 does not install due to the structure of the build with the new external model scripts.

bash-3.2$ npm install -g @accordproject/[email protected]
/Users/jeromesimeon/.nvm/versions/node/v8.12.0/bin/markus -> /Users/jeromesimeon/.nvm/versions/node/v8.12.0/lib/node_modules/@accordproject/markdown-cli/index.js

> @accordproject/[email protected] postinstall /Users/jeromesimeon/.nvm/versions/node/v8.12.0/lib/node_modules/@accordproject/markdown-cli/node_modules/@accordproject/markdown-common
> node ./scripts/getExternalModels.js

module.js:550
    throw err;
    ^

Error: Cannot find module '/Users/jeromesimeon/.nvm/versions/node/v8.12.0/lib/node_modules/@accordproject/markdown-cli/node_modules/@accordproject/markdown-common/scripts/getExternalModels.js'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Function.Module.runMain (module.js:694:10)
    at startup (bootstrap_node.js:204:16)
    at bootstrap_node.js:625:3
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! @accordproject/[email protected] postinstall: `node ./scripts/getExternalModels.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the @accordproject/[email protected] postinstall script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jeromesimeon/.npm/_logs/2019-10-26T18_52_10_502Z-debug.log

Remove unneeded encapsulation of Concerto / Concerto Tools

Markdown Transform use an uncommon idiom of packaging a dependency (composer-concerto and composer-concerto-tools) to avoid problems with multiple versions of the concerto package.

This can be removed whenever the corresponding issues has been resolved: accordproject/concerto#47 and

Attribute for HTML Inline should *not* decode as URI

There is no reason why arbitrary HTML inline attributes should be decoded, this is an assumption made only for CiceroMark wrapped variables:

bash-3.2$ cat test/data/inline.md
This is a <custom src="property%20or%20something"/> property.
bash-3.2$ ../markdown-cli/index.js parse --sample test/data/inline.md 
12:12:12 - info:
{
  "$class": "org.accordproject.commonmark.Document",
  "xmlns": "http://commonmark.org/xml/1.0",
  "nodes": [
    {
      "$class": "org.accordproject.commonmark.Paragraph",
      "nodes": [
        {
          "$class": "org.accordproject.commonmark.Text",
          "text": "This is a "
        }, 
        {
          "$class": "org.accordproject.commonmark.HtmlInline",
          "tag": {
            "$class": "org.accordproject.commonmark.TagInfo",
            "tagName": "custom",
            "attributeString": "src = "property%20or%20something" ",
            "attributes": [
              {
                "$class": "org.accordproject.commonmark.Attribute",
                "name": "src",
                "value": "property or something"
              }
            ],
            "content": "",
            "closed": true
          },
          "text": "<custom src="property%20or%20something"/>"
        }, 
        {
          "$class": "org.accordproject.commonmark.Text",
          "text": " property."
        }
      ]
    }
  ]
}

Mono-Repo Documentation

Tasks

• Ensure the main README captures the overall structure of the monorepo
• Ensure layout and structure of README is consistent with other AP repos
• Relevant information:
  • Markdown Transform
    • CommonmarkParser converts markdown text to an instance of the CommonMark AST
    • CommonmarkToString converts an AST to a markdown string
  • Slate Markdown Transform
    • The Cicero DOM extends this
    • Adds Clause, Variable, and ComputedVariable as Child nodes

Related Issues:

• Cicero UI Issue 65

Hacktoberfest:

This issue can provide for multiple pull requests.

Slate to ciceroMark transform sometimes omits nodes

Describe the bug
Some nodes disappear in the Slate to CiceroMark transform.

To Reproduce
Steps to reproduce the behavior:

1. Use the attached acceptance.txt file (which contains JSON)
   acceptance.txt
2. Execute: markus draft --slate --data acceptance.txt
3. Result is:

Acceptance of Delivery.
----

Inspection and Notice.
----

Acceptance Criteria.
----

The "Acceptance Criteria" are the specifications the <variable id="deliverable" value="%22Widgets%22"/>
must meet for the <variable id="shipper" value="%22Party%20A%22"/> to comply with its requirements and
obligations under this agreement, detailed in <variable id="attachment" value="%22Attachment%20X%22"/>, attached
to this agreement.

Instead of the expected:

Acceptance of Delivery.
----

<variable id="shipper" value="%22Partry%20eeA%22"/> will be deemed to have completed its delivery obligations
if in <variable id="receiver" value="%22Party%20B%22"/>'s opinion, the <variable id="deliverable" value="%22Widgets%22"/> satisfies the
Acceptance Criteria, and <variable id="receiver" value="%22Party%20B%22"/> notifies <variable id="shipper" value="%22Party%20A%22"/> in writing
that it is accepting the <variable id="deliverable" value="%22Widgets%22"/>.

Inspection and Notice.
----

<variable id="receiver" value="%22Party%20B%22"/> will have <variable id="businessDays" value="10"/> Business Days to inspect and
evaluate the <variable id="deliverable" value="%22Widgets%22"/> on the delivery date before notifying
<variable id="shipper" value="%22Party%20A%22"/> that it is either accepting or rejecting the
<variable id="deliverable" value="%22Widgets%22"/>.

Acceptance Criteria.
----

The "Acceptance Criteria" are the specifications the <variable id="deliverable" value="%22Widgets%22"/>
must meet for the <variable id="shipper" value="%22Party%20A%22"/> to comply with its requirements and
obligations under this agreement, detailed in <variable id="attachment" value="%22Attachment%20X%22"/>, attached
to this agreement.

Issues with Nested Lists roundtrip

Describe the bug
Nested lists are drafted with an extra new line, which means tight lists become non-tight.

Here is an example:

bash-3.2$ markus --version 
0.7.3
bash-3.2$ cat list.md 
1. item one
2. item two
   - sublist
   - sublist
bash-3.2$ markus normalize --sample list.md --output list2.md
13:27:42 - info: Creating file: list2.md
13:27:42 - info: 
1. item one
2. item two

   - sublist
   - sublist
bash-3.2$ markus parse --sample list.md
13:27:51 - info:
{
  "$class": "org.accordproject.commonmark.Document",
  "xmlns": "http://commonmark.org/xml/1.0",
  "nodes": [
    {
      "$class": "org.accordproject.commonmark.List",
      "type": "ordered",
      "start": "1",
      "tight": "true",         // HERE
      "delimiter": "period",
      "nodes": [
...
bash-3.2$ markus parse --sample list2.md
13:27:54 - info:
{
  "$class": "org.accordproject.commonmark.Document",
  "xmlns": "http://commonmark.org/xml/1.0",
  "nodes": [
    {
      "$class": "org.accordproject.commonmark.List",
      "type": "ordered",
      "start": "1",
      "tight": "false",        // HERE
      "delimiter": "period",
      "nodes": [
...
bash-3.2$ 

Some CommonMark spec examples do not roundtrip

The test is currently skipped here:
https://github.com/accordproject/markdown-transform/blob/master/src/CommonMark.test.js#L149

CONTRIBUTING file

Tasks

• Create information files for contributors

Related Issues:

• Cicero UI Issue 37

Monospace Link Roundtrip

Describe the bug
Monospace links are broken in the roundtrip

Screenshots

---

---

Additional context
This may need to be moved to markdown-editor?

Broken Link in README

Link to CiceroMark DOM schema
https://models.accordproject.org/ciceromark/ciceromark.html
is broken

DOCX -> CiceroMark

It would be useful to be able to create contracts from existing MS Word DOCX files.

https://github.com/mwilliamson/mammoth.js#readme
https://pandoc.org

This should be integrated with #145

AP Documentation Sync

Tasks

• Create charter file
• Ensure layout and structure of README is consistent with other AP repos

Related Issues:

• Cicero UI Issue 65

CLI is broken (missing some dependencies?)

Describe the bug
Installing the Command Line Interface works:

bash-3.2$ npm -g install @accordproject/markdown-cli 
+ @accordproject/[email protected]
updated 1 package in 10.988s

But trying to run it, you get:

bash-3.2$ mdtransform --help
module.js:550
    throw err;
    ^

Error: Cannot find module 'jsome'
    at Function.Module._resolveFilename (module.js:548:15)
    at Function.Module._load (module.js:475:25)
    at Module.require (module.js:597:17)
    at require (internal/module.js:11:18)
    at Object.<anonymous> (/Users/jeromesimeon/.nvm/versions/node/v8.12.0/lib/node_modules/@accordproject/markdown-cli/node_modules/@accordproject/markdown-common/lib/Logger.js:1214:39)
    at Module._compile (module.js:653:30)
    at Object.Module._extensions..js (module.js:664:10)
    at Module.load (module.js:566:32)
    at tryModuleLoad (module.js:506:12)
    at Function.Module._load (module.js:498:3)

Most likely some dependencies are missing from the package.json in markdown-cli.

Method for retrieving full non-variable clause text from a clause node

Is your feature request related to a problem? Please describe.
Because the clauseText attribute will be removed from the clause nodes in the concerto AST, I cannot easily obtain the full text of a clause without variable syntax from the markdown string for a contract.

Describe the solution you'd like
Method for retrieving non-variable text from a concerto AST clause node. I believe this method would be part of the cicero mark transformer, but not positive.

Additional context
My use case: I have markdown that includes clauses with variable syntax, and I want to get the clauses and their non-variable text. Once the clauseText attribute is removed from the clause code in the concerto AST, I will need to do this hacky code (which I am trying to avoid):

  function getClausesFromMarkdown(markdown) {
    const commonMarkTransformer = new CommonMarkTransformer({ tagInfo: true });
    const ciceroMarkTransformer = new CiceroMarkTransformer();
    let concertoAst = commonMarkTransformer.fromMarkdown(markdown, 'json');
    concertoAst = ciceroMarkTransformer.fromCommonMark(concertoAst, 'json');
    const clauseNodes = concertoAst.nodes.filter(({ $class }) => $class === 'org.accordproject.ciceromark.Clause');
    return clauseNodes.map((node) => {
      const {
        clauseid, src, clauseText, ...transformedNode
      } = node;
      return ({
        ...node,
        clauseText: ciceroMarkTransformer.toMarkdown({ ...transformedNode, $class: 'org.accordproject.commonmark.Paragraph' }, { wrapVariables: false })
      });
    });
  }

Issue roundtripping multiline block quotes

When saving from slate to markdown and then re-rendering as Slate, multiline block quotes get lost. The first line is preserved, but the subsequent lines fall outside the block quote when we reload.

Wrapped Variants of Conditional blocks

Is your feature request related to a problem? Please describe.

Provide new CiceroMark nodes for variables corresponding to conditional blocks in templates.

CiceroMark variables can be either seen as plain text or wrapped (e.g., <variable id="rate" value="3.5"/>).

Conditional blocks in CiceroMark can also be wrapped, but cannot be easily distinguished from regular variables.

Conditional variables corresponding to conditional blocks are not reflected in the Slate AST, which means they cannot easily be edited by the UI components.

Describe the solution you'd like

• A new kind of AST node in CiceroMark: ConditionalVariable
• A new kind of AST node in Slate: conditional
• The ability to parse/draft them. Suggested formats would be similar to variables but conditionals <if id="varName" value=""/>
• Translations to/from markdown, Cicero and Slate.

Additional context
Support for editing of conditional and list variables in Cicero UI:
accordproject/cicero-ui#233

Slate to markdown string translation does not work for Clauses

For instance:

bash-3.2$ cd packages/markdown-cli 
bash-3.2$ ./index.js parse --sample test/data/acceptance.md --withSlate --generateMarkdown  
13:27:39 - info: 
HELLO! This is the contract editor.
====

And below is a **clause**.

```
undefined```

Support for images

Image support is currently partial. We need all the transforms to support images.

Slate transform does not roundtrip when the text uses softbreaks

Describe the bug
Translating from markdown to slate and back sometimes introduces additional new lines.

To Reproduce

For instance on the attached sample.md:
sample.txt

$ cat sample.md 
## Fixed rate loan

This is a _fixed interest_ loan to the amount of 100000
at the yearly interest rate of 2.5%
with a loan term of 15,
and monthly payments of {{I'm not sure which amount right now}}
$ markus normalize --sample sample.md --slate
12:34:44 - info: 
Fixed rate loan
----

This is a *fixed interest* loan to the amount of 100000

at the yearly interest rate of 2.5%

with a loan term of 15,

and monthly payments of {{I'm not sure which amount right now}}

Roundtripping Headers made to List

Describe the bug
Roundtripping text in the contract editor when there are h1 elements with numbers at the beginning will cause the formatting to change.

Regular text

## 1. Heading two

## 2. Heading two again

## 3. Heading two third

Results in (I think):

Regular text

1. Heading two

---

1. Heading two again

---

1. Heading two third

---

With horizontal rules

Expected behavior
Formatting should be kept as the user sees when roundtripping.

Screenshots

Additional context
@Michael-Grover any other details?

Defense for Transforms

We should provide protections for the transform to handle more robust markdown transforms.

For instance, putting cicero-ui's README.md file through the transform fails at this point.

Variable values are quoted

Describe the bug

When variables are generated the value includes an extra quote (escaped as %22).

<variable id="shipper" value="%22Party%20A%22"/>