mjbvz / vscode-fenced-code-block-grammar-injection-example

Example of injecting a new grammar into VSCode's builtin markdown syntax highlighting for fenced code blocks

VSCode Markdown Fenced Code Block Grammar Injection Example

Demonstrates how an extension can inject support for a new grammar in VSCode's builtin markdown grammar for fenced code blocks. This extension injects an alias for JavaScript called superjs so support editor highlighting of blocks that look like:

```superjs
someJsCode
```

Structure

• package.json - VS Code extension manifest file. The contributes.grammars section registers the injected grammar.
• syntaxes/codeblock.json - The injected grammar itself

How to modify this repo to support a new language in fenced code blocks

1. In syntaxes/codeblock.json, change the begin rule from superjs to the identifier of your target language. This identifier is what people will write in markdown.
2. In syntaxes/codeblock.json, change the inner include rule from "source.js" to the scope of your target language. This scope can be found by looking at the target language's grammar.
3. In syntaxes/codeblock.json, change the contentName from using superjs to using a identifier for your language. This identifier may only contain letters but does not have to match the identifier from step 1.
4. In syntaxes/codeblock.json, change the scopeName from using superjs to using a identifier for your language. This identifier may only contain letters but does not have to match the identifier from step 1.
5. In package.json, change the scopeName to match the scopeName from step 4.
6. In package.json, change embeddedLanguages to map between the contentName from step 3 and the VS Code identifier for your language.
7. In package.json, change the id from using superjs-injection to using an identifier for your grammar injection. This identifier does not have to match the identifier from step 1.
8. In package.json, change the language from using superjs-injection to using an identifier for your grammar injection. This identifier has to match the identifier from step 7.

vscode-fenced-code-block-grammar-injection-example's Issues

ID and Name are displayed as undefined in Feature Contributions tab

Thank you for an excellent example. I noticed a minor issue in which ID and Name are displayed as "undefined" in Feature Contributions tab of the extension page.

Supporting aliases

Following these steps I was able to get haskell to work properly. I want to also add hs – is there any way to reuse a single json file and/or grammar config to support both aliases, or do I need to create a brand new copy of each? It's fine if the latter is the case, just checking. Thanks!

the language `begin` pattern is matching deep within in the code block (false positives)

An example like this (where ◆ = backtick for rendering purposes):

◆◆◆
this is some text
la la la
re
more text
◆◆◆

Matches the re for e.g. the ReasonML language which uses an identical injection file as this example repo. The desired behavior would only be to match language identifiers appearing after the triple backticks on the same line:

◆◆◆re
some text
more text
etc.
◆◆◆

The problem would seem to be that the regex in the begin property for this example appears to be too lax.

vscode-fenced-code-block-grammar-injection-example/syntaxes/codeblock.json

Line 11 in dd6961f

"begin": "superjs(\\s+[^`~]*)?$",

I tried playing with it a bit, but I was stymied as patterns which I thought would apply (matching against triple backticks, for example) failed to match appropriately. Accordingly I have a few questions:

1. How are the begin and end patterns actually applied in VSCode? Are they applied to entire files, or just to the contents of code fences, or something else?
2. What precise regex syntax does this example repo use? It doesn't appear to be JS regex considering the presence of the \G pattern.

Let me know if I am off-base, but since many language extensions are using this repo as a template without carefully analyzing how exactly it works, it seems worthwhile to make the pattern as bulletproof as is reasonably possible. I am seeing a lot of false positives.

(This issue is essentially a superclass of #5.)

What's wrong with my syntax highlighting grammar injection?

Sorry this is not a bug. I would like to ask you for some help regarding the grammar injection syntax as I couldn't find any relevant resources.

I am following codeblock.json for a simple grammar injection in markdown.

There is a small problem with the above code - As long as there is a string superjs appear in the fenced code block, the remaining would always be rendered as superjs, which would break fenced code block syntax highlighting for other embedded languages.

```json
{
   "foo": "superjs"
}
```

I would like to fix it by following https://github.com/Microsoft/vscode/blob/master/extensions/markdown/syntaxes/markdown.tmLanguage

"fileTypes": [],
"injectionSelector": "text.html.markdown",
"patterns": [
    {
        "include": "#fenced_code_block_superjs"
    }
],
"repository": {
    "fenced_code_block_superjs": {
        "begin": "(^|\\G)(\\s*)(`{3,}|~{3,})\\s*(?i:(superjs)(\\s+[^`~]*)?$)", 
        "beginCaptures": {
            "3": {
                "name": "punctuation.definition.markdown"
            }, 
            "5": {
                "name": "fenced_code.block.language"
            }, 
            "6": {
                "name": "fenced_code.block.language.attributes"
            }
        }, 
        "end": "(^|\\G)(\\2|\\s{0,3})(\\3)\\s*$", 
        "endCaptures": {
            "3": {
                "name": "punctuation.definition.markdown"
            }
        }, 
        "name": "markup.fenced_code.block.markdown", 
        "patterns": [
            {
                "begin": "(^|\\G)(\\s*)(.*)", 
                "contentName": "meta.embedded.block.superjs", 
                "patterns": [
                    {
                        "include": "source.js"
                    }
                ], 
                "while": "(^|\\G)(?!\\s*([`~]{3,})\\s*$)"
            }
        ]
    }
},
"scopeName": "markdown.superjs.codeblock"

However it doesn't work, and I don't know how to debug it either.

The related stackoverflow post:

https://stackoverflow.com/questions/47418275/whats-wrong-with-my-syntax-highlighting-grammar-injection

Installing locally for those not used to vscode extensions

Hello, great example, this was really helpful. However, for those of us that have no idea how extensions work for vscode, perhaps a quick local installation section would be helpful?

I did the following steps and it worked out great:

$ npm install -g vsce
$ vsce package
$ code --install-extension markdown-fenced-codeblock-example-injection-0.0.1.vsix

This isn't quite an issue, per-se. Unless you wanted to modify the README.md to add a section?

syntax highlighting still works inside markdown comments

Hi there,

First of all, thanks for your showcase example; it helped me and saved me a lot of time.
I came across an issue that syntax highlighting still works when the code block is inside a comment:

the example didn't work

VSCode version:1.15.1
OS version: Windows 10

Hi there, I have just downloaded the example and copy it to my vscode folder under ".vscode/extensions/".I can find the extension in the extensions list, But just didn't get highlight code in code block begin with superjs.
I also find in the file "package.json", there is a specific path "./syntaxes/test.json" which is not exists. Should I change the path to somthing else or just add this file?

https://github.com/mjbvz/vscode-fenced-code-block-grammar-injection-example/issues/15#issue-1163329724

Syntax highlighting of injected language not shown in Preview

Thanks @mjbvz for this super helpful example.
Running the example as it is I get the following behavior:

It seems within the md.-file the injected Syntax is highlighted but in the Preview, it isn't. Is that by design? Is it possible to transfer the Syntax highlighting functionality into the Preview as well?

My Setup:
Version: 1.46.1
Electron: 7.3.1
Chrome: 78.0.3904.130
Node.js: 12.8.1
V8: 7.8.279.23-electron.0
OS: Linux x64 5.3.0-61-generic

Adding support for Gherkin in Markdown files

Hi @mjbvz, we want to have Gherkin syntax highlight in Markdown code blocks.

This feature was previously requested at microsoft/vscode#23055 (comment) and now we're trying to implement it in this extension.

We tried to follow your README, but it didn't work. Can you please code review our codeblock.json and tell us where we messed up?

For me steps 2-4 were the most confusing. I skipped step 6 because I don't have a embeddedLanguages field anywhere.

We would appreciate if you could help us. Thanks.

```superjs someJsCode ```

Markdown Preview does not show highlighted code

I was able to get my language to highlight within the .md file, but when I go to view the rendered preview, the code is still not being highlighted.

false positives for superstrings

If we use an identifier foo, it will currently also match ofoo, abcdefghfoo, etc.

This is potentially problematic, e.g. for hs (Haskell) vs. lhs (Literate Haskell).

Is this repo out of date?

Thanks for putting together this example @mjbvz! I found a few references to your code in the issues of Microsoft/vscode repo and could use it as guidance to implement elm-tooling/vscode-elm-old#186.

I noticed a small issue when replicating your code ‘as is’, but am not sure if that's because of the recent changes in VSCode or simply due to the lack of my experience (I'm using VSCode just since last Wednesday). Here's where the main problem was:

vscode-fenced-code-block-grammar-injection-example/syntaxes/codeblock.json

Lines 10 to 12 in f341990

	"superjs-code-block": {
	"begin": "superjs",
	"end": "(^|\\G)(?=\\s*[`~]{3,}\\s*$)",

After replacing superjs with elm, ```elm blocks began to highlight, however, the same also became true for ```elm42, ```elmABCD and so on. Besides, in some cases the code was ‘spilling’ out of the fenced block, thus highlighting the following markdown incorrectly. This situation looks related to what's been reported in #2.

Could it be that your example needs updating by making codeblock.js a bit more similar to markdown.tmLanguage.json in the core repo? If so, it'd be great if you could show the up-to-date way of injecting the grammars!

Perhaps, another thing that'd be great to improve is README. It could contain a new section with steps to follow in order to get the result. These could be as simple as copy this bit from this file, ensure this is replaced with this, etc. I'm sure that the newcomers like me will appreciate such a guidance!

Please help me to highlight embed in XML

I have successfully accomplished highlights of ST language in markdown. Now I want to do the same in XML.

I have this example

<?xml version="1.0" encoding="utf-8"?>
<TcPlcObject Version="1.1.0.1" ProductVersion="3.1.4024.0">
  <DUT Name="ST_StateMachine" Id="{55662db1-7baf-4d79-8020-5daec45ea678}">
    <Declaration><![CDATA[TYPE ST_StateMachine :
STRUCT
	uiInit 		: UINT;
	uiReady 	: UINT;
	uiBusy 		: UINT;
	uiWaiting 	: UINT;
	uiIdle 		: UINT;
	uiError 	: UINT;
END_STRUCT
END_TYPE
]]></Declaration>
  </DUT>
</TcPlcObject>

1. Created xml.codeblock.json file

{
    "fileTypes": [],
    "injectionSelector": "L:text.xml",
    "patterns": [
        {
            "include": "#xml-code-block"
        }
    ],
    "repository": {
        "xml-code-block": {
            "begin": "(<)(Declaration|ST)(>)\\s*",
            "end": "\\s*(<\\/)(\\2)(>)$",
            "beginCaptures": {
                "1": {
                    "name": "punctuation.definition.tag.xml"
                },
                "3": {
                    "name": "punctuation.definition.tag.xml"
                },
                "2": {
                    "name": "entity.name.tag.localname.xml"
                }
            },
            "endCaptures": {
                "1": {
                    "name": "punctuation.definition.tag.xml"
                },
                "3": {
                    "name": "punctuation.definition.tag.xml"
                },
                "2": {
                    "name": "entity.name.tag.localname.xml"
                }
            },
            "patterns": [
                {
                    "begin": "\\s*(<!\\[CDATA\\[)\\s*",
                    "end": "\\s*(\\]\\]>)",
                    "beginCaptures": {
                        "1": {
                            "name": "string.unquoted.cdata.xml"
                        }
                    },
                    "endCaptures": {
                        "1": {
                            "name": "string.unquoted.cdata.xml"
                        }
                    },
                    "contentName": "meta.embedded.block.iecst",
                    "patterns": [
                        {
                            "include": "source.st"
                        }
                    ]
                }
            ]
        }
    },
    "scopeName": "xml.iecst.codeblock"
}

2. Add to project.json

"grammars": [
    {
        "scopeName": "source.st",
        "language": "st",
        "path": "./syntaxes/st.tmLanguage.json"
    },
    {
        "scopeName": "markdown.iecst.codeblock",
        "path": "./syntaxes/md.codeblock.json",
        "injectTo": [
            "text.html.markdown"
        ],
        "embeddedLanguages": {
            "meta.embedded.block.iecstmd": "st"
        }
    },
    {
        "scopeName": "xml.iecst.codeblock",
        "path": "./syntaxes/xml.codeblock.json",
        "injectTo": [
            "text.xml"
        ],
        "embeddedLanguages": {
            "meta.embedded.block.iecstxml": "st"
        }
    }
],

Everything works fine except thing I cannot understand.

Look somehow second declaration tag does not work and if I change PROTECTED key it works. If I change LIBRARY to FUNCTION_BLOCK for example it works.

I have no idea where to dig. Could you please help me?