How do I add custom directives? about commonmark

[philsturgeon]

HA.

Yeah so my concern here is that whatever sort of extensibility thing ends up getting done could end up conflicting with whatever directives, optional syntax or multiple levels of CommonMark ends up getting introduced.

I suspect we could just implement some sort of super vague addExtension() method, which would just provide a few new rules to the existing parser, then wrap that up with convenience methods and hide it depending on whatever the CommonMark folks end up doing at a later date?

This would basically get us tables pretty quickly.

[colinodell]

There are four extension points we'd probably need to expose:

• Detecting and parsing block elements
• Detecting and parsing inline elements
• Finalizing blocks
• Rendering elements found in the AST

I'm thinking it would be best to extract the current block/inline element rules to use whatever extension structure is implemented.

[philsturgeon]

Continue this conversation on the new repo https://github.com/thephpleague/markdown/issues/4

[aleemb]

Came here via @philsturgeon blog post on state of markdown. The repo linked in the previous comment is dead. Is this discussion on-going somewhere or has extension support been added?

Allowing extension support would be wonderful. Hope this issue can be re-opened.

[colinodell]

We're consolidating everything into this repo as it's the new home for the project.

I'm currently working on a major refactoring to decouple the directives from the parser engine (and reduce the general complexity of everything). Each directive will essentially have it's own parser, AST representation, and corresponding renderer. You should be able to add your own directives by extending/implementing the same base classes and registering your parsers just like the core engine does.

This should be (mostly) completed in the next week or so. I'll ping this thread once the code is up.

[aleemb]

So relieved to hear extensibility is being addressed, it will make the switch that much easier. I am in a similar boat as @josegonzalez but using http://github.com/chobie/php-sundown which exposes block/inline level calls as described at https://github.com/vmg/redcarpet.

I really wish extensibility was also standardised via CommonMark, for example through the approach you mentioned (or similar to all sundown/redcarpet implementations).

[philsturgeon]

It would be good to see how CommonMark plan to address this formally before we just march on and implement whatever the heck we like.

The discussions have dried up a little, but there are a few threads. Jeff Atwood said if I wanted the extension/directives conversation to progress then I should get involved and bump it, but I don’t have the time right now.

If anyone wants to do that then head are over to any of these:

1. Optional Syntax
2. Multiple levels of CommonMark specification
3. Generic directives/plugins syntax

[colinodell]

I definitely don't want to add anything that adds, modifies or relies on specific syntax (like this).

However, I am open to exposing the methods used to register parsers and renders. Here's a rough example of what that might look like:

    class YoutubeParser extends BaseParser
    {
        public function matches(Cursor $cursor)
        {
            return $cursor->startsWith('!youtube');
        }

        public function parse(Context $context)
        {
            $link = $context->get('LinkParser')->parse($context);

            return new YoutubeElement($link->getUrl());
        }
    }

    class YoutubeElement extends LinkElement
    {
    }

To use it, you'd just do something like this:

    $cm = DocParser::create();
    $cm->registerParser(new YoutubeParser());
    $cm->parse(file_get_contents('whatever.md'));

Or something to that effect. We wouldn't be adding the new directives ourselves, just allowing others to utilize the same functionality that our CommonMark directives use:

    class DocParser
    {
        public function registerParser(ParserInterface $parser)
        {
            ...
        }

        public static function create()
        {
                $cm = new static();
                $cm->registerParser(new League\CommonMark\Parser\LinkParser());
                $cm->registerParser(new League\CommonMark\Parser\ImageParser());
                $cm->registerParser(new League\CommonMark\Parser\EmphasisParser());
                $cm->registerParser(new League\CommonMark\Parser\ListParser());
                // ...

                return $cm;
        }
    }

Thoughts?

[josegonzalez]

That would be fine with me.

[philsturgeon]

Nice, thats kinda what I was getting at before with my comments about making a basic PHP API to add in super generic additions to the parser logic, then if CommonMark comes up with something format we can wrap it in more specific convenience methods.

I like the solution you have proposed, but I’m not sure how performant it will be to have this many directives. This is another one of those situations where “Nice OOP” v “Speed" are at odds.

[colinodell]

I like the solution you have proposed, but I’m not sure how performant it will be to have this many directives. This is another one of those situations where “Nice OOP” v “Speed" are at odds.

Great point. I'm hoping it won't be too bad though - I'm essentially just replacing switch statements with a loop over the registered objects array.

If anything, I'd expect to see a performance decrease from "decomplexifying" the core parts - things like using a cursor to track the string position instead of scattered variables. OTOH, it should be much easier to optimize to simpler code.

I guess we'll see what the benchmarks have to say :)

[philsturgeon]

Yeah sounds good, post em up!

Those massive switch statements had to go, and I hope you’re right. Zend Optimizer to the rescue? :)

-- 
Phil

From: colinodell [email protected]
Reply: thephpleague/commonmark [email protected]>
Date: December 8, 2014 at 5:17:12 PM
To: thephpleague/commonmark [email protected]>
Cc: Phil Sturgeon [email protected]>
Subject:  Re: [commonmark] How do I add custom directives? (#11)

I like the solution you have proposed, but I’m not sure how performant it will be to have this many directives. This is another one of those situations where “Nice OOP” v “Speed" are at odds.

Great point. I'm hoping it won't be too bad though - I'm essentially just replacing switch statements with a loop over the registered objects array.

If anything, I'd expect to see a performance decrease from "decomplexifying" the core parts - things like using a cursor to track the string position instead of scattered variables. OTOH, it should be much easier to optimize to simpler code.

I guess we'll see what the benchmarks have to say :)

—
Reply to this email directly or view it on GitHub.

[colinodell]

Initial results are in!

Our latest stable release clocks in at 36ms and my refactored branch is at 81ms - twice as slow, but still within reason. (I think most of the added time is due to the new abstractions)

It's definitely much easier to identify areas of optimization now, so that's the good news :)

I'll finish cleaning my branch and push it up sometime this week.

[cebe]

I think most of the added time is due to the new abstractions

have you tried a profiler tool like xhprof? It can show you where your code is spending most of the time.

I have done that for cebe/markdown https://github.com/cebe/markdown/blob/master/tests/profile.php
It is a bit clumsy to handle but it helps a lot in understanding what is going on and what you can improve.

[colinodell]

Yep, I'm currently using Blackfire.io which has been immensely helpful. I was able to get from ~250ms to ~80ms so far, and I still have some good candidates for further optimization.

Comparing profiles from both branches was less helpful than I hoped, but it makes sense considering the current release lumps parsing into giant switches instead of separate, profileable methods/classes. That's okay though, because I'm already getting valuable feedback about slow spots, so I'll just keeping working on those.

[colinodell]

I've pushed my code to the refactor branch: https://github.com/thephpleague/commonmark/tree/refactor

Some points of interest:

• Each directive now has it's own:
  • Parser (goodbye ugly switch statements!)
  • Unique class representing it in the AST
  • AST to HTML renderer
• Directive parsers and renderers are registered in the Environment class
  • Environment::createCommonMarkEnvironment() pre-registers everything you need for CommonMark compliance.
  • Custom directives can be added using the exact methods we use internally
• The block and inline engines use context classes to maintain their state
• Inline parsing uses a separate Cursor class to advance through the current text

I'd greatly appreciate any feedback or suggestions! Especially with the class structure, public methods, and names of things.

[colinodell]

And it looks like scrutinizer-ci really likes the refactored code: 6.14 => 9.04!

[philsturgeon]

Ha, I'm glad to see Scrutinizer is happy now. I spent quite a bit of time trying to tart up what was there but those massive switches were really the crux of the issue. Now they're gone I'm not a bit surprised that code quality is looking better.

I'd say that performance hit is probably reasonable enough. It should try to get down to ~50 somewhere for v1.0, but right now functionality and code quality is definitely more important than pure blind speed.

Get this branch into master and start on to 0.5.0. :)

[colinodell]

Merged :)

[aleemb]

Can't wait to try it out :)

With this fix, I am pretty sure this project is now the de facto markdown implementation in PHP.

[colinodell]

I've added a simple example of how you'd go about adding custom directives: https://github.com/thephpleague/commonmark#customization