Comments (5)
For context: after a fenced code block, the empty line is indeed tokenized as a BLOCK_EMPTY with a TEXT_NL token inside, so I guess the "smearing together" of indented code lines into a cohesive block across empty lines is being too greedy at the bottom edge
By "smearing together" I mean that these 3 lines become 1 cohesive block even though the middle line doesn't even have indentation characters (␣
denoting a space):
␣␣␣␣line 1 of code
␣␣␣␣line 3 of same block
from multimarkdown-6.
I'm struggling to understand under what circumstances this would matter?
The token tree does not describe the given source text (that is "described" by text itself), it is an abstract representation of the semantic meaning of the text. There are circumstances where a BLOCK_EMPTY
has meaning (in the sense that it splits something else into two or more parts) and circumstances where it does not (an indented code block ends when there is a non-indented line, whether there is an intervening blank line or not.) This means that there are circumstances where one or more empty lines are assumed, and times where they are explicitly noted -- and this often has to do with technical limitations of the parser, and sometimes just because I happened to code it differently. This includes the fact that I don't always explicitly waste the CPU cycles to remove a token that is no longer needed and sometimes just mark it as an empty token and allow it to be freed later.
Either way, a BLOCK_EMPTY
token results in NULL output, so its presence or absence in the token tree doesn't really matter, since the parsing has already been completed at this point.
Perhaps you could explain why this came up?
from multimarkdown-6.
Thanks for the explanation! I wanted to report this because it looked odd that the token tree's description had gaps: If the code block continued one line further down, I would have just thought that this is MMD6's way to interpret the situation. The gap looked suspicious and I didn't notice any mentioning of intentional gaps in the headers or source files, so I figured you might not be aware of this as well.
So this is a "wontfix because intentional" situation :)
from multimarkdown-6.
Or maybe a "quasi-intentional" situation and will consider fixing if there is a situation where it actually matters.
:)
from multimarkdown-6.
(But thank you for noticing!!)
from multimarkdown-6.
Related Issues (20)
- How to correctly use transclusion? HOT 4
- The first line containing a : (colon) is always treated as metadata HOT 5
- Problems with transclusion for streamed input on Windows^TM^ HOT 1
- [Feature request] File transclusion on remote raw sources HOT 4
- [Question] After stripping of MARKER_BLOCKQUOTE in tokenization, could we add back MARKER_BLOCKQUOTE when the line/block is finished? HOT 1
- Automatic abbreviation processing is not reliable HOT 2
- Export from Ghostwriter via Multimarkdown 6.6.0 to odt format fails when image is included HOT 4
- Missing Image Condition for ODT and Epub Could Be Improved HOT 6
- Broken Link to single document PDF on the GH-Pages site HOT 1
- Tables in lists with leading blanks don't render in MMDComposer5 HOT 3
- Build warning and question on directory for option HOT 4
- Please consider adding option to hide Markdown source and search windows in MMDComposer HOT 3
- Bold around a link doesn't display properly (in MultiMarkdown Viewer v5.0.0 (2023.06.08.16.13) HOT 1
- Underscores in LaTeX autorefs Are Unnecessarily Escaped HOT 1
- Image won't display in MMDComposer5 Version 5.0.0 (2023.11.03.03.02) HOT 1
- Grammar railroad diagram HOT 1
- File transclusion happens even when inside a html comment. HOT 1
- Install failed on macOS Sonoma - How to uninstall completely HOT 1
- Tokenizing empty fenced code block doesn't apply codeFenceLine to opening fence line HOT 1
- token_first_child_in_range, token_last_child_in_range, token_child_for_offset limited at the end of the document HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from multimarkdown-6.