FencedCode like structures and composite elements

Lezer
1

I am trying to add math parser to @lezer/markdown and use the latex parser with parseMixed to parse it afterwards just like is done with FencedCode.

In particular the block display math that spans multiple lines and behaves similarly to codeblocks (following mdast-util-math definition). Block display math looks like the following

$$
E = mc^2
$$

or

> $$
> E=mc^2
> $$

From looking at the code of @lezer/markdown it uses for (const m of line.markers) markers.push(m);

to add composite markers back in, but from the public api I couldn’t find a way to add those markers back. Is there a way to do this? or should the internal api not be touched?

2

What exactly does ‘add those markers back’ mean here?

For a leaf node like this (which contains no Markdown blocks inside it), you’ll want to use BlockContext.addLeafElement to emit its tree structure.

3

just like FencedCode, display math is not a leaf parser/block? at least couldn’t figure it out how to make it a leaf parser as $$ can disrupt the paragraph thus I check for that in endLeaf but that causes the $$ to not be included in the leaf. Also other implementations use parse and not leaf so I was thought that wasn’t possible?

Here is my broken implementation using the leaf parser for reference.

(in pandoc blank lines are not allowed but in mdast-util-math they are, but more importantly for a better editor experience they are not worth failing over and thus removing the math preview tooltip, concealment, etc)

Reason for putting the quotemarks in

with add those markers back I meant since they are already parsed and would have been in the tree if not for my construct?

I am specifically targeting the quotemarkers, such that when a newline in a lsp-like snippet is inserted, the appropriate level of qoutemarkers is continued.

currently I am getting the qoutemarkers with regex, but I am naively assuming they are well constructed, so I was (naively) hoping this would solve that, but just like the blankline case its probably not worth failing over.

Instead its probably better to count the amount of blockqoute parents and continue appropriate level that way.

examples of edge cases

Just like FencedCode (using pandoc)

>$$
>>E=mc^2
>$$

results in

<blockquote>
<p><span
class="math display"> &gt; <em>E</em> = <em>m</em><em>c</em><sup>2</sup></span></p>
</blockquote>

like how

>```
>>text
>```

results in

>text

and then an example of an edge case that’s probably not worth failing over: (splits blockquote in both parser and pandoc)

>```

>text2
>```

or (splits blockquote only in parser but not in pandoc)

>```
>
text
>```

and this only targeting blockquotes, I am well aware that my parsing is kinda broken/incorrect for lists, like

1. $$
2. first
$$
second

the first $$ are not a displaymath delimiter, but I am treating it as such.

4

FencedCode is a leaf block, and unless I’m missing something this math syntax is too. You cannot have emphasized text or new Markdown block structure inside it, can you? The meaning of the content inside the dollar sign delimiters is entirely determined by the LaTeX syntax, just like a fenced block treats it as plain code.

Leaf blocks can interrupt paragraphs via endLeaf.

5

oh yes, I meant I can’t make use of BlockParser.leaf to parse it, I am currently using BlockParser.parse and BlockParser.endLeaf together to parse it, but inside BlockParser.parse you can’t use BlockContext.addLeafElement only BlockContext.addElement as you don’t have the LeafBlock object.

or am I misunderstanding something? Because from the description of endLeaf I am assuming its not meant for that (also not sure how that would work)

6

That’s what doesn’t sound right. I am not going to debug the code you linked but if you can describe here what’s going wrong, maybe we can figure out the problem.

7

BlockParser.leaf isn’t used for FencedCode so how does that sound wrong?

given a simple end leaf

const endLeaf = (_,line) => /^\${2,}[^$]*$/.test(line.text.slice(line.pos))

then using BlockParser.leaf, the parser doesn’t have access to the full equation as its not one leaf (and it needs the full equation) .

without endLeaf, it has the same problem.

(don’t think startComposite would make but just trying all options I can think of)

and maybe with BlockContext.startComposite and parse the inside with BlockParser.leaf? but the parser can’t end the composite correctly as for

const composite = (cx: BlockContext, line: Line, delimLength: number) => {
	if (isDisplayBlockEnd(line, delimLength) !== null) {
		return false
	}
	return true
}

would end it before the line not after, causing the closing $$ to be another open $$.

8

Apologies, I got the terminology mixed up. A block like this is called a leaf block in the interface’s vocabulary, but it is indeed parsed with parse, not leaf. Here’s a (very crude) parser for blocks like this. Does that help?

9

Parsing the delimiters isn’t the problem? It’s that the QuoteMark are not in the final syntaxtree inside the math environment, whereas they are inside the Fencedcode cause it adds the markers from line.markers but that’s not public API.

It’s working with and without using that internal API, but when I don’t use it, the quote marks inside the math environment don’t end up in the tree.

Maybe skipping the focus on the math implementation, how would I parse FencedCode the same way with composite markers that are skipped for the element CodeText and are put in front of it, without accessing line.markers.

As without i currently get (dont expect you to debug it)

Blockquote: (1, 19): 
      QuoteMark: (1, 2): 
      DollarDisplayMath: (2, 19): 
        Dollar: (2, 4): 
        DisplayMath: (6, 11): 
        DisplayMath: (12, 16): 
        Dollar: (17, 19): 
    Blockquote: (21, 48): 
      QuoteMark: (21, 22): 
      FencedCode: (22, 48): 
        CodeMark: (22, 25): 
        QuoteMark: (26, 27): 
        CodeText: (28, 33): 
        QuoteMark: (33, 34): 
        CodeText: (35, 43): 
        QuoteMark: (44, 45): 
        CodeMark: (45, 48):

For the text


>$$
>a+bj
>c+d
>$$

>~~~
> text
> mor text
>~~~

With

Blockquote: (1, 19): 
      QuoteMark: (1, 2): 
      DollarDisplayMath: (2, 19): 
        Dollar: (2, 4): 
        QuoteMark: (5, 6): 
        DisplayMath: (6, 11): 
        QuoteMark: (11, 12): 
        DisplayMath: (12, 16): 
        QuoteMark: (16, 17): 
        Dollar: (17, 19): 
    Blockquote: (21, 48): 
      QuoteMark: (21, 22): 
      FencedCode: (22, 48): 
        CodeMark: (22, 25): 
        QuoteMark: (26, 27): 
        CodeText: (28, 33): 
        QuoteMark: (33, 34): 
        CodeText: (35, 43): 
        QuoteMark: (44, 45): 
        CodeMark: (45, 48):
10

Okay, wow, looking closer I realize how infuriatingly useless my answers in this thread so far have been. I guess I sometimes have trouble knowing when to move from ‘steer someone in the right direction without wasting my own time’ mode to ‘actually think about what this person is asking’ mode. Re-reading your messages you did actually describe the problem clearly. I just didn’t read what you wrote.

And I think your diagnosis is correct, there’s a piece of API missing for doing this kind of thing, where you need to integrate line markers into your node’s tree.

On the one hand, it would be convenient if this could be largely automatic, to make it harder to accidentally drop these. injectMarks is already defined, and could make this pretty easy. On the other hand, there currently isn’t really anything in the system with the right scope to track these, and it’s probably simpler, if a little more cumbersome, to have block parsers integrate these in their output themselves.

Does exposing Line.marks in the public interface sound like a workable solution to you?

11

yes

block parsers already have to “decide” to use Line.next and Line.pos so having Line.marks also be explicit would fit right in.

(I am in no way familiar with all the markdown “flavors” or the broken commonmark derivatives, to say if dropping composite markers would be correct or not, but I assume there probably is a flavor like that)

no worries, thank you for time (maybe I should have started with the syntaxtree but wanted to keep it compact)

12

Great. I’ve tagged 1.7.0 with this change.