cx.addElement not adding elements, depending on order of addition of the elements.

bxff · July 21, 2022, 5:07pm

cx.addElement from the inline parser of the Markdown language, doesn’t add elements depending of the order they are added, it doesn’t give any errors, and I couldn’t find any thing in the reference documentation that would reason with this.

In this case, where I first add a Element from 0 to 7, and then add a Element from 1 to 6, it seems to work as intended.

In this case, where I first add a Element from 1 to 6, and then add a Element from 0 to 7, it doesn’t seem to add the second(from 0 to 7) element.

I think so this is a bug, if not, what is the best way to add a bunch of elements to the inline parser?

marijn · July 27, 2022, 10:17am

Why are you adding overlapping elements? Could you say a bit more about what you are trying to do when this happens?

bxff · July 27, 2022, 12:11pm

Well, I am trying to recreate the emphesis parser to parse incompletely, as in if there is a starting delimiter, and no ending delimiter it would still parse the whole section(paragraph): Older topic I created on this matter, which goes into depth on what I was trying to achieve.

The way I am achieving this is by recreating the current emphasis parsing system to fit my needs, the current parser first incorrectly parses the delimiters, which then gets correctly parsed inside of resolveMarkers at the end of the parser cycle.

I have a similar system, where I recreate this whole parsing system inside of the extension architecture of the lezer markdown parser, and modify this system to fit my needs. At the end I am left with a list of elements which I need to then add the to parser using cx.addElement. And trying to add these elements caused the problem.

This doesn’t seem to be a problem in the current default parser as resolveMarkers replaces the element list after everything is parsed, rather than adding these elements at post as I am trying to(I can’t find exactly where this happens, but resolveMarkers is required for parsing to correctly work, and it is referenced inside of parseInline which itself is is referenced inside of finishLeaf, where I am guessing this really happens):

github.com

lezer-parser/markdown/blob/1c43d7a141efd4e1c9c70d869e5e8a2beb378508/src/markdown.ts#L1224-L1236


      
            parseInline(text: string, offset: number) {
              let cx = new InlineContext(this, text, offset)
              outer: for (let pos = offset; pos < cx.end;) {
                let next = cx.char(pos)
                for (let token of this.inlineParsers) if (token) {
                  let result = token(cx, next, pos)
                  if (result >= 0) { pos = result; continue outer }
                }
                pos++
              }
              return cx.resolveMarkers(0)
            }
          }

marijn · July 27, 2022, 1:01pm

So could it be that the 1-6 element should be a child of the 0-7 element?

bxff · July 27, 2022, 1:37pm

Hmm… Yeah, it most likely is the case which I haven’t accounted for. At-least in the default parser everything is wrapped inside of each other which I didn’t account for as my implementation had overridden this part, in favour of the requirements. Thanks a lot.

Also could I use injectMarks, to insert these marking together, seems like this is meant to do this exact thing, but I can’t say for sure. Or would I create a custom function to do this:

github.com

lezer-parser/markdown/blob/1c43d7a141efd4e1c9c70d869e5e8a2beb378508/src/markdown.ts#L1728-L1743


      
          function injectMarks(elements: readonly (Element | TreeElement)[], marks: Element[]) {
            if (!marks.length) return elements
            if (!elements.length) return marks
            let elts = elements.slice(), eI = 0
            for (let mark of marks) {
              while (eI < elts.length && elts[eI].to < mark.to) eI++
              if (eI < elts.length && elts[eI].from < mark.from) {
                let e = elts[eI]
                if (e instanceof Element)
                  elts[eI] = new Element(e.type, e.from, e.to, injectMarks(e.children, [mark]))
              } else {
                elts.splice(eI++, 0, mark)
              }
            }
            return elts
          }

Lastly, I created a pull request a while back, didn’t want to create another topic for it as its something you don’t recommend, but maybe you can take a look into it, again thanks a lot, this was very helpful: Adds internal documentation for `resolveMarkers` and fixes typos of `takeContent`. by bxff · Pull Request #18 · lezer-parser/markdown · GitHub