Token end out of bounds with parseMixed

I’m working on a language similar to the Twig example in the mixed-language docs. I keep getting Uncaught RangeError: Token end out of bounds errors.

I extended the Twig example with comment tokens to isolate the problem. See lang-twig.

It’s the weirdest thing. When the comment is indented, it works as expected:

But if the comment is not indented then it throws the RangeError. I’m stumped.

@marijn I’d love your help if you have a sec to look at lang-twig. I know your time is precious. :pray:

This seems to have been a bug in the handling of input ranges by the tokenizer in @lezer/lr. Below patch should help.

Brilliant, that fixed it! Thank you so much. I’ll write some tests in my language package to be certain I’m not missing any cases. Is there a way to unit test the final tree when using parseMixed with overlays? Like this lezer-parser/html mixed test but for overlay nesting as opposed to hierarchical nesting?

e.g. a mixed test something like this

I’ve been testing overlays manually using resolveInner so far.

Shoot, the lezer update addressed that simple overlay case but causes big issues (missing highlighting and browser crashing) for other overlay cases. I’ve updated lang-twig with examples and a failing test to show what I mean.

Indeed, I messed something up. I’ve reverted that patch and pushed this one instead.

That fixed it, thank you Marijn!