always-reduce vs @skip

Lezer
1

I started with the indentation example, but scaled back @skip considerably. my external margin token is the only thing left:

@external tokens xt from "./support.ts" { dedent indent margin }
@skip { margin }

this seemed to be working well until I added a “this is just unparsed text” block:

Text { indent Line+ dedent }
@tokens { Line { ![\n]+ "\n" } }

it goes into a (via always-reduce Line+) loop and slurps up all the remaining input. I see a workaround (a fourth external could grab the whole text block in one call). but it feels like this greedy always-reduce tight loop should only ever be entered if @skip{/*never*/} because skipped things might show up anywhere… like between the Lines (sorry, couldn’t resist).

I’m new to lezer so maybe I’ve missed something (would appreciate pointers), but this behavior was surprising and confusing, so thought it worth mentioning.

2

Sounds like your dedent token is somehow not being generated. Is the @external tokens declaration before the regular @tokens one? Any reason the external tokenizer wouldn’t fire for the line where it should end the Text block?

3

I appreciate your desire to help fix my grammar. I saved a commit and will find some time to go back and investigate. my gut says you’re right that token precedence is a problem.

until then I want to say that my motivation for posting was partly about messages produced by @lezer/generator, not anything specific to my grammar. a warning along the lines of “skip tokens {X, Y} can’t happen inside rules {Line+}” might be worth considering? it certainly could be intentional - a few authors might need to opt in to that effect, so leave the door open for that. but I imagine that most of the time people write @skip with the expectation that those tokens can happen anywhere.