i’m no expert, but i think what you might need is a parsing algorithm like the markdown one. as you can see, it’s not defined with a .grammar file, because it’s just… not possible due to its complexity (i.e., the need to look ahead like you described).
while browsing this forum earlier, i found a thread confirming that markup languages are either super difficult or downright impossible with lezer grammar.
though from what i can gather, you’re not making a markup language (lucky you :P), i think it’s worth a shot. the more hoops you have to jump through with .grammar files, the more you have to wonder if it’s the right way to go about things.
or… maybe all you need is an external tokenizer / specializer. it worked for me for a bit until i ended up needing to create a makeshift state machine that was simply an array with elements unshift()ed into it to see which token was most recently accepted hah.
sorry that i don’t have any definitive answers, i’m still trying to comprehend it myself.
1 Like