Overlapping tokens

Hello there.
I am writing a custom grammar like as follows

@tokens {
  Identifier { $[a-zA-Z_\-0-9]+ }

  Boolean { "true" | "false" }


then throw error when running “rollup -c”, as follows

 Overlapping tokens Identifier and Boolean used in same context (example: "f" vs "false")

can someone tell me how can i solve this issue

When the input stream the word true or false, it doesn’t know whether to treat that as an identifier or a boolean. You could add @precedence { Boolean, Identifier } to your @tokens block to tell it to pick booleans in that case. But that leaves you with another problem — input like truer would match a boolean and then an identifier. The way to handle keywords like this is usually to use the @specialize syntax, which, after a token has been parsed, changes its type if it matches a given string. Outside of your tokens block, you could say something like:

Boolean { @specialize<Identifer, "true" | "false"> }

That should work better.

1 Like

Thanks a lot, it works well.
I also have another issue. How can I dynamically match a keyword when its targetList changing?
I tried but our Tokens are looks like static so cant find a way to match the dynamic targetList.

How does it change? Dynamically changing parsers are not supported (incremental parsing wouldn’t work if the way the input is parsed isn’t stable). But if you can treat this as a parameter to parser configuration, you could create an external specializer that does the lookup, and use the specializers (new in @lezer/lr 1.2.0) option to replace it when configuring the parser.

I’m intrigued by this new specializers option and am curious if there’s a usage example available. I was able to find an example of the related tokenizers support in the SQL language package; using the same technique for the specializers leads me to a replacement loop in the code containing a find that I can’t seem to make anything, well, found in.