Overlapping tokens

Lezer
1

Hello there.
I am writing a custom grammar like as follows

@tokens {
  Identifier { $[a-zA-Z_\-0-9]+ }

  Boolean { "true" | "false" }
  
  ...

}

then throw error when running “rollup -c”, as follows

 Overlapping tokens Identifier and Boolean used in same context (example: "f" vs "false")

can someone tell me how can i solve this issue

2

When the input stream the word true or false, it doesn’t know whether to treat that as an identifier or a boolean. You could add @precedence { Boolean, Identifier } to your @tokens block to tell it to pick booleans in that case. But that leaves you with another problem — input like truer would match a boolean and then an identifier. The way to handle keywords like this is usually to use the @specialize syntax, which, after a token has been parsed, changes its type if it matches a given string. Outside of your tokens block, you could say something like:

Boolean { @specialize<Identifer, "true" | "false"> }

That should work better.

1 Like
3

Thanks a lot, it works well.
I also have another issue. How can I dynamically match a keyword when its targetList changing?
I tried but our Tokens are looks like static so cant find a way to match the dynamic targetList.

4

How does it change? Dynamically changing parsers are not supported (incremental parsing wouldn’t work if the way the input is parsed isn’t stable). But if you can treat this as a parameter to parser configuration, you could create an external specializer that does the lookup, and use the specializers (new in @lezer/lr 1.2.0) option to replace it when configuring the parser.

5

I’m intrigued by this new specializers option and am curious if there’s a usage example available. I was able to find an example of the related tokenizers support in the SQL language package; using the same technique for the specializers leads me to a replacement loop in the code containing a find that I can’t seem to make anything, well, found in.