External tokenizers precedence vs tokens?

Lezer
1

In Lezer grammars, we define tokens with precedence in relation to each other via the following example:

@tokens {
   // some tokens here...
  @precedence { token1, token2, token3 }
}

I was wondering how external tokenizers have their precedence set in relation to these tokens. Is there a way to configure them as higher/lower precedence than the regular tokens?

2

The order of the @tokens / @external tokens blocks determines which gets tried first. Because the parser cannot automatically determine whether external tokens overlap with any other tokens (they are just opaque functions), it can’t warn about overlap, and will just run tokenizers in order until one matches a token.

1 Like