Tokens containing an underscore?

Lezer
1

Hello, we are writing a PostgreSQL esque grammar that has tokens, that have underscores in them. For example, is a token_like_this possible? When we defined tokens such as that, it does compile but when we actually try to use that token in the editor, it is being clubbed together as a single identifier. This also happens if we try to join three different tokens with an underscore. Like (tokenA_tokenB_tokenC) will also get clubbed as an identifier, while (tokenA-tokenB-tokenC) will be treated as three different tokens. Is this expected behaviour? If not, is there a way to solve this?

Also is it possible to have numbers in tokens? like (tokenA_2 or tokenA-2)?

2

There is nothing special in the way Lezer treats underscores or numbers. If you allow underscores in your identifiers, then I would expect token_like_this to be a single token, yes.

1 Like
3

Thank you!