Requiring different characters for a token based on context

Lezer
1

I’m working on putting together syntax highlighting for a language. However I’m running into a little bit of an issue on doing a custom lexer. I need to have the language store several characters in the context object and be able to check their values when scanning a token to determine if a later character is the same before accepting the token. Is there an easy way to do that?

| :name: some text | * something * another thing

| - another name - some more text |

the language I’m doing is a rule based language with delimiters between rules and names of individual things that are matched. The | character is a delimiter that the parser needs to store once at the very beginning of the file and can change, but stays the same throughout a code file. The other special characters : * and - are swapped out every time the first delimiter is found, but stay the same up until that point.

2

For a little more context on exactly what I’m asking, I want to be able to get the actual text of the last token when shift is called on the context.

3

You can make an external tokenizer emit different tokens for a character depending on context, which might somehow help here. But depending on the details it may be that this language isn’t context-free enough to parse with Lezer.

4

hmm, yeah. It seems like its definitely a little too context sensitive to easily do it. The shift and reduce functions on the context seem to only give the token id so unless there’s something else in the api to get the actual text I can’t find any way of getting around it.