grammar for a language that is indent-sensitive

Lezer
1

Hi,

I am trying to write a simple parser (link to repo) for a language that is indent-sensitive like python. However, I fail at the indentation tracking.
The grammar rules are specially modeled on the python rules. And the external tokenizer is also almost copy and pate but with a few cuts (if I use the custom tokenizer in python, the indentation tracking still works).

for the following example I get this syntax tree (generated with)

test0| val:20
test1| val:asdf
test2| 
  val:21
test3| val:22

image
image460×708 21.3 KB

Any ideas what I’m doing wrong?

2

Nothing jumps out. Try building with --names and setting the environment variable LOG=parse to see what, exactly, the parser is doing.

3

Thanks for the answer. LOG=parse does not provide any additional output although I used npx with --names. LOG=lr works unfortunately I don’t see how this could help me.

Am I doing something wrong? Is there any other way to find the error?

4

LOG=parse should always produce output. But you must provide it when parsing, not when generating the parse tables.

5

Oh ok thanks for the information. I just don’t understand how I should provide the system variable in this case. And how do I then display the logging result? I couldn’t find anything regarding this online.

6

Run your parse in node, and start it LOG=parse node myscript.js.

7

I’m sorry I don’t really understand what you mean by that. Is there also a good possibility for vue cli projects?

8

I finally found the problem: the problem was the wrong order of “@tokens, @context, @external tokens”. It seems that the external tokens must be mentioned before the tokens.