What different between inside and outside @tokens?

[lzxgh]

Parse content:

(test),(test

Fisrt grammer:

@top { statement+ }

statement { Test | Content }

Test { "(" Content ")" }

@tokens {
    Content { ![\n()]+ }
}

Parsed result:

Second grammar:

@top { statement+ }

statement { Test | Content }

@tokens {
    Test { "(" Content ")" }

    Content { ![\n()]+ }
}

Parsed result:

Why first grammar parse “(test” as Test node?

[marijn]

Tokens are always ‘atomic’, even if they use capitalized sub-rules. So in the second grammar, Content doesn’t even show up as a term in the generated parser, it’s conceptually inlined into Test.

[lzxgh]

Ok. But I still don’t understand why it parse "(test" as Test node in first grammar, "(test" is clearly missing ")" .

[marijn]

That’s why there is an error node after the Content node—the error correction broke off the Test rule early because it couldn’t produce a valid parse. Pass {strict: true} if you want the parser to throw an error in cases like that.

[lzxgh]

I get it now. Thank you.