Can child/sub tokens appear in the parse tree?

1

I feel like the answer is “no” but I have to ask just to make sure.

This grammar and input yield the following the parse tree:

@top Program { Either* }

@tokens {
  Letter { @asciiLetter+ }
  Number { @digit+ }
  Either { Letter | Number }
}
---
a1b2
---
Program (a1b2)
	Either (a)
	Either (1)
	Either (b)
	Either (2)

Is there a way to have this be the parse tree without moving Either out of @tokens?

Program (a1b2)
	Either (a)
		Letter (a)
	Either (1)
		Number (1)
	Either (b)
		Letter (b)
	Either (2)
		Number (2)

In my non-toy-language, moving that token out of @tokens causes additional complexity I’d love to avoid (@skip, ExternalTokenizer, etc.)

2

No. Tokens are the atomic elements in the parse tree. No structure is recorded inside them.

1 Like