How to get access to tokens that were consumed when a specific grammar production matched?

Lezer
1

I am working on a parser for Wolfram language. It has Out operator (Out—Wolfram Documentation), which can take different forms, including %%…% (% repeated arbitrary number of times). I can write %, or %1, %%%, etc.

I have:

@tokens {
  backref { "%" @digit+ }
  ref { "%"+ }
  @precedence { backref, ref }
}
...
Out { backref | ref }

After I get a syntax tree, I traverse it for my purposes:

    tree.iterate({
        enter: (node: SyntaxNodeRef) =>...)
    });

Inside enter I receive the current syntax node. How do I get to the relevant tokens? In my case I need to extract the number from backref token or count the length of the ref token.

2

Because tree nodes only store their extent and type, getting information (such as identifier name, literal value, or in this case, reference number) involves fetching that range from the document text and looking at its content.