external tokens are not attached to tree

Lezer
1

Hello, I am using a custom tokenizer for a single token (aliasTextOrSymbol) but the tokens it returns to do not appear in the resulting syntax tree:

rql.grammar

tokens.js

I have confirmed that the code is called and calls acceptToken. The examples of input and the resulting syntax tree suggest that the call to acceptToken does not yield a token to the tree. The purpose of the aliasTextOrSymbol token is to determine whether the text inside the alias parenthesis is Text or a Symbol.

[Highlight red & Underline blue] (Hr Ub)
----------------------------------------------
Document
·Transform
··Operation
···Symbol<Highlight>
···Symbol<red>
··Operation
···Symbol<Underline>
···Symbol<blue>
··Alias
···⚠<>
·Block
··Reference
···Select
····Symbol<Hr>
····Symbol<Ub>
····⚠<)>

[Highlight red] (HR)
----------------------------------------------
Document
·Transform
··Operation
···Symbol<Highlight>
···Symbol<red>
··Alias
···⚠<>
·Block
··Reference
···Select
····Symbol<HR>
····⚠<)>

Thank you,
Nathan

2

The first thing that I would try is give the token a name:

@external tokens aliasTextOrSymbol from "./tokens" {
  aliasTextOrSymbol[@name=aliasTextOrSymbol] 
}
3

Just like other terms, external tokens are only included in the tree when they start with an upper-case letter (or are assigned an explicit @name).

4

I still can’t make sense of this. I updated the grammar and external token code to simplify and add more robust logging:

image
image830×348 22.6 KB

rql.grammar

tokens.js

These are the console log statments with input: [Highlight red] (HR)

Enter stack:17<H> input:17<H>                                                                                                                                                                                                                                              
Read H                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
Read R                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
input.acceptToken(Symbol, 1)<HR>                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
Exit stack:17<H> input:19<)>                                                                                                                                                                                                                                             
Enter stack:20<￿> input:20<￿>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
Terminate stack:20<￿> input:20<￿>

From all appearances it is doing exactly what I would expect but no symbols are showing up in the tree:

Document
·Transform
··Operation
···Symbol<Highlight>
···Symbol<red>
··Alias
···⚠<>
·Block
··Reference
···Select
····Symbol<HR>
····⚠<)>