[help/bug?] local tokens group with `@speciailize`

Lezer
1

I’ve written this minimal reproducible code:

@top Document { markup }

@local tokens {
  hash { "#" }
}

markup { Code }

Code { hash code_expression }

@local tokens {
  identifier { @asciiLetter+ }
  auto { "auto" }
}

code_expression {
  CodeAuto  
}

CodeAuto { kw<"auto"> }
// CodeAuto { identifier }
// CodeAuto { auto }


kw<term> { @specialize[@name={term}]<identifier, term> }

For test

# CodeAuto

#auto

==>

Document(Code(CodeAuto))

CodeAuto { kw<"auto"> } doesn’t work because of error SyntaxError: No parse at 1.

CodeAuto { identifier } and CodeAuto { auto } works.

Is it a bug?

2

I suspect it’s matching the auto { "auto" } token you also for some reason have there, though your CodeAuto token specializes identifier. Because auto is specified as a direct literal, it’ll take precedence of identifier tokens that happen to match the same input.

3

It seems not the reason.

I delete the auto rule and have this grammar now:

@top Document { markup }

@local tokens {
  hash { "#" }
}

markup { Code }

Code { hash code_expression }

@local tokens {
  identifier { @asciiLetter+ }
}

code_expression {
  CodeAuto  
}

CodeAuto { kw<"auto"> }


kw<term> { @specialize[@name={term}]<identifier, term> }

It still complains the same error.

4

That was indeed a bug in @lezer/generator. It didn’t properly handle specializations of local tokens. The below patch should help. (But also note that it is not typical to use local token groups for things like identifiers, and it may not be necessary when you don’t need the @else feature.)

1 Like
5

Thank you!