[Help] local token group used together with EOF

Lezer
1

I write this grammar code to reproduce my error (though not exactly same):

@top Document { markup }

// Markup =============
@local tokens {
  hash { "#" }
}

markup {
  Code
}

Code { hash code_element }


// Code ==============
@local tokens {
  codeWhitespaces { @whitespace+ }
  identifier { @asciiLetter+ }
  leftParen { "(" }
  rightParen { ")" }
  comma { "," }
}

Identifier { identifier }

code_element {
  Identifier | LetBinding
}

LetBinding {
  kw<"let"> codeWhitespaces+ LetLhs
}

LetLhs {
  (Identifier ParamList) | LetPattern
}

ParamList {
  leftParen codeCommaSep<Param>? rightParen
}

Param {
  SimpleParam { Identifier } // Just a name
}

LetPattern {
  Identifier
}



kw<term> { @specialize[@name={term}]<identifier, term> }

codeCommaSep<content> {
  content (comma content)* comma?
}

I cannot compile this grammar since it emits an error:

Tokens from a local token group used together with other tokens (leftParen with ␄) (src/expr.grammar 1:1)

I find this part of the code is related to the error (though may not be the true source):

LetLhs {
  (Identifier ParamList) | LetPattern
}

If I change it to either LetLhs { Identifier ParamList } or LetLhs { LetPattern }, it works.

Thank you!

2

Your LetLhs rule can match either an identifier, or an identifier followed by a param list. That means that after an identifier, the parser must accept both a left paren and whatever can come after LetLhs (which is just the end of input, in this case). Because the paren is, for some reason, in a local token group, that’s not allowed.

Are you sure you need local token groups for the things you’re applying them to? Because a left paren sounds like something that should probably not be a local token.

3

Thank you! Sorry for the late reply.

Are you sure you need local token groups for the things you’re applying them to? Because a left paren sounds like something that should probably not be a local token.

Actually I’m working on typst grammar. You can have a glimpse of its syntax at Syntax – Typst Documentation . It’s basically markdown but can have math, code expressions inside(not just raw block). I think it’s better to handle these tokens independently. By using the local tokens group, grammar code is not much concise (e.g. I need to handle whitespace tokens carefully, not just putting it inside skip), but I think it’s worthwhile. Currently it’s the best option in my view from my limited parser experience and lezer knowledge. If you know what’s the appropriate/better way to handle it, please let me know. Thank you! <3