[help] local tokens used together with other tokens

Lezer
1

I have written this grammar:

@top Document { markup+ }

@skip {
  whitespaces
}

@tokens {
  whitespaces { @whitespace+ }
}

@local tokens {
  star { "*" }
  @else text
}

markup {
  Strong
}

@skip {} {
  Strong {
    star text star
  }
}

However, when I compile the grammar, there is an error message:

$ rollup -c

./src/typst.grammar → ./dist/index.js, ./dist/index.cjs...
[!] (plugin rollup-plugin-lezer) Error: Could not load /home/meowking/proj/personal/lezer-typst/src/typst.grammar: Tokens from a local token group used together with other tokens (star with whitespaces) (/home/meowking/proj/personal/lezer-typst/src/typst.grammar 1:1)

I don’t understand why star is mixed with whitespaces since I have used the @skip {} rule, which should exclude the whitespaces skip rule. My guess is @skip { whitespaces } indicates whitespaces is possible to be everywhere in Document, which conflicts with Document { markup+} (only allow markup).

Does anyone know why this problem occurs and how to fix it while using local tokens group?

Thanks!

2

Before a Strong production, both whitespace and star may occur. Local tokens really have to be enclosed in some other token (and, as you’re doing, used in @skip {} rules) to work. For example, they are often used for the content of a string, where the string’s opening quote clearly moves the context to that set of local tokens. The token at the start of a rule that’s used outside of the @skip {} block will be tokenized in a context where the outer whitespace is still active.

1 Like
3

Thank you! Do you mean that local tokens must be "enclosed" within another clearly defined token that unambiguously signals the start of a context where local tokens apply?

I’ve changed my code a little bit to test and it works.

@skip {} {
  Strong {
    <<<     star text star
    ---
    >>>     "*" text star
  }
}