Resolving a skip rule conflict

Lezer
1

Hey!

Consider this simple grammar where newline should generally be skipped, but there are places where it is required as a terminator:

@top Source {
  foo | bar
}

@skip { newline }

foo { "foo" }
bar { "bar" terminator }

terminator { newline }

@tokens {
  newline { "\n" }
}

The terminator obviously conflicts with newline. Is there a way to give terminator a higher precedence over the skip rule, so we parse newline as terminator in all places where it is allowed? Or is the best solution an external tokenizer for terminator that checks if it’s allowed in the given state and acts based on that?

2

If you make them both tokens, and use @precedence within the @tokens block to give terminator a higher precedence, that should work, I expect. Contextual tokenization will make sure that terminators are only tokenized in states where they can be shifted.

3

@marijn that’s what I tried initially, but it seems to always tokenize the highest precedence token, regardless the state. Example:

@top Source {
  foo | bar
}

@skip { newline }

foo { "foo" }
bar { "bar" terminator }

@tokens {
  newline { "\n" }
  terminator { newline }
  @precedence { terminator, newline }
}

This fails to parse foo\n, I think because the \n is still tokenized as terminator.

4

Oh, you are right. Because there is a precedence declared, the parser generator doesn’t consider these tokens in conflict, and won’t generate different contextual-token groups for them.

You may have more luck with an external tokenizer (for terminator) declared before the regular @tokens block.

5

Thank you : ) I’ve just tried and external tokenizer works as expected, I mostly wanted to make sure there is no simpler way.