Null Tokens

Hello Marjin,

It seems that using multiple “?” symbols slows down the compilation a lot. On the forum, in one of the answers you had mentioned that this happens because the symbol expands the given expression. For example, a? b c? expands to b | a b | b c | a b c.

Now consider an expression like (a|b)? Does this expression expand to (a|b| null token) ? Is there such a thing as a null token that can be defined?

The "" or () syntax can be used to explicitly indicate matching nothing. So yeah, (a|b)? would expand to something like a|b|"".

But that won’t help with optimizing the number of generated rules—the | operator, when nested in parentheses as part of some bigger rule, also duplicates the rest of the rule. Factoring parts of the rule into new rules is (paradoxically) a more effective way to keep the number of rules that’ll be in the expanded grammar low.

1 Like