Tokens Case insensitive

Lezer
1

I want to create a list allowed tokens but manage case insensitivity on them, ie I want to be able to put JaN20 or jan20 or JAN20. How can I do that, one option is to write all permutations but i’d prefer to avoid that if possible.

@tokens {

Tenor {
((“f”|“jan”|“g”|“feb”|“h”|“mar”|“j”|“apr”|“k”|“may”|“m”|“jun”|“n”|“jul”|“q”|“aug”|“u”| “sep”|“v”|“oct”|“x”|“nov”|“z”|“dec”|“a”|“q1”|“b”|“q2”|“c”|“q3”|“d”|“q4”)
(std.digit std.digit|“20” (“1”|“2”|“3”|“4”) std.digit))
}

}

2

I think the easiest approach would be an external tokenizer—either make it responsible for parsing all identifiers, and have it distinguish these tokens, or have it recognize only these, and have a higher precedence than the built-in tokenizer.

3

Sorry to dig up a dead thread but, how would I go about doing this?

4

And external specializer is probably the best approach for this. See the guide.