Hello, can external tokenizer return zero length (or fractional length, like 0.5) tokens? Or can it break the parser? Thanks!
They can, but you have to be really careful that your grammar doesn’t allow any zero-length thing to occur on repeat, or your parser will hang just running that production endlessly.