External tokenizer getting invoked too late

[marijn]

@dhrubomoy

Is there anything wrong I am doing in the tokenizer function that you can spot?

Well, yes. Your loop blindly scans forward to the next colon, and always accepts a token if there’s any content before that. You’ll probably want to check that you only read identifier characters.