Performance of Lexer and GetTokenAt


I have two “guis” of codemirror connected to the same underlying document. Both are specified to use my custom mode, for which a lexer is registered using defineMode (startState and token).

This works well for short expressions.

However, each time the user changes the text there are several other gui components I need to update, depending on the type of token and the content of the token etc. Thus I am listening for “keydown” in which I call GetToken at.

Now, for longer expressions (~40 tokens in one line, yes the lexer is relatively slow but that it outside my control) I get very sever performance problems where it may take 0.5s of profile time just to add a key.

Profiling indicates that:

  1. The entire line is re-lexed mutliple times, 1 for each of the connected CodeMirrors (could they not share since they are connected to the same document?) and 1 additional for each call to GetTokenAt.

I have managed to reduce this to 3 lexings per keypress by caching the calls to GetTokenAt using GetLineTokens.

However, I seem not to be able to go from 3 to 1, each editor instance seems to do an individual lexing, and I find no way to register on any event like “on lexing finsihed”, meaning that I listen to key presses as do a third round of lexing by calling getLineTokens.

Is this the expected behavior or am I missing some obvious way to reduce it to only requiring one lexing pass per keypress? (Or ideally I guess not one lexing per keypress but per “batch”, e.g. holding down backspace seems to make CodeMirror lex on each removal instead of only re-lexing occasionally to make the operation smooth).

Thanks for any pointers,


Yes, this is the expected behavior. Lexers are expected to be fast. If you’re getting half a secon for lexing a line three times, your responsiveness is going to be horrible for longer lines even with a single editor and no addons. I know you said this is outside of your control, but the only recommendation I have is to write a faster lexer.

Ok, thanks for the clarification