State being reset part way through tokenizing

I’m trying to parse a piece of code that is an array of 264 strings with each string on a different line thus:

'String 1',
'String 2',
'String 264'

For some reason my parser is recording errors from line 258 onwards even though the codes is fine. When I added some trace lines to the tokenize and getInitialState methods I see the following;

2 calls to getInitialState
11 sets of calls to tokenize for lines 0 - 10 (3 calls per line which I expect)
Another call to getInitialState
A call to tokenize the last line (line 263)
A series of calls to tokenize the lines from 11-263
2 more calls to getInitialState
A series of calls to tokenize lines 0 - 98
A call to getInitialState
A series of calls to tokenize lines 258 - 263.

The call to getInitialState before jumping to lines 258 - 263 seems to be what is causing all the errors to appear from line 258 onwards.

Can anyone suggest why the tokenizer is jumping around like this.

For context, this is in a React application and I am simply opening the window to display the a form which contains a codemirror instance.

Any suggestions gratefully received as I’m stuck at this point.

Hard to say much without knowing more about your mode, but it is rather easy to create bugs where modes accidentally rely on shared state or don’t copy their state properly, causing a continued parse to get confused about its state.

It’s not so much an issue with the content of the state it’s more the sequence of calls to tokenize and getInitialState that I don’t understand. The series of calls above happen when React first renders the codemirror instance. Is it possible for anything in the tokenize or getInitialState methods that I have written that could change the sequence of calls. I don’t call these methods myself anywhere. I assumed that codemirror had complete control over when these methods are called and the order in which lines are passed for tokenizing. It’s that order that seems unexpected i.e. why at the end would it try to parse the first 99 lines and then jump to parsing the last 6 lines missing out all the ones in between ?

It’s reusing the state from the first parse—re-parsing the entire document every time would be somewhat inefficient. See CodeMirror's mode system

The thing that confuses me most - and causes the problem - is that when it jumps from parsing the first 99 lines to parsing the last 6 lines it doesn’t appear to be using the previous state - which would work fine I think - but instead calls getInitialState - which throws away the saved state and resets it. If it didn’t call getInitialState I think everything would be fine.

Can you suggest anything I might have done to cause it to call getInitialState and then parse only the last 6 lines.

Assuming getInitialState is part of your startState method, it should only be calling that for the start of the document, or when it needs the state at a given line but can’t find a saved state anywhere near it (which, for highlighting, should only happen if you quickly scroll down a long document).