Does the cm.getModeAt(pos) always return (or it should return) the same values for the same pos, or it adds a new token every call? If the latter, how can I reset it?
What do you mean by adding a new token? The result of getModeAt depends on the tokenizer state, so it’s not entirely stable (if you, for example run a query at the end of a big document, and the tokenizer hasn’t ran that far yet, you may get a different result later when the tokenizer has caught up and, for example, seen the <script> tag that it hadn’t gotten to before).
Sorry, I somehow managed to overlook that I inserted the wrong function name - it should have been getTokenAt, not getModeAt.
getTokenAt on http://pl.wikipedia.org changes the returned values on every call (string and start/end position; and after that it breaks the highlighting after adding a newline). So I started to wonder if “updating” of getTokenAt is by design, or it’s because of different implementation (because on folding demo page getTokenAt is “stable” [at least at the point that I’ve checked] - but it may be a difference in size/implementation).
getTokenAt will re-run the tokenizer for at least the line you’re looking at. If it keeps returning different values that suggests the tokenizer might be broken (they should usually be deterministic).
I guess it’s kinda deterministic - getTokenAt cycles through all the tokens in the line (for the same pos) - first it’s the token at pos, then what’s before it, and then adds the tokens after it (one by one), until the end of line, where it resets and starts the cycle again.
Calling getTokenAt also influences what getLineTokens returns: normally it returns an array for entire line, but if getLineTokens will be called after getTokenAt, on its first call the returned array will have less elements than normal.