I’m trying to get syntax highlighting working for relatively large documents (1-2k LOC, max 5k LOC). I’ve noticed that the current behavior of the tokenizer is to always give up after a couple hundred LOC. This results in a lot of unstyled snippets.
In my case, I’m prioritizing accuracy over performance. The editor is always set to be readonly and having a initial slower load is fine. Is there a way to set a scanLimit, similar to what you did with the @codemirror/merge editor? Or any other way to workaround the lazy tokenizer?
The highlighting in that example is working fine. But it parses most of that document as a JSX tag because you enabled TSX and are using an arrow function with a type parameter at line 183.
On closer look, TypeScript’s parser appears to have some messy heuristics to disambiguate type parameter lists from JSX tags. I haven’t been able to find a definitive source on how they are supposed to work, but I’ve added logic to @lezer/javascript 1.4.11 that makes <T,> and <T extends Foo> parse as type parameters.
Ahh, you are so right! Thanks for all the help! After upgrading to the latest patch and correctly setting JSX based on file extension, I can confirm that all syntax highlighting is working as expected.