I can share some experience on implementing syntax highlighting from scratch, in case someone comes across this topic. Crudely the syntax highlighting behavior can be viewed as the following layers:
- A worker that produces
Decoration.mark from data structure generated by the parser. It is also responsive for mapping those ranges across document changes. In particular, the built-in
syntax package and Lezer cooperate with a simple scheduler so that it does not stress the browser for a large document.
- A highlighter that map token name (e.g.
lineComment) to pre-defined tags (a
TagSystem), and then to the decoration spec, i.e. classes that should be applied on some range.
ViewPlugin that glue them together, clipping decorations to visible ranges of the viewport.
As the author says, the neat thing is that these works are parser agnostic, and incremental parsing is one of the design goals of Lezer. I think they are deemed to be less efficient if switched to another parser system, but not infeasible.