using different parser with CodeMirror 6

Lezer
1

Hi,

I’d like to provide syntax highlight/autocompletion but have all this parsed server-side.

Would codemirror 6 work with parsers like chevrotain(https://github.com/SAP/chevrotain)/Nearley/PEG?

Or is Lezer the only way to go?
On the backend is it good practice to duplicate parser definitions in another language like Chevrotain/etc. above?

Thank you!

2

Lezer takes care of the efficient incremental re-parsing and the highlight package does highlighting based on Lezer-style trees, but there’s no reason similar things couldn’t be done with other parsers (that’s the idea behind moving highlighting out of the core view library).

3

Thanks!

I’m not that experienced into parser internals and since Lezer goes hand-in-hand with CodeMirror 6, I guess it’s not worth integrating some other parser myself.
So would Lezer work in node as well to use it in backend code?

And re to moving highlighting out, are there any plans to provide any adapters for other parsers still? Something that takes care of incremental re-parsing, etc.? (sorry for my ignorance if this is not possible)

4

Lezer runs on node. But the trees it produces are very simple (not abstract), and may not be convenient to work with for doing more advanced types of code analysis.

Not from me. That’s something that the people working with those parsers will have to address.

5

Thanks @marijn!

Guess I’ll start with Lezer both ui/backend and introduce another parser on backend only if necessary.
But the introspection I need is quite basic, a subset of the python grammar.

6

I can share some experience on implementing syntax highlighting from scratch, in case someone comes across this topic. Crudely the syntax highlighting behavior can be viewed as the following layers:

  1. A worker that produces Decoration.mark from data structure generated by the parser. It is also responsive for mapping those ranges across document changes. In particular, the built-in syntax package and Lezer cooperate with a simple scheduler so that it does not stress the browser for a large document.
  2. A highlighter that map token name (e.g. lineComment) to pre-defined tags (a TagSystem), and then to the decoration spec, i.e. classes that should be applied on some range.
  3. A ViewPlugin that glue them together, clipping decorations to visible ranges of the viewport.

As the author says, the neat thing is that these works are parser agnostic, and incremental parsing is one of the design goals of Lezer. I think they are deemed to be less efficient if switched to another parser system, but not infeasible.