Are there any test utilities for ensuring that ExternalTokenizers work (or for allowing easier debugging)

Hello, I’ve on-and-off writing a grammar for over a year, and am currently stuck on a custom tokenizer that I need to handle this situation:


  • {{! ... }}
  • {{!-- ... --}}
  • <!-- ... -->

Not Comments

  • {{ ... }}

so, since, so far, my only way of debugging a grammar has been to load it up in codemirror, I figure there must be a better way, yeah?

I took a poke through lang-html, html/test/test-incremental.js at main · lezer-parser/html · GitHub, and it seemed very manual / 0 abstractions – so I’m hoping there exists some abstraction somewhere to make testing not painful?


Have you seen this page?

Yes! That’s fantastic for testing the whole of the grammar implementation.

Is that as fine-grained as testing can get atm?

Pretty much, though of course you can write your own custom tests that call your parser’s parse method and do different things with the tree. Putting it into a CodeMirror instance should definitely not be necessary.