Are there any test utilities for ensuring that ExternalTokenizers work (or for allowing easier debugging)

1

Hello, I’ve on-and-off writing a grammar for over a year, and am currently stuck on a custom tokenizer that I need to handle this situation:

Comments

  • {{! ... }}
  • {{!-- ... --}}
  • <!-- ... -->

Not Comments

  • {{ ... }}

so, since, so far, my only way of debugging a grammar has been to load it up in codemirror, I figure there must be a better way, yeah?

I took a poke through lang-html, html/test/test-incremental.js at main · lezer-parser/html · GitHub, and it seemed very manual / 0 abstractions – so I’m hoping there exists some abstraction somewhere to make testing not painful?

Thanks!!!

2

Have you seen this page?

3

Yes! That’s fantastic for testing the whole of the grammar implementation.

Is that as fine-grained as testing can get atm?

4

Pretty much, though of course you can write your own custom tests that call your parser’s parse method and do different things with the tree. Putting it into a CodeMirror instance should definitely not be necessary.