What am I missing about ParseContext.skipUntilInView?

I’m updating my parser to the most recent CodeMirror 6 version, and I’m having an issue with CodeMirror seemingly not respecting calls I make to ParseContext.skipUntilInView.

image

In the last image, you can see the boundary where my parser stopped and called skipUntilInView from that position until the end of the parse region.

Obviously, it would be helpful if I provided some code, but an entire parser is quite a lot of code! The main part of the code can be found here, but be warned that some of the comments are outdated.

I can provide how I am calling the function, though. When I am assembling the Tree that will be returned from the parser, I run this bit of code:

    const context = ParseContext.get()

    // inform editor that we skipped everything past the viewport
    if (
      context?.skipUntilInView &&
      this.pos > context.viewport.to &&
      this.pos < this.region.original.to
    ) {
      console.log("skipping", this.pos, this.region.original.to)
      console.log(tree)
      context.skipUntilInView(this.pos, this.region.original.to)
    }

You can treat this.region.original as being one of the ranges provided to the parser when it was constructed.

Basically, what am I missing? What problems should I be looking for? Could my Tree be misconfigured, and that would cause it? I would imagine that skipUntilInView simply marks a region as “you should parse as soon as the viewport reaches this range”, but it doesn’t seem to behave that way.

Can you say a bit more about how it is not respecting the call? As in, what happens vs what did you expect to happen?

Sure. I’ll go with an example scenario:

  • The editor has a document 50k characters long.
  • The viewport is from 0-10k.
  • The parser is given the range 0-50k to parse.

With that all setup, let’s say I have the parser parse up to something like 15k and then stop, because it’s beyond the viewport. It then calls skipUntilInView(15000, 50000). What I would expect to happen, is that as soon as the viewport crosses somewhere beyond 15k, parsing is requested again.

But, for my parser, that doesn’t happen. Instead, I can scroll around and click in that 15k-50k region all I like, and nothing happens. I have to actually type something, which then finally causes the parser to be requested.

Thanks for the clarification. I was able to reproduce that—it was a bug. This patch should fix it.

Oh, that’s a relief, haha. I figured it was something I was doing wrong. Thanks for figuring out (and fixing) the issue.

Also, the new mixed parsing and per-node props is fantastic. The way my parser handles nested content is waaaay simpler now.

That’s great to hear. Is this going to be a generic monarch-style parser? That’d be nice to have (I haven’t worked on this myself since I’m not a big fan of that style of parser, but if it’s easy to adopt existing VS Code definitions to this, I’m sure it’ll come in handy for someone).

It’s an offshoot from a Monarch-style parser, yeah. It unfortunately has a somewhat different syntax compared to Monarch, which means adopting a VS Code definition would take some work, but it’s not too bad. However, it’s quite a bit more capable.

Although, honestly, right now most of the work I’ve put into this parser is into the “backend” of it. The way grammars are defined is a bit… chaotic and/or ugly as compared to say, Lezer, but that can change. The main goals for this system is that it is extremely flexible, and can handle absurd Markdown-esque languages. I would never use this system if I could use Lezer instead.

EDIT: I should add, and you may already know this, is that Monarch is basically the forgotten stepchild of VS Code language definitions, so honestly even if there was direct support for it I don’t think it would be all that useful. If there was an extension for Textmate languages, that would be very interesting, and I have thought about doing it. The issue is that Textmate definitions depend on some esoteric regex features, which I think only recently have become workable due to match indices support being added to JS regex recently.

Hey, sorry to bother you @marijn, but could you publish a build to NPM for @codemirror/language? Thanks again for fixing the bug.

Oh, right, sure. Just tagged 0.19.3