Retrieving changes to build autocompletions

[supriyo-biswas]

I’m trying to build a generic autocompletion module that works independently of language modes, and my idea here is to:

1. Receive ViewUpdate events from the editor.
2. Identify the new tokens from the changed part.

Since tokenizing the entire document on every change is likely to be a problem, I would like to identify the changed content, and update the list of tokens to be suggested. The latter part is on me to implement; my question is about the former.

Playing around for a bit, I can see that there is a ViewUpdate.changes which contains a list of insertions, but there doesn’t seem to be a way to identify deletions. So, my questions are:

1. Am I thinking correctly about how a generic autocompletion may be implemented? Or is there another API I should be looking at to build this feature?
2. Assuming the answer to (1) is yes, how would I go about identifying deletions and retrieving the changed line efficiently?

Thank you.

[marijn]

@supriyo-biswas

Since tokenizing the entire document on every change is likely to be a problem

This is what incremental parsing (as done by CodeMirror language modes) intends to solve. If you can use the syntax tree that was already built up by the library, you don’t have to worry about this entire problem.

@supriyo-biswas

Playing around for a bit, I can see that there is a ViewUpdate.changes which contains a list of insertions, but there doesn’t seem to be a way to identify deletions.

Change sets also identify deletions (changes that have fromA < toA delete content).

[supriyo-biswas]

@marijn

This is what incremental parsing (as done by CodeMirror language modes) intends to solve. If you can use the syntax tree that was already built up by the library, you don’t have to worry about this entire problem.

If I have a legacy mode such as https://github.com/codemirror/legacy-modes/blob/9fd24d9968b83e2c23a5ea96dc72e2d60b7c0bb3/mode/rust.js (I know there’s a @codemirror/lang-rust these days, but I’m only using this is an example), can I still use syntax tree hints?

@marijn

Change sets also identify deletions (changes that have fromA < toA delete content).

Thank you, the positions in the changeset seem to identify the character position in terms of the entire document. Is there a way to map the character positions to a specific line efficiently so that I don’t have to look through all the lines?

[marijn]

@supriyo-biswas

can I still use syntax tree hints?

You can still use the syntax tree. Implementing completions on top of it is something your code will have to do.

@supriyo-biswas

Thank you, the positions in the changeset seem to identify the character position in terms of the entire document.

Yes, all positioning in the library works that way. If you subtract the start of the line you have a line-relative position.

[supriyo-biswas]

Apologies for the many questions, I’m a bit lost in the docs as to how to achieve the things you mentioned…

@marijn

If you subtract the start of the line you have a line-relative position.

I noticed the {from/to}{A/B} properties, but the only way I see to map a position to a line number would be to iterate over EditorState.doc.text and find out the relevant line. Is there a more efficient way to do this? I’d need this to identify token boundaries in the line and add or remove the tokens from the autocompletions list that I manage manually.

@marijn

You can still use the syntax tree. Implementing completions on top of it is something your code will have to do.

I understand that I can get the syntax tree by using syntaxTree(editor.state), but is there any example that shows how I can get the individual tokens? The tree object seems to use packed integers for representing its children, and TreeCursor seems to be specific to Lezer parsers, and I think it won’t apply to simple/legacy modes like the Rust one?

Thank you for your help once again.

[marijn]

@supriyo-biswas

but the only way I see to map a position to a line number would be to iterate over EditorState.doc.text

No, you can use doc.lineAt(pos)

@supriyo-biswas

TreeCursor seems to be specific to Lezer parsers, and I think it won’t apply to simple/legacy modes like the Rust one?

The Rust mode is in @codemirror/lang-rust is also a Lezer parser. But even the legacy mode builds up a tree using the data structure defined in @lezer/common, and supports the same operations (though it will be shallower, only listing tokens and no larger structures).

[supriyo-biswas]

I have a follow up question to all of this.

I can identify the changed sections ViewUpdate.changedRanges, but what would be the most efficient way to detect the changed text?

The easiest way would be to do editor.state.doc.slice(x, y) but I assume it uses slicing under the hood. There’s also ViewUpdate.changes.inserted which does seem to have the changed text, but I have a few concerns:

1. For insertions/replacements, I always see an additional TextLeaf object of zero-length for each change, is this expected?
2. Can I assume that the order of ViewUpdate.changedRanges and ViewUpdate.changes.inserted are the same? For example, if I have a ViewUpdate object that looks like this:

ViewUpdate {
  changedRanges: [range0, range1],
  changes: {
    inserted: [TextLeaf{length: 0}, TextLeaf{length: ...}, TextLeaf{length: 0}, TextLeaf{length: ...}]
  }
}

Can I assume that the text corresponding to range0 is always given by changes.inserted[1] and range2 is always given by changes.inserted[3]?

[marijn]

@supriyo-biswas

There’s also ViewUpdate.changes.inserted which does seem to have the changed text,

Please don’t go digging through objects for undocumented stuff. That can change and will not provide a usable, stable interface. The docs tell you what you can do with a ChangeSet object, and that’s what you will have to work with.