Nested Autocomplete for Custom Languages

v6
1

When going through the examples, i noticed that I can configure a basic mixed HTML parser with autocomplete via the following:

import { parser as htmlParser } from "@lezer/html";
import { parseMixed } from "@lezer/common";
import {
  foldInside,
  foldNodeProp,
  LanguageSupport,
  LRLanguage,
} from "@codemirror/language";
import { javascript, javascriptLanguage } from "@codemirror/lang-javascript";

export const mixedHTMLParser = htmlParser.configure({
  wrap: parseMixed((node) =>
    node.name == "ScriptText" ? { parser: javascriptLanguage.parser } : null
  ),
  props: [foldNodeProp.add({ Element: foldInside })],
});

const mixedHTMLLanguage = LRLanguage.define({ parser: mixedHTMLParser });

export const mixedHTML = () => {
  return new LanguageSupport(mixedHTMLLanguage, [javascript().support]);
};

When using this plugin, i get all the language support from javascript inside of script tags!

However, when implementing my own mixed language with javascript language support:

import { parser as templateParser } from "lib/parser";
import { parseMixed } from "@lezer/common";
import {
  foldInside,
  foldNodeProp,
  LanguageSupport,
  LRLanguage,
} from "@codemirror/language";
import { javascript, javascriptLanguage } from "@codemirror/lang-javascript";

export const templateStringParser = templateParser.configure({
  wrap: parseMixed((node) => {
    return node.name == "JavaScriptText"
      ? { parser: javascriptLanguage.parser }
      : null;
  }),
  props: [
    foldNodeProp.add({
      Expression: foldInside,
    }),
  ],
});

export const templateStringLanguage = LRLanguage.define({
  parser: templateStringParser,
});

export const templateString = () => {
  return new LanguageSupport(templateStringLanguage, [
    javascript({ jsx: false }).support,
  ]);
};

I get syntax highlighting for javascript but no autocomplete. Any idea what could be going wrong here? I could post the grammar for the templateParser if needed.

2 Likes
2

Also running into this

3

I don’t have your grammar, but when I change that code to use the @lezer/html parser and target the UnquotedAttributeValue nodes instead, JavaScript completion works inside of those.

4

Also running into this highly specific issue 2 hours after the post was made? That’s quite a coincidence. Or, in case you’re a coworker of the OP ‘bumping’ this or a spam account trying to build up post count, please don’t — it’s annoying.

1 Like
5

Thanks for the prompt response Marjin. Would the grammar/tokenizer have any impact on how support extensions for nested parsers are run? The HTML example works well for me, so wondering what the difference could be.

Here’s my grammar:

@top TemplateString { Expression+ }

Expression {
    OpenExpression JavaScriptText CloseExpression
}

OpenExpression {
    "{{"
}

JavaScriptText { javascriptText* }

@external tokens scriptTokens from "./tokens.js" {
    javascriptText
    CloseExpression
}

with tokens.js being

import { ExternalTokenizer } from "@lezer/lr";
import { CloseExpression, javascriptText } from "./parser.terms";

const closeTemplate = 125;

function expressionTokenizer() {
  return new ExternalTokenizer((input) => {
    let i = 0;
    let state = 0;
    let javascriptTextLength = 0;
    while (true) {
      if (input.next < 0) {
        if (i) input.acceptToken(javascriptText);
        break;
      }
      // first close template
      if (state == 0 && input.next == closeTemplate) {
        state++;
      }
      // second close template
      else if (state == 1 && input.next == closeTemplate) {
        // if we have javascriptTextLength then accept that token
        if (javascriptTextLength) {
          input.acceptToken(javascriptText, -javascriptTextLength);
        } else {
          input.acceptToken(CloseExpression, 1);
        }
        break;
      } else {
        // reset
        javascriptTextLength++;
        state = 0;
      }
      input.advance();
    }
  });
}

export const scriptTokens = expressionTokenizer();

With the following string: {{asdf}} being parsed here are the nodes:
Node TemplateString from 0 to 8
Node Expression from 0 to 8
Node OpenExpression from 0 to 2
Node JavaScriptText from 2 to 6
Node CloseExpression from 6 to 8

which seems like I should be able to just run the javascript parser on JavaScriptText and get the appropriate support extensions

6

My apologies, I just wanted to follow along with the thread, I should have just bookmarked it or watched it. Won’t happen again!

7

Ah actually got this to work by removing the ifNotIn block from the JavaScript autocomplete.

Thanks for the help again! Really appreciate it and the work on this library :slight_smile:

8

Although interestingly, this suggests that my mixed language parsing isn’t exactly correct as i shouldn’t need to modify the javascript language extensions…:thinking:

9

Ok final update for anybody else who’s following along – turns out my grammar and the JavaScript grammar had a node collision!

Important to note that you should prefix all grammars if you plan on mixing them and don’t want any node collisions :slight_smile:

10

I see. This patch should prevent that in the future. Definitely don’t prefix your node names, that’d get very ugly.

11

Hi AashJ,

Now I also need to parse like text {{ var a = Date.now() }} text content, the whole content is plain text, but content between {{ and }} parsed as javascript.

Can you help me? give me your whole parse code?

1 Like