Parsing JS in double curly braces

1

What is the best way to enable JavaScript syntax checking only within double curly braces ({{ }})? For example, I want JavaScript syntax checking for {{ some expression }}, but treat everything outside as plain text.

Currently, I have JavaScript syntax checking enabled globally, but I need it to be restricted to within the curly braces. I attempted to use a combination StreamParser and StreamLanguage, but it didn’t work as expected.

This seems like it should be straightforward with a custom language, but I haven’t found the right combination of packages to achieve this in the editor. What is the best approach to accomplish this?

import { LanguageSupport, type StreamParser, StreamLanguage } from '@codemirror/language';
import { javascriptLanguage } from '@codemirror/lang-javascript';

const customLanguageParser: StreamParser<{ inJS: boolean; jsState: any }> = {
    startState() {
        return { inJS: false, jsState: null };
    },
    token(stream, state) {
        if (!state.inJS) {
            if (stream.match('{{')) {
                state.inJS = true;
                state.jsState = javascriptLanguage.parser.startParse('');
                return 'brace';
            }
            stream.next();
            return 'text';
        } else {
            if (stream.match('}}')) {
                state.inJS = false;
                return 'brace';
            }
            // Use the JavaScript parser for content inside curly braces
            const jsToken = javascriptLanguage.parser.token(stream, state.jsState);
            return jsToken ? jsToken.type : 'javascript';
        }
    },
};

const customLanguage = StreamLanguage.define(customLanguageParser);

export const customLanguageSupport = new LanguageSupport(customLanguage);

This returns a “Property ‘token’ does not exist on type ‘LRParser’” error.

2

That is a pretty good summary of your problem. You’re calling token on a class that doesn’t support such a method.

Nesting languages inside stream parsers doesn’t really work in this system. You’ll want to use a Lezer parser for the outer structure and set up mixed parsing to wire up the inner parser.

1 Like
3

Thank you, this is very helpful to me

1 Like
4

@marijn Can’t seem to pinpoint what part of this is incorrect. Any input based on my files? It is still parsing JS inside and outside the double curly braces, rather than just within the double curly braces.

This is my .grammar file:

@top Document { (TextContent | JsBlock)* }

JsBlock { "{{" JsContent "}}" }

@tokens {
  TextContent { ![\{\}]+ }
  JsContent { ![\{\}]+ }
}

This is the combinedParser file:

import { parser as jsParser } from '@lezer/javascript';
import { parseMixed } from '@lezer/common';
import { LRLanguage } from '@codemirror/language';
import { parser as outerParser } from './CustomLanguage/parser';

const mixedParser = outerParser.configure({
    wrap: parseMixed((node) => {
        if (node.type.name === 'JsBlock') {
            return { parser: jsParser };
        }
        return null;
    }),
});

const mixedLanguage = LRLanguage.define({ parser: mixedParser });

export { mixedLanguage as language };

This is my editorState:

import { EditorState } from '@codemirror/state';
import { basicSetup } from 'codemirror';
import { EditorView } from '@codemirror/view';
import { javascript } from '@codemirror/lang-javascript';
import { autocompletion } from '@codemirror/autocomplete';
import highlightExtensions from './SyntaxHighlighting/syntaxhighlighting.config';
import { combinedJavascriptAndCustomCompletions } from './Autocomplete/autocomplete.config';
import { language as mixedLanguage } from './CustomLanguage/combinedParser';

export const createEditorState = (doc: string): EditorState => {
    return EditorState.create({
        doc,
        extensions: [
            basicSetup,
            javascript(),
            combinedJavascriptAndCustomCompletions,
            autocompletion(),
            ...highlightExtensions,
            EditorView.lineWrapping,
            mixedLanguage,
        ],
    });
};

I generated the parser.js and parser.terms js file using: yarn lezer-generator outer.grammar -o parser, and this is what the parser.js and parser.terms file looks like:

// This file was generated by lezer-generator. You probably shouldn't edit it.
export const
  Document = 1,
  TextContent = 2,
  JsBlock = 3,
  JsContent = 4

and parser.js:

// This file was generated by lezer-generator. You probably shouldn't edit it.
import {LRParser} from "@lezer/lr"
export const parser = LRParser.deserialize({
  version: 14,
  states: "zQQOPOOOYOQO'#C_OOOO'#Ca'#CaQQOPOOO_OPO,58yOOOO-E6_-E6_OOOO1G.e1G.e",
  stateData: "d~OQQOVPO~OSSO~OWUO~O",
  goto: "aUPPPVPZTQORQRORTR",
  nodeNames: "⚠ Document TextContent JsBlock JsContent",
  maxTerm: 8,
  skippedNodes: [0],
  repeatNodeCount: 1,
  tokenData: "!l~RVO#oh#o#p!U#p#qh#q#r!a#r;'Sh;'S;=`!O<%lOhRoTSQQPO#oh#p#qh#r;'Sh;'S;=`!O<%lOhR!RP;=`<%lh~!XP#o#p![~!aOV~~!dP#q#r!g~!lOW~",
  tokenizers: [0, 1],
  topRules: {"Document":[0,1]},
  tokenPrec: 0
})

Under dev dependencies, this is the lezer version: “@lezer/generator”: "^1.7.1.

These are the versions under dependencies: “@codemirror/autocomplete”: “^6.18.0”, “@codemirror/lang-javascript”: “^6.2.2”, “@codemirror/state”: “^6.4.1”, “@codemirror/view”:“^6.33.0”, “@lezer/lr”: “^1.4.2”, “@lezer/javascript”: “^1.4.17”, and “@lezer/common”: “^1.2.1”.

Thanks!

5

The last version that contained this add[0].set expression seems to have been lezer-tree (not even @lezer/common yet) 0.13.2. You may have some really old versions somewhere in your dependencies. Check with npm ls -a

7

@marijn Thanks! I was able to figure out how to resolve the add[0].set error by adjusting some version settings!

How exactly can we support autocomplete for javascript within the curly braces and also support custom completions for a custom language with mixed parsing?

This is my editorState:

import { EditorState } from '@codemirror/state';
import { basicSetup } from 'codemirror';
import { EditorView } from '@codemirror/view';
import { autocompletion } from '@codemirror/autocomplete';
import highlightExtensions from './SyntaxHighlighting/syntaxhighlighting.config';
import { javascriptAndPlainTextParser } from './CustomLanguage/Parser/mixedParser';
import { combinedJavascriptAndCustomCompletions } from './Autocomplete/autocomplete.config';

export const createEditorState = (doc: string): EditorState => {
    return EditorState.create({
        doc,
        extensions: [
            basicSetup,
            javascriptAndPlainTextParser,
            autocompletion(),
            combinedJavascriptAndCustomCompletions,
            ...highlightExtensions,
            EditorView.lineWrapping,
        ],
    });
};

And this is the autocomplete.config file:

import type { CompletionContext, CompletionResult } from '@codemirror/autocomplete';
import { syntaxTree } from '@codemirror/language';
import { type SyntaxNode } from '@lezer/common';
import { javascriptAndPlainTextParser } from '../CustomLanguage/Parser/mixedParser';

export function getJavascriptAutocompleteSuggestions(context: CompletionContext): CompletionResult | null {
    const word = context.matchBefore(/[\w><=!&|]*/);
    if (!word || (word.from === word.to && !context.explicit)) {
        return null;
    }
    return {
        from: word.from,
        options: [
            { label: 'data store property 1', type: 'data store' },
            // TODO: Just a placeholder for now, iterate over the data store properties and add them here
        ],
        validFor: /^[\w><=!&|]*$/,
    };
}

export function combinedAutocomplete(context: CompletionContext): CompletionResult | null {
    const tree = syntaxTree(context.state);
    const node = tree.resolve(context.pos, -1);

    let currentNode: SyntaxNode | null = node;
    while (currentNode !== null) {
        if (currentNode.type.name === 'JsBlock') {
            return getJavascriptAutocompleteSuggestions(context);
        }
        currentNode = currentNode.parent;
    }

    return null;
}

export const combinedJavascriptAndCustomCompletions = javascriptAndPlainTextParser.data.of({
    autocomplete: combinedAutocomplete,
});

This is my mixedParser file:

import { parser as jsParser } from '@lezer/javascript';
import { parseMixed } from '@lezer/common';
import { LRLanguage } from '@codemirror/language';
import { parser as outerParser } from './parser';

const mixedParser = outerParser.configure({
    wrap: parseMixed((node) => {
        if (node.type.name === 'JsBlock') {
            return { parser: jsParser };
        }
        return null;
    }),
});

const customLanguage = LRLanguage.define({ parser: mixedParser });

export { customLanguage as javascriptAndPlainTextParser };

In the editor state, I removed the javascript extension given that the javascriptAndPlainTextParser is there, but it seems to not work with autocomplete. It parses correctly inside and outside the double curly braces (refer to parser files in above comment), but it seems to not work in conjunction with the autocomplete configuration I have. Ideally, it should provide autocomplete suggestions for javascript within the curly braces. Any clue what the issue is here with my autocomplete.config? Syntax highlighting is working fine, so it seems to be an issue with autocomplete only, parser seems to parse as expected.