Template style language

Ultimately I want a user’s input into the codemirror editor to be a string which concatenates consectutive parts where a part is either a template or not and where the template will contain a javascript expression to be evaluated.

For example (if x is number 2),

“abc {{ x + 2 }} zep”

Would evaluate to “abc 4 zep”.

As you probably gathered “{{” and “}}” are used to determine the javascript expression.

From what I’ve researched I will need to use the “mixed parser”.

However I’m struggling to jump over the first hurdle of building the outside parser.

Based on “lang-json” I’ve got something close but need a little help pointing me in the right direction or telling me what I’m aiming for isn’t even a go-er.

This is the grammar.

@top Templates { value* }

value { Object | String | whitespace }

Object { "{{" Stuff* "}}" }

String {
  string
}

Stuff {
  string | whitespace
}

@tokens {

  string { '"' char* '"' }
  char { $[\u{20}\u{21}\u{23}-\u{5b}\u{5d}-\u{10ffff}] | "\\" esc }
  esc  { $["\\\/bfnrt] | "u" hex hex hex hex }
  hex  { $[0-9a-fA-F] }

  whitespace { $[ \n\r\t] }

  "{{" "}}"
}

list<item> { item ("," item)* }

@detectDelim

And these tests are passing.

# Basic

{{"foo"}}

==>

Templates(Object(Stuff))

# Multiple Templates

{{"foo"}}{{"foo"}}

==>

Templates(Object(Stuff), Object(Stuff))

# Spaced out multiple templates

{{"foo"}} {{"foo"}}

==>

Templates(Object(Stuff), Object(Stuff))

# Strings outside templates

"abc" {{"foo"}} "mno" {{"foo"}} "xwz"

==>

Templates(String, Object(Stuff), String, Object(Stuff), String)

All I need is to change it so plain text, instead of strings, are permitted inside and outside the templates.

However if I make this slighty change

Stuff {
  char* | whitespace
}

I’m getting a conflict which needs “reducing”.

shift/reduce conflict between
  Object -> "{{" · "}}"
and
  Stuff -> 
With input:
  "{{" · "}}" …
Shared origin: value -> · Object
  via Object -> "{{" · Stuff+ "}}"
    via Stuff+ -> · Stuff
      Stuff -> ·

Have you seen the Mixed-Language parsing example?

Thanks for the quick response.

I’ve had a look again at the example. It seems though that this takes for granted an outside parser already and only explains how to then mix it with another. An example of a templating language like what I’m trying to achieve would be helpful. Mine though is perhaps a tad trickier than many because “{{” could be interpreted as plain text or the beginning of a template. On that note I did just try adding in a “precedence” list inside the tokens and that’s now got something working. Whether it’s “right” though I’m not sure. I guess so long as I have a token which contains each template I should be able to get started with now mixing this in with the javascript parser.

Ta

The example links the grammar it uses, which might be useful.

Hi rossm6,

do you resolve it? I need to parse like This is a {{ new Date() }} content with js expression also.
Can you help me?

I’m also curious in what your final approach was. My goal was to be able to allow arbitrary text within double braces {{ }} within a JSON file as we run the content within those braces as an expression which creates a resulting valid JSON file.

I tried following the Mixed Language example and swapped out HTML → JSON languages within the outer mixed parser (wrapper), but if I load a regular JSON file without a double braced expression {{ }}, I get an error message like:

index.js:1823 CodeMirror plugin crashed: TypeError: Cannot read properties of undefined (reading 'some')
    at hasChild (index.js:1007:26)
    at TreeNode.nextChild (index.js:552:91)
    at TreeCursor.enterChild (index.js:803:42)
    at TreeCursor.firstChild (index.js:813:32)
    at HighlightBuilder.highlightRange (index.js:336:25)
    at HighlightBuilder.highlightRange (index.js:329:26)
    at highlightTree (index.js:267:13)
    at TreeHighlighter.buildDeco (index.js:1672:13)
    at new TreeHighlighter (index.js:1654:33)
    at ViewPlugin.create (index.js:1877:42)

I originally used the raw twig.grammar example (with the highlights commented out) and tried simplifying the grammar to just the double braces, but I can’t figure out where I’m going wrong or what I should be looking at for troubleshooting.

Simplified `expression.grammar`
// Very crude grammar for splitting expressions from regular text

@top Template { (directive | Text)* }

directive {
  Expression
}

@skip {space} {
  Expression { "{{" ExpressionContent "}}" }
}

@tokens {
  Text { ![{] Text? | "{" (@eof | ![%{] Text?) }
  ExpressionContent { ![%}] ExpressionContent? | $[%}] (@eof | ![}] ExpressionContent?) }
  space { @whitespace+ }
  @precedence { space ExpressionContent }
  "{{" "}}"
}
Simplified `expressionLanguage.js`
import {parser as expressionParser} from "./expression-parser.js"
import {jsonLanguage as originalJsonLanguage} from "@codemirror/lang-json"
import {parseMixed} from "@lezer/common"
import {LRLanguage} from "@codemirror/language"


const mixedJsonParser = expressionParser.configure({
    wrap: parseMixed(node => {
        return node.type.isTop ? {
            parser: originalJsonLanguage.parser,
            overlay: node => node.type.name == "Text"
        } : null
    })
})

export const expressionLanguage = LRLanguage.define({parser: expressionParser});
export const jsonLanguage = LRLanguage.define({parser: mixedJsonParser})
Snippet from CM Editor Setup
import {expressionLanguage, jsonLanguage as customJsonLanguage } from "./codeMirror/expressionLanguage.js"

generateEditorExtensions(){
   //the languageSupport is set dynamically based on the selected language
   //we were previously using basic HTML+JS or JS language based on the need
   //we started by layering basic jsonLanguage and are now trying to replace that 
   //with a mixed customJsonLanguage that handles the {{ expressions }} 
   let languageSupport = new LanguageSupport(customJsonLanguage);
   return [
      //keymap, basicSetup,
      languageSupport,
      //oneDark, tabCustomizations, etc
   ]
}

//...

this.editor = new EditorView({
    state: EditorState.create({
        doc: this.value,
        extensions: this.generateEditorExtensions(),
    }),
    parent: this.$refs.editor,
})

If I use the expressionLanguage export directly, things load fine, but if I use the mixed language jsonLanguage export, I get the error mentioned above.

Edit: If I use the ‘expressionLanguage’ and it’s associated highlighting directly, it seems to be highlighting each part correctly:

image

And the tree item in hasTree() that seems to be missing the expected children property is a TreeBuffer with a buffer and set (<NodeSet>) property.

Hi both

Sorry for the delay responding.

I tricked myself into thinking I’d solved this when I claimed this earlier in the thread. To be honest I’ve the feeling the problem I had can’t be solved with the lezer parser at all and so the other way of supporting a language would need to be explored. “Stream parsing” I think is the other way (from memory), which the docs explain is a bit more involved. Things have changed for me though, so I won’t be needing this till later.

If either of you do work out how to solve the problem I mentioned, please do write it up here.

Thanks.

TLDR: It ended up being a dependency conflict with my local project.

As part of the troubleshooting process, I put together a minimal reproduction and was surprised to find that it was working there.

:sunglasses: The StackBlitz above could be a helpful sandbox if you want to play around with your own grammar. It uses Vite to dynamically rebuild the project when content changes and has a special hook to watch the .grammar file and rebuild it if you edit it.

I was really scratching my head as the sample code and package.json dependency definitions all came from my local local project. I started to wonder if it was potentially some other dependency or some part of my local build process that was contributing to the issue

…then I recalled one of the challenges I ran into when I moved from CM5 to CM6 where some old dependencies were cached in node_modules and were unexpectedly getting re-used.

So I cleared out my node_modules and package-lock.json and performed a fresh install of all the dependencies and lo-and-behold, it started working! :blush: :smiley:

Interesting. Thanks for sharing.

I haven’t had chance to check myself but I wonder if it will work mixed with a javascript parser where it understands “{{{}}}” to be an empty object literal.

That’s the thing which made me think it wouldn’t be possible without a stream parser.

I’ll look into it later and report back for anybody else following this thread.

No, this grammar wouldn’t cover that, but it’s not a case I needed to worry about for my use-case. Best of luck!