How do I debug my lezer grammar?

Lezer
1

I’m trying to write a Lezer Grammar for Glimmer.

The tree-sitter equiv is here: tree-sitter-glimmer/grammar.js at main · alexlafroscia/tree-sitter-glimmer · GitHub
But it also completely parses HTML as well, which I’m hoping to avoid by using parseMixed – so I only want to parse things between {{ and }} and ignore everything else (for now (I do need to parse some HTML stuff for better component invocation highlighting, named-blocks, etc))

I’m running in to an issue right now where where I have two types errors (when running my tests):

SyntaxError: No parse at 5 # or whatever number

or

Error: Unexpected end of Value

(specifically when trying to test strings (which could be either double or single quoted).

So, I guess my questions are:

  • where do I go from here?
  • how do I debug?
  • I really hope I don’t need any “tokens.js” file like a bunch of the example grammars have (I don’t want to write JS :sweat_smile: ) – I don’t have to, do I?

Here is my grammar so far:

@top Glimmer {
  list<
    Expression
    | Block
    | BlockComment
  >
}

Expression { moustache<SubExpression> }
Block { StartBlock any* EndBlock }
StartBlock { moustache<"#" name SubExpression? As?>  }
As { kw<"as"> "|" list<name> "|" }
EndBlock { moustache<"/" name> }

SubExpression { list<Value> NamedArgs? }
Value {
  boolean
  | null
  | undefined
  | string
  | Number
  | Invocation
  | Argument
  | ThisExpression
  | PropertyPath
}
Invocation { "(" SubExpression ")" }
NamedArgs { list<Pair> }
Pair { string "=" Value }

Argument { "@" name }
ThisExpression { this "." PropertyPath }
PropertyPath { name ("." name) }

boolean {
  @specialize[@name=BooleanLiteral]<identifier, "true" | "false">
}

null {
  @specialize[@name=Literal]<identifier, "null">
}

undefined {
  @specialize[@name=Literal]<identifier, "undefined">
}


LongComment { "!--" any* "--" }
ShortComment { "!" any* }
BlockComment { moustache<(LongComment | ShortComment)> }

@tokens {

  Number { int frac? }
  int { '0' | $[1-9] @digit* }
  frac { '.' @digit+ }
  this { "this" }

  identifierChar { @asciiLetter | $[_$\u{a1}-\u{10ffff}] }
  word { identifierChar (identifierChar | @digit)* }
  identifier { word }

  @precedence { identifier, this }
  @precedence { name, identifier, this }

  name { word }
  any { $[.] }

  attributeContentDouble { !["]+ }
  attributeContentSingle { ![']+ }

  string {
   "\"" attributeContentDouble* "\"" |
    "\'" attributeContentSingle* "\'"
  }

  space { (" " | "\t" | "\r" | "\n")+ }


  @precedence { " ", space }
}

@skip { space }


// Helper And Special Things
list<item> { item (" " item)* }
kw<term> { @specialize[@name={term}]<identifier, term> }
moustache<content> { "{{" content "}}" }


@external propSource glimmerHighlighting from './highlight'

And some failing tests:

# Multiple Integer

{{42}} and {{24}}

==>

Glimmer(
  Expression(
    SubExpression(
      Value(Number)
    )
  )

  Expression(
    SubExpression(
      Value(Number)
    )
  )
)

# Basic - Double

{{"Some String"}}

==>

Glimmer(
  Expression(
    SubExpression(
      Value(String)
    )
  )
)

And here is my code so far: Implementation by NullVoxPopuli · Pull Request #2 · NullVoxPopuli/glimdown · GitHub

Thanks!

1 Like
2

Did you find a solution here? What was the issue for you?

3

I did not. Last I was trying to work with lezer, the lack of visibility and debugging completely made me lose motivation in making a lezer grammar :cry:

Maybe things are different now, but idk