Function names aren't highlighted when keywords are used in custom DSL

Lezer
1

We use a custom DSL for some templating stuff within our app and have the following grammar:

@top Instructions { content+ }

content { Comment | Command | ConditionalOpen | ConditionalBranch | ConditionalClose | Text }

@tokens {
  seperatableDigit { $[0-9_] }

  FunctionName { $[a-zA-Z.]+ }
  NumberLiteral { $[-]? seperatableDigit+ ("." seperatableDigit+)? }
  StringLiteral {
    '"' (!["\\] | "\\" _)* '"' |
    "'" (!['\\] | '\\' _)* "'"
  }

  Text[group=TextContent] { ![{]+ }
  space { (@whitespace | "\t" | "\r" | "\n")+ }

  OpenTag[closedBy=CloseTag] { "{" }
  CloseTag[openedBy=OpenTag] { "}" }

  OpenArgs[closedBy=CloseArgs] { "(" }
  CloseArgs[openedBy=OpenArgs] { ")" }

  OpIn[group=Operators] { "in" }
  OpAdd[group=Operators] { "+" }

  StartIf[group=Logic] { "#if" }
  ElseIf[group=Logic] { "#elseif" }
  Else[group=Logic] { "#else" }
  EndIf[group=Logic] { "#endif" }

  CommentOpen[closedBy=CommentClose, group=Comment] { "{!" }
  CommentClose[openedBy=CommentOpen, group=Comment] { "!}" }
  CommentContent { (![\!] | "!" ![\}])+ }

  @precedence { CommentOpen, OpenTag }
  @precedence { CommentClose, Text }
  @precedence { OpIn, FunctionName }
}

Comment[isolate=ltr] {
  CommentOpen CommentContent* CommentClose
}

@skip { space } {
  argument { NumberLiteral | StringLiteral | Invocation }
  operation { argument | OperationAdd | OperationInclude }

  OperationAdd {
    argument (OpAdd argument)+
  }

  OperationInclude[isolate=ltr] {
    argument OpIn argument
  }

  ConditionalOpen[closedBy=ConditionalClose,isolate=ltr] {
    OpenTag StartIf ((OpenArgs operation CloseArgs) | operation) CloseTag
  }

  ConditionalBranch[isolate=ltr] {
    OpenTag ElseIf ((OpenArgs operation CloseArgs) | operation) CloseTag |
    OpenTag Else CloseTag
  }

  ConditionalClose[openedBy=ConditionalOpen,isolate=ltr] {
    OpenTag EndIf CloseTag
  }

  FollowupParameter[isolate=ltr] {
    "," operation
  }

  Invocation[isolate=ltr] {
    FunctionName (OpenArgs (operation FollowupParameter*)+ CloseArgs)?
  }

  Command[isolate=ltr] {
    OpenTag Invocation CloseTag
  }
}

For some reason, when I enter {foo("bar")} it detects it just fine as a function, with the argument foo. However, if I call a function called “include”, it no longer gets highlighted: {include("bar")} as it probably thinks it got the “in” keyword.

I tried a couple things but only seem to have made it worse. Sometimes it says I can’t move the OpIn part out of the @skip block, if i change the precedence, other things seem to break.

image

Any help would be much appreciated!

2

Don’t use literal tokens ("in") for keywords. They’ll, as you find, match any word with in as a prefix. You’ll want to use @specialize with your identifier token type (which I guess is FunctionName instead, so that the tokenizer first matches the entire word, and then looks whether the resulting word matches an identifier.

3

Ok, thanks! I assume this is correct then?

OpIn {
  @specialize<FunctionName, "in">
}

It shows up correctly at least:
image

When we need more keywords (like and, or), I guess I just have to create new blocks like above?

4

Yes, that looks correct. Many grammars use a macro like this to easily define keywords.

kw<term> { @specialize[@name={term}]<identifier, term> }
1 Like