Terminating a rule with an identifier contained within it

Lezer
1

I’m continuing my halting progress on a YAML grammar and running into an issue with the YAML key definition.

For example these are all valid YAML keys/value pairs:

# ":foo" -> "value"
:foo: value
# "/foo/bar:/foo" -> "value"
/foo/bar:/foo: value

So : is both part of the content of a key and it’s final delimeter. I’d like to demarcate the final : as not part of the key (for syntax highlighting purposes).

The approach that I tried to take was use a @local tokens expression:

// https://lezer.codemirror.net/docs/guide/#writing-a-grammar
@top Program { Property+ }

Property {
  keyExp spaces* Value (newline | eof)
}

@local tokens {
    KeyEnd { ": " }
    @else Key
}

@skip {} {
  keyExp {
    Key+ KeyEnd
  }
}

Value {
  Plain | Boolean
}

@tokens {
  Boolean { "true" | "false" | "nor" }
  spaces { $[ \t]+ }
  newline { "\n" }
  eof { @eof }
  Comment { "#" ![\n]+ }
  Plain { (@asciiLetter|@digit) ![\n#]* }
  @precedence { Boolean, Plain}
}

@skip { Comment }
@detectDelim

}

But that results in

Tokens from a local token group used together with other tokens (Key with Comment)

If I remove Comment I get the same message but it’s (Key with Boolean). Is there a way to accomplish what I want with @localor do I need to reach for an external tokenizer that can detect the final": "` that termintes a key?

2

Since keyExp starts with Key, it will be used in contexts where other tokens are also valid. That doesn’t work, with local token groups—they can only be used when delimited on both sides so that no other tokens are valid where they appear.

This case seems simple enough to use a regular token for (Key { ![:]+ } or something?).

3

They can only be used when delimited on both sides so that no other tokens are valid where they appear.

I think that makes sense, so that’s why they are typically perfect for things like matching pairs of " (e.g. in the Python grammar) or similar delimeters? So not a good fit for this use case, alas :smile_cat:

This case seems simple enough to use a regular token for (Key { ![:]+ } or something?).

I don’t think I can use this since the delimiter that marks the end of a key is : (a colon and a space)

a: key
# totally valid :\ just becomes `{ "also:a": "key"}`
also:a: key

I may have missed something there in the explanation though, and i’d much prefer to use a regular token for this.

Otherwise, I guess I could create an external tokenizer for KeyEnd or something similar that can wait for : and check if the next token is a space.

4

That sounds like using an external tokenizer will be easiest, yes.