Handling Overlapping Tokens

Let’s say I have a very simple language that can handle only line comments that start with a double-slash and paths that are delimited with a single slash. For example:

// this is a comment

The grammar I’ve written for this is:

@top {

Path { (identifier "/")* identifier }

@tokens {
  whitespace { std.whitespace+ }
  LineComment { "//" ![\n]* }
  identifier { (std.asciiLetter | std.digit)+ }

@skip { whitespace | LineComment }

lezer-generator gives me the following error

Overlapping tokens “/” and LineComment used in same context (example: “/” vs “//”)

My intuition tells me that I need to use the ambiguity marker so that the parser splits when it reaches a slash, and will test the next character to determine which path to take, but I’m having trouble figuring out exactly how to do that. Any help would be greatly appreciated!

No need for ambiguity marker at all. The tool is just telling you that one of your tokens is a prefix of another (which can sometimes be surprising) and asking you to specify an explicit precedence. Inside you @tokens block, write @precedence { LineComment, "/" } and you should be good.

That worked perfectly, thanks so much!