Handling Overlapping Tokens

Lezer
1

Let’s say I have a very simple language that can handle only line comments that start with a double-slash and paths that are delimited with a single slash. For example:

// this is a comment
this/is/a/path

The grammar I’ve written for this is:

@top {
  Path*
}

Path { (identifier "/")* identifier }

@tokens {
  whitespace { std.whitespace+ }
  LineComment { "//" ![\n]* }
  identifier { (std.asciiLetter | std.digit)+ }
}

@skip { whitespace | LineComment }

lezer-generator gives me the following error

Overlapping tokens “/” and LineComment used in same context (example: “/” vs “//”)

My intuition tells me that I need to use the ambiguity marker so that the parser splits when it reaches a slash, and will test the next character to determine which path to take, but I’m having trouble figuring out exactly how to do that. Any help would be greatly appreciated!

2

No need for ambiguity marker at all. The tool is just telling you that one of your tokens is a prefix of another (which can sometimes be surprising) and asking you to specify an explicit precedence. Inside you @tokens block, write @precedence { LineComment, "/" } and you should be good.

3

That worked perfectly, thanks so much!