How to debug "Overlapping character range"?

Lezer
1

Hello,

I’ve been working on refactoring some parts of lezer-julia, which defines a lot of unicode operators. The current version uses a separate string for each operator, and I want to replace this with character classes/sets (at least for single character operators).

However, if I try to define multiplicative operators this way:

// Minimal example
@tokens {
  times-operator {
    // ASCII operators:
    $[*/%&\\] |
    // unicode operators:
    $[⌿÷··⋅∘×∩∧⊗⊘⊙⊚⊛⊠⊡⊓∗∙∤⅋≀⊼⋄⋆⋇⋉⋊⋋⋌⋏⋒⟑⦸⦼⦾⦿⧶⧷⨇⨰⨱⨲⨳⨴⨵⨶⨷⨸⨻⨼⨽⩀⩃⩄⩋⩍⩎⩑⩓⩕⩘⩚⩜⩞⩟⩠⫛⊍▷⨝⟕⟖⟗⨟]
  }
}

I get the following error:

Overlapping character range (src/julia.grammar 7:4)
  • Defining a RegExp with the same character class in javascript seems to work fine, so I don’t think this is a problem with javascript strings.
  • None of the other Julia operators have any issues and I was able to write them as character classes. Only the times operators above raises an error.

Basically, I haven’t been able to figure out what the error actually is. What is overlapping what? Why is that a problem?

How should I go about fixing this issue? Should I write a program to test each character individually?

2

The · character (Unicode 183) occurs twice (directly next to each other) in your set.

3

Ok, that was indeed the problem. For future reference, the dots are supposed to be different, but at some point (heh) they got normalized to the same character. The proper characters should be:

'·': Unicode U+00B7 (category Po: Punctuation, other) MIDDLE DOT
'·': Unicode U+0387 (category Po: Punctuation, other) GREEK ANO TELEIA

Thanks for the help!