ChordPro Environment grammar question

Lezer
1

I am creating a lezer grammar for the ChordPro format. I am currently working on the Environment Directive.

A start to an environment with label can be described in the following two ways:

{start_of_verse: Verse 1}
// "Verse 1" is an unnamed value which 
// symbolized the label of the environment.

// and

{start_of_verse: label=“Verse 1”} 
// "label="Verse 1"" is an attribute of 
// the environment, where the "name" 
// is "label", and "value" is "Verse 1".

I currently have the following grammar working:

environmentStart {
    "{" 
        environmentStartKeyword 
        directiveSeparator
        (EnvironmentLabel | DirectiveAttributes)
    "}"
}

// Currently, the label can only consist of 
// what `identifier` is and the additional
// rules/tokens i add here. I'll rather like
// a restrictive rule `![...]`.
EnvironmentLabel {
   (identifier | labelNumber)
   (identifier | labelNumber | " ")*
}

DirectiveAttributes {
    DirectiveAttribute (space DirectiveAttribute)*
}

DirectiveAttribute {
    DirectiveAttributeName "=" DirectiveAttributeValue
}

DirectiveAttributeName { 
    identifier
}

DirectiveAttributeValue { string }

@tokens {

    space { $[ \t]+ }

    identifier {
        (@asciiLetter | "_")
        (@asciiLetter | @digit | "_")*
    }
    
    labelNumber {
        @digit+
    }

    string { "\"" (!["\\] | "\\" _)* "\"" }
}

When using the {start_of_verse: Verse 1} syntax of writing the environment start, basicly all non-functional characters and symbols should be allowed, as } denotes the end of the environment label. Although to make the (EnvironmentLabel | DirectiveAttributes) grammar work, EnvironmentLabel and DirectiveAttributes must share identifier, and then further characters can be allowed for EnvironmentLabel by adding characters/tokens to the rule. I ideally want to use a “anti”-rule instead for EnvironmentLabel, so that i dont have to specify every possible character out there for compatibility. This rule (or something similar) would be ideal:

EnvironmentLabel {
    directiveUnnamedLabel 
}

@tokens {
    directiveUnnamedLabel { 
        ![{}\[\]\r\n\t="':]
        ![ {}\[\]\r\n\t="':]*
    }
}

But this rule gives me an overlapping tokens error between directiveUnnamedLabel and identifier.

So my question is:

  • Can i somehow have an “anti”-rule (![...]) in EnvironmentLabel, and still have (EnvironmentLabel | DirectiveAttributes) working for the two cases {start_of_verse: Verse 1} and {start_of_verse: label="Verse 1"}? How do i make it work then?
  • Or do i need to keep the “positive”-rule and specify every character that i want to allow in the EnvironmentLabel rule?
Ambiguity problem in ChordPro (new question created)
2

The most straightforward way to express this is probably an external tokenizer, with a higher precedence than the regular tokens, which scans the text up to the } and only returns an unnamed label token if the content doesn’t look like a valid attribute.

3

Thank you for the response:)

Okay, this means i have to implement javascript logic into the external tokenizer that checks whether the content looks like an attribute? Seems extensive… but of course, you might be right to compare the content against attribute syntax, because i only want it to be parsed as an attribute if it actually is an attribute; at all other cases, it should be parsed as a label.

I would hope for a solution that could reuse more of the lezer grammar language functionality, but if you don’t have any other way, then i’ll try implement the external tokenizer.