Parse string on single line with only start delimited

1

I have the following grammar:

@top Program { String* }

@skip { whitespace }

@skip {} {
  String {
    '"' (Escape | stringContentDoubleQuote)* '"' |
    "«" stringContentSmartQuote* "\n"
  }
}

@tokens {
  whitespace { std.whitespace+ }
  
  Escape { "\\" _ }

  stringContentDoubleQuote { ![\\\n"] }
  stringContentSmartQuote { ![\n] }
}

@detectDelim

The aim is to support two types of strings:

  • Strings defined between a pair of double quotes
  • Strings that follow a special start delimiter “«”, and which should match everything up until the end of the line

But when I test it using the following test cases, I get a parse error

# String

"this is a quoted string"
"this\tis a quoted\nstring\"with escapes"
« This is smart-quoted string

==>

Program(String, String(Escape, Escape, Escape))

The error is:

bun test v1.1.12 (43f0913c)

test/grammar.test.ts:
1323 |                 return this.stackToTree(finished);
1324 |             }
1325 |             if (this.parser.strict) {
1326 |                 if (verbose && stopped)
1327 |                     console.log("Stuck with token " + (this.tokens.mainToken ? this.parser.getName(this.tokens.mainToken.value) : "none"));
1328 |                 throw new SyntaxError("No parse at " + pos);
                             ^
SyntaxError: No parse at 97
      at advance (/Users/erik/Code/codemirror-lang-arturo/node_modules/@lezer/lr/dist/index.js:1328:23)
      at parse (/Users/erik/Code/codemirror-lang-arturo/node_modules/@lezer/common/dist/index.js:1738:24)
      at run (/Users/erik/Code/codemirror-lang-arturo/node_modules/@lezer/generator/dist/test.js:174:26)
✗ cases > String [2.28ms]

 0 pass
 1 fail
Ran 1 tests across 1 files. [17.00ms]

Any ideas?

2

The test runner will trim the input string to the end of the last line with content before the ==>, so there won’t be a newline after your « string. Might want to introduce an eof { @eof } token and make the end of that rule ("\n" | eof).

1 Like
3

Thank you so much! That solves it.