I’d like to write a JavaScript based template language which embeds JavaScript code inside {{ }}.
The syntax looks like
aa{{ a }}aa
which should output a syntax tree
JSTemplate(Text,CustomInterpolation(VariableName),Text)
The template grammar in mix-parsing documentation is not used because it doesn’t perform well with {{ "}}" }} and some other cases. So I decided to write it based on official lezer-javascript package
Here are my changes to lezer-javascript
- Added some lines to grammar file
@top JSTemplate { (Text | CustomInterpolation)* }
CustomInterpolation { "{{" expression? "}}" }
@external tokens tokenizeText from "./tokens" { Text }
- Implement the external tokenizer named
tokenizeTextwhich provides the Text token (To better explain the issue, I simplify the logic which only process characteraas Text token)
export const tokenizeText = new ExternalTokenizer((input, stack) => {
if (String.fromCharCode(input.next) === 'a') {
input.advance();
input.acceptToken(Text);
}
})
After doing the above work, I build and use the parser with the following code
const templateParser = parser.configure({
top: 'JSTemplate'
})
templateParser.parse(`aa{{ a }}aa`)
The following syntax tree was generated
JSTemplate(⚠,CustomInterpolation(VariableName),⚠)
The expected syntax tree should be
JSTemplate(Text,CustomInterpolation(VariableName),Text)
It seems tokenizeText is not called at the beginning of the input, and no Text token is generated there.
Is there something I’m doing wrong cause the strange behavior?
