When using external tokens, why do we create a circular reference between the grammar file and tokens.js?

1

For example, in the grammar file, we do this:

@external tokens htmlCommentContent from "./tokens.js" { htmlCommentContent }

and then in ./tokens.js we import from an output of the grammar file:

import {
  htmlCommentContent as htmlCommentToken,
} from './syntax.grammar.terms';

export const htmlCommentContent = new ExternalTokenizer((input) => {
  // ...
});

This feels like a cycle:

  • ./tokens.js must import from the *.terms file
  • *.terms file doesn’t exist until *.grammar is compiled
  • *.grammar imports from ./tokens.js

Why is it this way?