How to reprocess syntax tree?

Is it possible to reprocess the whole sytanx tree again?

The context:

I’m creating a language package for Excel functions. One of the peculiarities is that some tokens change according to the idiom. For example, the Boolean token can be TRUE (English) or VERDADEIRO (Portuguese).

I created an external token that handles this based on the currIdiom variable:

File: tokens.ts

let currIdiom = 'en-US';

const i18n = {
  BoolToken: {
    'en-US': ['TRUE', 'FALSE'],
    'pt-BR': ['VERDADEIRO', 'FALSO']

export const isBoolean = (value: string, stack: Stack): number => {
  return i18n.BoolToken[currIdiom].indexOf(value.toUpperCase()) !== -1
    ? tokens['BoolToken']
    : -1;

To be able to change the idiom, I exported a function that changes the currIdiom variable:

export const setLezerIdiom = (newIdiom: supportedIdioms): supportedIdioms =>
  (currIdiom = newIdiom);

Everything is working, but when I switch between languages, it seems to me that it keeps the previous syntax tree, I need to type something to update. See the example below, TRUE is in purple (recognized as a BoolToken when language is en-US), but when I change it to Portuguese, I need to type something to stop being purple (= no longer recognized as a BoolToken):

excel formula

How can I force a reprocess so that highlighting works when switching idiom?

Pass the language as a parameter to whatever creates the parser, and update your configuration to use a new parser whenever the language setting is changed.

1 Like

I tried here but without success… It always depends on an interaction for the highlighting to update. I must be doing something wrong…

export function spreadsheet(idiom: supportedIdioms = 'en-US') {
  return new LanguageSupport(spreadsheetLanguage);

Every dropdown change triggers this here:

    effects: languageCompartment.reconfigure(spreadsheet(newIdiom))

It’s ugly, but this is what solved it:

    effects: languageCompartment.reconfigure(spreadsheet(newIdiom)),
    changes: { //Force reprocessing
      from: 0,
      to: editor.state.doc.length,
      insert: editor.state.doc.toString()

That will destroy your undo history and all other editor state that is tracking document positions, though.

Does this mean using buildParser to create a parser on the fly, instead of having a static parser?

Looking at the source code, not sure how one build a static parser that allows changing idiom or decimal separator.

No. You can use dialects or reconfigured external tokenizers to make small changes like that. Recompiling the grammar should never be necessary at run-time.

reconfigured external tokenizers to make small changes like that.

Is there a lezer example lang that does this which I could reference?

Edit: Guessing we have to use configure and use tokenizers and specializers.

You can see the lang-sql reconfiguring the parser to wire in a customized external tokenizer here.

1 Like