Highlighting that seems ignored in CM6

Has I said in another entry, switching to CM6 0.20 lost the highlight for markdown and yaml languages.
The editor runs in Chrome on Windows in a Vue 3 application (if this matter). Before, under 0.19 the highlighting was perfect. Here are, hoping they could help diagnosing the problem: the node_modules content, the code I use for markdown and an error that appears as soon as I start writing in the editor. A small example will follow shortly.
These are the related node_modules entries:

+-- @codemirror/commands@0.20.0
| +-- @codemirror/language@0.20.0 deduped
| +-- @codemirror/state@0.20.0 deduped
| +-- @codemirror/view@0.20.2 deduped
| `-- @lezer/common@0.16.0
+-- @codemirror/lang-markdown@0.20.0
| +-- @codemirror/lang-html@0.20.0
| | +-- @codemirror/autocomplete@0.20.0
| | | +-- @codemirror/language@0.20.0 deduped
| | | +-- @codemirror/state@0.20.0 deduped
| | | +-- @codemirror/view@0.20.2 deduped
| | | `-- @lezer/common@0.16.0
| | +-- @codemirror/lang-css@0.20.0
| | | +-- @codemirror/autocomplete@0.20.0 deduped
| | | +-- @codemirror/language@0.20.0 deduped
| | | +-- @codemirror/state@0.20.0 deduped
| | | `-- @lezer/css@0.16.0
| | |   +-- @lezer/highlight@0.16.0 deduped
| | |   `-- @lezer/lr@0.16.2
| | |     `-- @lezer/common@0.16.0
| | +-- @codemirror/lang-javascript@0.20.0
| | | +-- @codemirror/autocomplete@0.20.0 deduped
| | | +-- @codemirror/language@0.20.0 deduped
| | | +-- @codemirror/lint@0.20.1
| | | | +-- @codemirror/state@0.20.0 deduped
| | | | +-- @codemirror/view@0.20.2 deduped
| | | | `-- crelt@1.0.5 deduped
| | | +-- @codemirror/state@0.20.0 deduped
| | | +-- @codemirror/view@0.20.2 deduped
| | | `-- @lezer/javascript@0.16.0
| | |   +-- @lezer/highlight@0.16.0 deduped
| | |   `-- @lezer/lr@0.16.2
| | |     `-- @lezer/common@0.16.0
| | +-- @codemirror/language@0.20.0 deduped
| | +-- @codemirror/state@0.20.0 deduped
| | +-- @lezer/common@0.16.0
| | `-- @lezer/html@0.16.0
| |   +-- @lezer/highlight@0.16.0 deduped
| |   `-- @lezer/lr@0.16.2
| |     `-- @lezer/common@0.16.0
| +-- @codemirror/language@0.20.0 deduped
| +-- @codemirror/state@0.20.0 deduped
| +-- @codemirror/view@0.20.2 deduped
| +-- @lezer/common@0.16.0
| `-- @lezer/markdown@0.16.0
|   +-- @lezer/common@0.16.0
|   `-- @lezer/highlight@0.16.0 deduped
+-- @codemirror/language@0.20.0
| +-- @codemirror/state@0.20.0 deduped
| +-- @codemirror/view@0.20.2 deduped
| +-- @lezer/common@0.16.0
| +-- @lezer/highlight@0.16.0 deduped
| `-- @lezer/lr@0.16.2
|   `-- @lezer/common@0.16.0 deduped
+-- @codemirror/legacy-modes@0.20.0
| `-- @codemirror/language@0.20.0 deduped
+-- @codemirror/search@0.20.1
| +-- @codemirror/state@0.20.0 deduped
| +-- @codemirror/view@0.20.2 deduped
| `-- crelt@1.0.5
+-- @codemirror/state@0.20.0
+-- @codemirror/view@0.20.2
| +-- @codemirror/state@0.20.0 deduped
| +-- style-mod@4.0.0
| `-- w3c-keyname@2.2.4

The piece of code:

    import {defaultHighlightStyle, HighlightStyle} from "@codemirror/language";
    import {tags} from "@lezer/highlight";
    import {markdown} from "@codemirror/lang-markdown";

    const myHighlightStyle = HighlightStyle.define([
        {tag: tags.keyword,          color: "#FC6"},
        {tag: tags.comment,          color: "#16A004", fontStyle: "italic"},
        {tag: tags.strong,           color: "#66D9EF", fontStyle: "bold"},
        ...
    ]);
    state: EditorState.create({
            extensions: [
                markdown(),
                syntaxHighlighting(myHighlightStyle),
                syntaxHighlighting(defaultHighlightStyle),
                ...
             ])

If I write in the editor **word** it is not rendered as per my style.
As soon as I write anything in the editor, the following error appears:

CodeMirror plugin crashed: TypeError: tags3 is not iterable
    at HighlightStyle.style (index.js:227:29)
    at highlightTags (index.js:244:33)
    at HighlightBuilder.highlightRange (index.js:294:30)
    at highlightTree (index.js:262:13)
    at TreeHighlighter.buildDeco (index.js:1582:13)
    at new TreeHighlighter (index.js:1564:33)
    at ViewPlugin.create (index.js:1798:42)
    at PluginInstance.update (index.js:1817:44)
    at new EditorView (index.js:5962:20)
    at loadConfigurationData (EditConfig.vue:90:12)
ο»Ώ```
Hope this is sufficient.
Thanks for looking!
mario

I’m sorry for the details.
Do you import syntax Highlighting?

import {defaultHighlightStyle, HighlightStyle} from "@codemirror/language";
import {defaultHighlightStyle, HighlightStyle, syntaxHighlighting} from "@codemirror/language";

I have the same behavior (same exact error message) with a language that I developed (grammar, style, highlights) that worked fine in 19.x, but fails with 20.
I followed the basic guidelines for updating: moving imports for the new package layout, and wrapping styles with syntaxHighlighting(…).
Let me know what kind of info I can provide to help find the issue, and thanks for your help.
The error appears as soon as some text is entered

Francesco

Check npm ls --all, figure out if there are 0.19 packages installed and why.

No 0.19 packages in my case:

β”œβ”€β”¬ @codemirror/autocomplete@0.20.0
β”‚ β”œβ”€β”€ @codemirror/language@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/state@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/view@0.20.2 deduped
β”‚ └── @lezer/common@0.16.0
β”œβ”€β”¬ @codemirror/commands@0.20.0
β”‚ β”œβ”€β”€ @codemirror/language@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/state@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/view@0.20.2 deduped
β”‚ └── @lezer/common@0.16.0
β”œβ”€β”¬ @codemirror/language@0.20.0
β”‚ β”œβ”€β”€ @codemirror/state@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/view@0.20.2 deduped
β”‚ β”œβ”€β”€ @lezer/common@0.16.0
β”‚ β”œβ”€β”€ @lezer/highlight@0.16.0 deduped
β”‚ └── @lezer/lr@0.16.2 deduped
β”œβ”€β”¬ @codemirror/search@0.20.1
β”‚ β”œβ”€β”€ @codemirror/state@0.20.0 deduped
β”‚ β”œβ”€β”€ @codemirror/view@0.20.2 deduped
β”‚ └── crelt@1.0.5
β”œβ”€β”€ @codemirror/state@0.20.0
β”œβ”€β”¬ @codemirror/view@0.20.2
β”‚ β”œβ”€β”€ @codemirror/state@0.20.0 deduped
β”‚ β”œβ”€β”€ style-mod@4.0.0
β”‚ └── w3c-keyname@2.2.4
β”œβ”€β”¬ @lezer/generator@0.16.0
β”‚ β”œβ”€β”€ @lezer/common@0.16.0
β”‚ └── @lezer/lr@0.16.2 deduped
β”œβ”€β”¬ @lezer/highlight@0.16.0
β”‚ └── @lezer/common@0.16.0
β”œβ”€β”¬ @lezer/lr@0.16.2
β”‚ └── @lezer/common@0.16.0

Same for me. No 0.19 packages.

Yes imported syntaxHighlighting too. Was a cut&paste glitch.

Odd. I tried to set up an editor based on your code snippet, and I’m not seeing the crash β€” highlighting works as expected.

Don’t know if it’s relevant, but in my case it’s a react application, build with ViteJS.
The problem happens both in dev mode (JS source transpiled on the fly) and in production builds (assets packaged using rollup)

This is relevant! My application too is built with Vue + vite. If I compile for development the problem is the previous one. If I build for production the resulting bundled application says: Unrecognized extension value in extension set ([object Object]). This sometimes happens because multiple instances of @codemirror/state are loaded, breaking instanceof checks. Not simple to debug, but I’ll try and report back.
The message is from node_modules/@codemirror/state/dist/index.js (suggestion: add a JSON.stringify() call for ext to understand where the problem happens)

I have edited node_modules/@codemirror/state/dist/index.js to add the JSON.stringify call (bad thing, I know…) and magically no more Unrecognized extension value in extension set error, but returned the previous CodeMirror plugin crashed: TypeError: n is not iterable without applying styles.

In my opinion, this points to some module loading order problem. I’ll try to change the module order for rollup to see if anything changes. The current order (from vite.config.js) is:

    build: {
        reportCompressedSize: false,
        emptyOutDir: true,
        rollupOptions: {
            output: {
                manualChunks: {
                    codemirror1: [
                        "@codemirror/view",
                        "@codemirror/state",
                        "@codemirror/commands"
                    ],
                    codemirror2: [
                        "@codemirror/lang-markdown",
                        "@codemirror/search",
                        "@codemirror/language",
                        "@codemirror/legacy-modes/mode/yaml",
                        "@lezer/highlight"
                    ]
                }
            }
        }
    }

Seems (magically) solved. I tried to add console.log calls to @lezer/highlight/dist/index.js and restarted development server with the --force option (npx vite --force) and now the highlight works and no more error about iterable.

Same solution for production build. Added a console.log call to @lezer/highlight/dist/index.js rebundled for production. It works. Then removed the print statements, rebundled and it worked again.

Maybe vite has some hidden cache that this way has been reloaded (there is no --force option for production).

@fbellomi let me know if this magical spell solved also your problem.

Thanks for your patience, support and knowing I was not alone with Vue+vite problems!
mario

thanks Mario,

I tried to add some console.log to @lezer/highlight/dist/index.js but it did not solve the problem, even with --force (I had already tried to clear the cache before).

Also, I need to guarantee the reproducibility of the build, since the production build will be generated in a CI/CD pipeline, not on my local machine. If the sequence of your action somewhat modified some state of the system, it is not clear to me where this state is held (afaik, vite cache is simply a cache of transpiled files)

Is this a matter of the order of the import statements / initialization? Is there a way to programmatically force the correct initialization order from the code?

to be precise: my app is react + ViteJS

I don’t know what to answer. In my case a stale cache could be the main explanation, but the fact that changing a string made the error disappears points to other problem, like related to loading order, maybe triggered by the module reorder and merging. But till the problem is not reliably reproducible I don’t know what to suggest.

The issue disappears for me if I modify

changing tags with tags = [], and tags with (tags || []) in line 229

My understanding is that function is called the first time with an undefined value, and this causes the plugin to crash and be disabled; preventing this allows the function to be recalled with a suitable value

I’m not sure what is the root cause of the problem (init order?), and why it seems dependent on the specific build system; I’m not knowledgeable enough to address this in a more principled way.

1 Like

Hello,

I was facing the same issue as described. The fix proposed by @fbellomi also worked for me.
@marijn Do you want me to create a pull request proposing this little fix ?

Have a nice day,

@rdesille

another way is to patch the highlighter:

const highlighter = HighlightStyle.define([
  ...
]);
const fn0 = highlighter.style;
// noinspection JSConstantReassignment
highlighter.style = tags => fn0(tags || [])

With this approach, you can avoid modifying the source code of the library, which is a good idea if you don’t want to depend on unpublished artifacts.
It requires modifying a TS constant, however

Francesco

1 Like

No, that is just masking an underlying problem of using mismatched library versions, it doesn’t seem reasonable to add a kludge for that to the code.

1 Like

Had a similar issue with highlighting/theme post upgrade to 0.20.0…

Had to delete the yarn.lock file AND nuke the node_modules folder.

Hope this helps someone!