Token end out of bounds

Lezer
1

When I input or paste some javascript codes below, It print “RangeError: Token end out of bounds” in the console. This error is related to @lezer/lr/dist/index.js → acceptToken(token, endOffset = 0).

- my_code
    ```js
    {
        url: `${}`,
    }
    ```

or this one

- my_code
    ```js
    const sql = `INSERT INTO  ()
        VALUES ($1, $2,)
    `;
    ```

when I remove the js or the - at the beginning , this error disappear. When I remove the markdown highlight extension, it also disappear.

The same character with the two example is $ in code box and - at the beginning, and such code is really common in javascript. I hope that this bug will be fixed, thanks~

ps: I encounter this error in my React Project: GitHub - ryankolter/whalesnote: A markdown note-taking software that supports win, mac and linux. Easily store and categorize tens of thousands of notes., which use the newest version of CodeMirror 6. I developed this note’s software from last year, I found this error when I open an old note but the software throw uncaught error, which means that this error does’t present at the previous beta version of CodeMirror 6.

2

I cannot reproduce this error. Are you sure your CodeMirror- and Lezer-related packages are at the latest version?

3

My CodeMirror version is the newest:

"@codemirror/state": "^6.2.1",
"@codemirror/view": "^6.12.0",

the lang-javascript in node_modules:

"name": "@codemirror/lang-javascript",
"version": "6.1.8",
...
"dependencies": {
  "@codemirror/autocomplete": "^6.0.0",
  "@codemirror/language": "^6.6.0",
  "@codemirror/lint": "^6.0.0",
  "@codemirror/state": "^6.0.0",
  "@codemirror/view": "^6.0.0",
  "@lezer/common": "^1.0.0",
  "@lezer/javascript": "^1.0.0"
},
"devDependencies": {
  "@codemirror/buildhelper": "^0.1.5",
  "@lezer/lr": "^1.0.0"
},

the @lezer package.json in node_modules:

"name": "@lezer/lr",
"version": "1.3.3",

"name": "@lezer/javascript",
"version": "1.4.1",

It is related to the markdown extension:

import { markdown, markdownLanguage } from '@codemirror/lang-markdown';
import { languages } from '@codemirror/language-data';

markdown({ base: markdownLanguage, addKeymap: false, codeLanguages: languages }),
4

You can see the actual versions installed with npm ls -a. Specifically, make sure @lezer/common is at 1.0.2

5

Thanks for your reply. After checking the package’s version with command npm ls -a, all of the @lezer/common version is 1.0.2.

I try hard to simplify the code in my React Project, and put it into the “Try CodeMirror” page, but I also can not reproduce this error. And the error in my project never disappear.

I think the only difference between them is the React ? So, I use a codeSandBox to reproduce it :

when you paste the code below, it throw this error.

- my_code
    ```js
    {
        url: `${}`,
    }
    ```
6

If this can only be reproduced in a React app, I’m not going to debug it, sorry.

7

Thanks, I know it’s hard to debug. :sweat_smile:

I think this error doesn’t exist in the previous CodeMirror 6 version. Because I continuously developed the same React note-taking software based on CodeMirror 6 among the last year. It appear suddenly when I upgraded to a new version and opened an old note file with such code which pasted at last year.

I try to degrade the version. When I degrade the @codemirror/language from 6.6.0 to 6.5.0 and the @codemirror/language-data from 6.2.0 to 6.1.0, this problem disappear. I hope this information will help the debug, which means that the problem is not about the @lezer~

The high version React project: (with problem)

The degraded version React project:(no problem)

8

I fix it, thank you~ @marijn

Answer: @lezer/lr need to be update from 1.3.3 to 1.3.5

I try to explicity set the version in package.json:

"@codemirror/language": "^6.7.0",
"@codemirror/language-data": "^6.3.1",
"@lezer/common": "1.0.2",
"@lezer/lr": "^1.3.5",
"@lezer/markdown": "1.0.2",

then, delete the node_modules, remove package-lock.json file and run npm install again.

Those steps above are not necessary, all you need to do is updating the @lezer/l manually.

In the React project, I just click the refresh button at the right of @lezer/lr to update, the error disappear~