Lezer-cpp throws when parsing malformed C++ raw string literal

hjr265 · March 23, 2021, 6:23am

The following bit of code causes Uncaught TypeError: match is null.

#include <string>

int main() {
	string x = R(";
	return 0;)
}

I was seeing this happen on and off in production (caught through error tracking) but finally managed to dedicate some time to reproduce it locally.

The above bit of code that I shared really makes no sense (it is not using correct syntax). It wouldn’t even compile. But, I suppose lezer-cpp should still be able to deal with it.

Here’s where the error happens:

      if (!markerString) {
        let ruleStart = stack.ruleStart;
        let match = /"(\S*?)\(/.exec(input.read(ruleStart, Math.min(token.start, ruleStart + 100)));
        markerString = match[1] + '"';
      }

/"(\S*?)\(/.exec(...) // => null

Stack trace:

Uncaught TypeError: match is null
    rawStringContinue webpack://toph/./node_modules/lezer-cpp/dist/index.es.js?:56
    updateCachedToken webpack://toph/./node_modules/lezer/dist/index.es.js?:812
    getActions webpack://toph/./node_modules/lezer/dist/index.es.js?:781
    advanceStack webpack://toph/./node_modules/lezer/dist/index.es.js?:1035
    advanceFully webpack://toph/./node_modules/lezer/dist/index.es.js?:1062
    runRecovery webpack://toph/./node_modules/lezer/dist/index.es.js?:1106
    advance webpack://toph/./node_modules/lezer/dist/index.es.js?:961
    work webpack://toph/./node_modules/@codemirror/language/dist/index.js?:265
    apply webpack://toph/./node_modules/@codemirror/language/dist/index.js?:380
    update webpack://toph/./node_modules/@codemirror/language/dist/index.js?:399
    slot webpack://toph/./node_modules/@codemirror/state/dist/index.js?:935
    ensureAddr webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1133
    field webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1549
    syntaxTree webpack://toph/./node_modules/@codemirror/language/dist/index.js?:165
    matchBrackets webpack://toph/./node_modules/@codemirror/matchbrackets/dist/index.js?:84
    update webpack://toph/./node_modules/@codemirror/matchbrackets/dist/index.js?:40
    slot webpack://toph/./node_modules/@codemirror/state/dist/index.js?:935
    ensureAddr webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1133
    EditorState webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1539
    applyTransaction webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1589
    get state webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1304
    update webpack://toph/./node_modules/@codemirror/view/dist/index.js?:5016
    _dispatch webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4956
    dispatch webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4999
    doPaste webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2927
    paste webpack://toph/./node_modules/@codemirror/view/dist/index.js?:3106
    InputState webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2680
    InputState webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2671
    EditorView webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4967
    Editor webpack://toph/./codepanel/editor.coffee?:111
    render webpack://toph/./codepanel/codepanel.coffee?:345
    initialize webpack://toph/./codepanel/codepanel.coffee?:164
    later webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:29
    setTimeout handler*debounce/< webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:35
    triggerEvents webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:425
    trigger webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:364
    _onModelEvent webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1165
    triggerEvents webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:426
    trigger webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:365
    set webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:966
    success webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1084
    Backbone 2
    sync webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1431
    <anonymous> Backbone
    wrap webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:14
    sync webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:850
    fetch webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1089
    initialize webpack://toph/./codepanel/codepanel.coffee?:159
    View webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1253
     Backbone
    CodePanel webpack://toph/./codepanel/codepanel.coffee?:60
    build webpack://toph/./scripts/plugins/codepanel.js?:22
    init webpack://toph/./scripts/plugins/codepanel.js?:60
    initNow webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:24
    initLater webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:35
    requestIdleCallback handler*initLater webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:34
    add webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:91
    <anonymous> webpack://toph/./scripts/plugins/codepanel.js?:49
    js http://localhost:5000/assets/js/plugins/codepanel.js?_=30b68b9fd86e53d9c8c8f259a9e434843926a78d:524
    __webpack_require__ http://localhost:5000/p/squeaky-board:49
    checkDeferredModulesImpl http://localhost:5000/p/squeaky-board:57
    webpackJsonpCallback http://localhost:5000/p/squeaky-board:55
    <anonymous> http://localhost:5000/assets/js/plugins/codepanel.js?_=30b68b9fd86e53d9c8c8f259a9e434843926a78d:9
codepanel.js:56:9
    rawStringContinue webpack://toph/./node_modules/lezer-cpp/dist/index.es.js?:56
    updateCachedToken webpack://toph/./node_modules/lezer/dist/index.es.js?:812
    getActions webpack://toph/./node_modules/lezer/dist/index.es.js?:781
    advanceStack webpack://toph/./node_modules/lezer/dist/index.es.js?:1035
    advanceFully webpack://toph/./node_modules/lezer/dist/index.es.js?:1062
    runRecovery webpack://toph/./node_modules/lezer/dist/index.es.js?:1106
    advance webpack://toph/./node_modules/lezer/dist/index.es.js?:961
    work webpack://toph/./node_modules/@codemirror/language/dist/index.js?:265
    apply webpack://toph/./node_modules/@codemirror/language/dist/index.js?:380
    update webpack://toph/./node_modules/@codemirror/language/dist/index.js?:399
    slot webpack://toph/./node_modules/@codemirror/state/dist/index.js?:935
    ensureAddr webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1133
    field webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1549
    syntaxTree webpack://toph/./node_modules/@codemirror/language/dist/index.js?:165
    matchBrackets webpack://toph/./node_modules/@codemirror/matchbrackets/dist/index.js?:84
    update webpack://toph/./node_modules/@codemirror/matchbrackets/dist/index.js?:40
    slot webpack://toph/./node_modules/@codemirror/state/dist/index.js?:935
    ensureAddr webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1133
    EditorState webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1539
    applyTransaction webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1589
    get state webpack://toph/./node_modules/@codemirror/state/dist/index.js?:1304
    update webpack://toph/./node_modules/@codemirror/view/dist/index.js?:5016
    _dispatch webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4956
    dispatch webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4999
    dispatch self-hosted:1177
    doPaste webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2927
    paste webpack://toph/./node_modules/@codemirror/view/dist/index.js?:3106
    InputState webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2680
    (Async: EventListener.handleEvent)
    InputState webpack://toph/./node_modules/@codemirror/view/dist/index.js?:2671
    EditorView webpack://toph/./node_modules/@codemirror/view/dist/index.js?:4967
    Editor webpack://toph/./codepanel/editor.coffee?:111
    render webpack://toph/./codepanel/codepanel.coffee?:345
    initialize webpack://toph/./codepanel/codepanel.coffee?:164
    later webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:29
    (Async: setTimeout handler)
    debounce webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:35
    triggerEvents webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:425
    trigger webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:364
    _onModelEvent webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1165
    triggerEvents webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:426
    trigger webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:365
    set webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:966
    success webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1084
    Backbone 2
    sync webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1431
    <anonymous> Backbone
    wrap webpack://toph/./node_modules/@toph/kernel.js/dist/fn.js?:14
    sync webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:850
    fetch webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1089
    initialize webpack://toph/./codepanel/codepanel.coffee?:159
    View webpack://toph/./node_modules/exoskeleton/exoskeleton.js?:1253
     Backbone
    CodePanel webpack://toph/./codepanel/codepanel.coffee?:60
    build webpack://toph/./scripts/plugins/codepanel.js?:22
    init webpack://toph/./scripts/plugins/codepanel.js?:60
    initNow webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:24
    initLater webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:35
    (Async: requestIdleCallback handler)
    initLater webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:34
    add webpack://toph/./node_modules/@toph/kernel.js/dist/plugin.js?:91
    <anonymous> webpack://toph/./scripts/plugins/codepanel.js?:49
    js http://localhost:5000/assets/js/plugins/codepanel.js?_=30b68b9fd86e53d9c8c8f259a9e434843926a78d:524
    __webpack_require__ http://localhost:5000/p/squeaky-board:49
    checkDeferredModulesImpl http://localhost:5000/p/squeaky-board:57
    webpackJsonpCallback http://localhost:5000/p/squeaky-board:55
    webpackJsonpCallback self-hosted:1224
    <anonymous> http://localhost:5000/assets/js/plugins/codepanel.js?_=30b68b9fd86e53d9c8c8f259a9e434843926a78d:9

marijn · March 23, 2021, 7:31am

Thanks for reducing that. The error-recovery confused the custom tokenizer, causing it dereference a null. Version 0.13.2 should fix this.

hjr265 · March 23, 2021, 7:40am

That works like a charm. Thank you!