Issue with copying state

I have written a tokenizer for Codemirror 5.6 for a scripting language that is based around nested functions. In order to validate whether or not certain tokens are valid I need to keep track of the current function that you are in. I do this using a stack, adding a function object to the stack when I encounter an “(” symbol and popping off the stack when I encounter a “)” symbol. This generally works fine but I am running into problems sometimes which can be illustrated with a simple case. Consider the code below:

do("a","b",
"c","d")

When this is first parsed for display in the editor, everything is fine. If I then click on the second line,"c","d") everything is still fine. If I click on the second line again then the tokenizer starts to record errors.

The reason is, when I click on the second line after the initial render it retrieves a copy of the state object which still has a copy of the stack with the “do” function in it so I know I’m in the do function. However as part of processing the line it pops the “do” function off the stack when it encounters the final “)” symbol. The next time I click on the line it is this version of the stack i.e. with no function left in it that is passed back in to the tokenizer which leads the tokenizer to believe that the “c” token is outside of a function, which in this language is an error.

I thought that CM kept a copy of the state at the end of each line and when it started to process a line it copied the state from the end of the previous line and passed it as the starting state for this line. This doesn’t seem to be the case when I click on the second line a second time.

My copyState function is shown below:

copyState(state){
        let newState = {};
        let {functionStack, ...other} = state;
        Object.assign(newState,other);
        let newFunctionStack = [];
        for (let fun of functionStack){
            let newFun = Object.assign({}, fun);
            newFunctionStack.push(newFun);
        }
        newState.functionStack = functionStack;
        return newState;
    }

Any suggestions on what I am doing wrong ?

Having read my own post I can see the obvious error !!! :blush:

Also, for this kind of thing, if the scope objects are immutable, a linked list solution is often practical—i.e. an object with the necessary properties plus a parent property pointing at the parent scope, where popping off a scope just moves the state field over to the parent scope.