How can I extend modes so that their token performs an action at end of line?

I’m working on a mode that extends several built-in language modes (namely, YAML, JavaScript and C-like), whose token functions sometimes uses stream.skipToEnd() and then returns a token. I’d like to be able to have some way in my mode extension to know when the end of the stream is reached so that I can handle it and do some other actions. Is there a way to hook into that?


You could write a wrapping mode which, after it calls the inner mode’s token method, check for stream.eol().

Great, that totally helped - but I have a follow up question, and quite possibly I’m looking at my problem in the wrong way.

So if I set state at end of line, the next line picks up that state. Which is what I expect. But I’m using that as a sort of reset, so on the next line, I’d like to look ahead at the stream and see if I should either keep that state at position 0 or not. The problem is if I set anything on the state during stream.sol() or stream.pos === 0, this state doesn’t take effect until {line: x, ch: 1} whereas I want to take affect during {line: x, ch: 0}.

Practically, this is what I’m trying to do:

I have a YAML file where many values are multi-line blocks containing other code snippets (like JavaScript or C-like language). I’m taking advantage of multiple inner modes to format those code snippets as well as the YAML. But I want the JavaScript and C-like modes to be scoped to the lines that they’re on. If a multi-line JavaScript block ends and the next line is another YAML section, the state at {ch: 0} of the YAML line still thinks it’s in JavaScript - so if I try to do a line comment, for example, at the start of that line, it comments with the Javascript // instead of the YAML #. Only if I comment at {ch: 1}, then it uses the correct YAML comment character. So I was previously trying to reset the state at the beginning of each line by doing it at the end of each line, but that just traded off one set of problems for another. What’s the best way to approach this?


You can return from a tokenizer without advancing the stream, in order to update the state for a given position. Just make sure the next call does advance the stream, or the mode runner will, after a few no-op tokens, raise an error.

In my token function, how do I return without advancing the stream, after updating state for {line: x, ch: 0} ? Even if I return null at stream.pos === 0 or stream.sol(), it does not update the state at that position. Am I doing this wrong?

You’re right, getToken() and similar won’t use the updated state in this case. There isn’t really an easy way to get at the state like that at the moment.