How to match consecutive invalid characters as a single token ?

1

I have a custom mode which is defined like this (to colorize a script) :
EDIT : I fixed it (see solution below). I will not delete topic in case it would be useful for someone else. If there is a better solution, please let me know about it.

function tokenBase(stream, state) 
{		
	if (stream.eat(...)) return ...; //punctuation, ...
	if (stream.eat(...)) return ...;
	...

	if (stream.match(some_regex)) return ...; //constant, identifiers, ...
	if (stream.match(some_regex)) return ...;	
	...
	
	//take care of comments
	...
	
	//anything not matched so far is considered as an error (will be underlined)
	return "error";
}

It works fine, the issue is for errors (usually not allowed characters) which get consumed one by one.

For example, a script containing $$$$$$ will result in something like this in the editor:
<span class="error">$</span><span class="error">$</span><span class="error">$</span>...

I would like to have those errors to be grouped together:
<span class="error">$$$$$$</span>

This is annoying because I also reuse that tokenizer for other things and I would like consecutive errors to be considered as a single token.

EDIT : here is solution:

if (stream.match([/^only_allowed_chars]+/))
      return "error"; //try to consume several errors at once
return "error"; //all the other cases
2

The editor will group adjacent tokens with the same style into a single DOM element by default, so I assume you have the flattenSpans option turned off explicitly?

3

flattenSpans : I didn’t know it existed.
Here is my config :

CodeMirror(editorDiv, 
{		
	lineNumbers: false,
	showCursorWhenSelecting: false,
	lineWrapping: true,
	configureMouse: (cm, repeat, ev) =>
	{ 
		return { addNew : false };
	},
	mode: "mymode"
});

And here is what I got from Chrome if same token is reported repeatedly :

image

I have tried to explicitly set flattenSpans to false but that does not change anything.
I use 5.63.3 which is latest version.

4

Odd. If I use a dummy mode that just assigns "underline" to one character at a time, I get a single big span per line. Can you reproduce this outside of your system in a small script?

5

I found what’s wrong : I forgot I would later call markText() on each error.

this.cm.markText(
	{ line: ..., ch: ... },
	{ line: ..., ch: ... },
	{ 	
		className:"cm-underline",
		__annotation: error.message
	});

Removing className:"cm-underline" prevent splitting the grouped <span> text into smaller chunks (which expected behavior).