StringStream.match, token is not the matched string

I have a text like this:

AAAAAA [image,] BBBB

I want get parse result like this:

AAAAAA [image,] BBBB
↑ text ↑ image                       ↑ text

I wrote a stream parser like this:

const language = StreamLanguage.define<{}>({
  startState() {
    return {}

  token(stream, state) {
    let m = stream.match(/\[image,[^\]]+\]/) as RegExpMatchArray
    if (m) {
      return `image`
    return `text`

Then I got this:

AAAAAA [image,] BBBB
↑ image                 ↑ text

// image token is "AAAAAA [image,"
// stream.string is "AAAAAA [image,]"
// m[0](matched string) is "[image,]"

I found that, when stream.match matched, i will got a token from pos 0 to length of the matched string. Why?
I had used serval parse system, it’s first time i see one works in this way.

Put a ^ at the start of your regexp to make sure it only matches the token at the start of the stream.

Thanks, but the token not in the start of the line.
Working on a polyfill to the stream, and close to success.

I still don’t know why the stream is so different than others, including can’t read a ‘\n’ from the stream.

That doesn’t matter, the docs for StringStream.match tell you to always anchor it with a ^.

The right solution:

Put a ^ at the start of regex

My old solution(do not use it, just for record):

store text in state, and clear state.text when eol:

      state.text +=

      if (stream.eol()) {
        state.text = ''

replace stream.match code with regexp.exec:

          // let m = stream.match(/\[image,[^\]]+\]/g) as RegExpMatchArray
          let m = /\[image,[^\]]+\]/g.exec(state.text) as RegExpMatchArray
          if (m) {
            state.nextN = [`image-${}`]
            stream.start -= m[0].length
            state.text = ''
            return `image`

Now it works:

OK, I will try it.

I’m sorry, you are right, it’s a better solution.
The last few days with lezer have not been very pleasant, it made me a little grumpy, hope you don’t mind.