Challenges on implementing operational transformation server in Golang due to javascript unicode string length

v6
1

Everything works pretty well when Golang server implements codemirror/state text.ts in Go language and uses utf8.RuneCountInString, which counts how many characters there are in a string. It works very well even with many unicode characters, but now when I tested it with emoji unicode characters that you can type e.g. on MacOS like :+1:, Javascript string length starts to differ.

‘:+1:’.length is 2 and that doesn’t any longer match with rune length in Go that is 1 and applying ChangeSet fails.

I tried a quick test to fork codemirror/state and change anything related to line.length and line.slice to

export function getCharacterLength(str: string) {
  return [...str].length
}

export function sliceByCharacterLength(str: string, from: number, to?: number) {

  if (!to) {
    to = -1
  }

  return [...str].slice(from, to).join(''); 
}

But that wasn’t a solution for this, since there are other places that uses string length to access text.

Would you have any ideas what kind of solutions could be tested out?

2

CodeMirror intentionally uses UTF16 code point counts throughout, in order to be able to use String.length. If you are using another convention in the OT system, you’ll have to make sure you convert to and from that when going between CodeMirror and that system. Forking @codemirror/state is not going to be needed.

3

Thanks! I’ll test that out. Btw, nice work, CodeMirror is quality software.