CodeMirror 6: stream-syntax with custom tags

In an attempt to create a Markdown syntax with the new CM6, I’ve been using the stream-syntax module to parse by line. Then I created a custom TagSystem to house tags and types for markdown tokens, as they are quite different from standard programming token types.
The issue I’m having is that the stream-syntax module has hard-coded imports of defaultTags and therefore does not recognize my custom tags I have defined.
I believe this should be parameterized, maybe adding an argument to the StreamSyntax constructor.

That would make sense. However, there’s also a good point to be made for trying to fit your tokens in the default tags, so that they work with regular themes. What kind of tokens do you need that aren’t covered by the default set?

Your point is fair, and in a normal use case, like all editors do, I’d make it work with what token types I already have. I was thinking about including sub-syntaxes and at the same time to style markdown tokens differently.

I’ll use an example to explain: let’s say that I assign all header tokens to keyword. In markdown I’d like to scale the font-size of these tokens based on the header type(ex; H1 -> 32px, H2 -> 24px and so on). The current implementation allows me to do this with the subtype feature, and it’s great. But here comes the problem: suppose I have a code block with some javascript code with keywords in it. The sub-highlighter would give them the keyword type and therefore they’d recieve the font-size values I mentioned above.

I am aware it’s a specific use case and a future markdown implementation would most likely use the default set of token types, but the ability to provide a custom TagSystem to the StreamSyntax would make it much more sane.

That is definitely a good idea too. I’ve opened #275 to track it.

I am looking for something exactly like this. Can you provide a quick example on how can I add my custom tags in my StreamLanguage?
As per the commit, I am trying to define myCustomTags in the tagSystem property of my StreamParser but it is not working. Also I cannot find this anywhere in the documentation or examples.
Here is what I have done so far:

const hissabTags = {
controllerToken: Tag.define(),
functionToken: Tag.define(),
}

const HissabHighlightStyle = HighlightStyle.define([
{tag: hissabTags.functionToken, color: “blue”},
{tag: hissabTags.controllerToken, color: “brown”},
])

const hissabLang = {
token(stream) {
//some logic
return “functionToken”
// OR
return “controllerToken”
},
tagSystem: hissabTags
};

Getting error -

Unknown highlighting tag functionToken

Hello,
Just checking if anyone can help me here. thanks!

For anyone else looking for similar solution, I found one. I extended the tags object with my own custom tags and was able to use those successfully.

const customTags = {
  tagA: Tag.define(),
  tagB: Tag.define(),
};

Object.assign(tags, customTags);

export const customHighlightStyle = HighlightStyle.define([
  { tag: tags["tagA"], color: "#5C6E74" },
  { tag: tags["tagB"], color: "#DD4A68" },
]);

That’s assigning to library-internal objects though, which is a bit dodgy.

Does the patch below look like it’d address this use case for you?

Yes, this seems perfect and would work for my use case.
Whats the status of this patch and release plans?

Glad to hear. I’ve tagged 0.19.4

1 Like

Hello @marijn
I have been using custom tokens in my stream parser currently which is working fine during development. However during production there is no syntax highlighting.
My suspect is that if the custom tag name is being used as is in your patch, then in the production build due to minification/uglification the names will change. This may be breaking the prod builds.

Is this an easy/quick fix and something you have bandwidth for? if not its fine, just lmk and I will revert to using builtin tags until there is a fix.

Minifiers do not, generally, change property names, unless they have enough type information to make sure it’s safe (I think Clojure does something like that?), so I’m not sure what might be going on here.

For those who may be confused how to do this … I think I’ve got it:

import { Tag } from '@lezer/highlight'

... = {
  tokenTable: {
    'my-tag': Tag.define()
  }
}