# CodeMirror 6: stream-syntax with custom tags

**URL:** https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578
**Category:** Uncategorized
**Created:** [August 22, 2020, 7:56am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578 "2020-08-22T07:56:52Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![lucat1](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/lucat1/32/865_2.png) [@lucat1](https://discuss.codemirror.net/u/lucat1)
#### Post date: [August 22, 2020, 7:56am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/1 "2020-08-22T07:56:52Z")

</div>

In an attempt to create a Markdown syntax with the new CM6, I’ve been using the `stream-syntax` module to parse by line. Then I created a custom `TagSystem` to house tags and types for markdown tokens, as they are quite different from standard programming token types.  
The issue I’m having is that the `stream-syntax` module has hard-coded imports of `defaultTags` and therefore does not recognize my custom tags I have defined.  
I believe this should be parameterized, maybe adding an argument to the `StreamSyntax` constructor.

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [August 22, 2020, 3:26pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/2 "2020-08-22T15:26:21Z")

</div>

That would make sense. However, there’s also a good point to be made for trying to fit your tokens in the default tags, so that they work with regular themes. What kind of tokens do you need that aren’t covered by the default set?

---

<div class="post-metadata">

### Author: ![lucat1](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/lucat1/32/865_2.png) [@lucat1](https://discuss.codemirror.net/u/lucat1)
#### Post date: [August 22, 2020, 5:18pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/3 "2020-08-22T17:18:33Z")

</div>

Your point is fair, and in a normal use case, like all editors do, I’d make it work with what token types I already have. I was thinking about including sub-syntaxes and at the same time to style markdown tokens differently.

I’ll use an example to explain: let’s say that I assign all header tokens to `keyword`. In markdown I’d like to scale the font-size of these tokens based on the header type(ex; H1 -\> 32px, H2 -\> 24px and so on). The current implementation allows me to do this with the `subtype` feature, and it’s great. But here comes the problem: suppose I have a code block with some javascript code with keywords in it. The sub-highlighter would give them the `keyword` type and therefore they’d recieve the font-size values I mentioned above.

I am aware it’s a specific use case and a future markdown implementation would most likely use the default set of token types, but the ability to provide a custom TagSystem to the StreamSyntax would make it much more sane.

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [August 22, 2020, 5:38pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/4 "2020-08-22T17:38:22Z")

</div>

> [@lucat1](#):
>
> but the ability to provide a custom TagSystem to the StreamSyntax would make it much more sane.

That is definitely a good idea too. I’ve opened [#275](https://github.com/codemirror/codemirror.next/issues/275) to track it.

---

<div class="post-metadata">

### Author: ![prenx4x](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/prenx4x/32/1638_2.png) [@prenx4x](https://discuss.codemirror.net/u/prenx4x)
#### Post date: [December 21, 2021, 3:15pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/5 "2021-12-21T15:15:36Z")

</div>

I am looking for something exactly like this. Can you provide a quick example on how can I add my custom tags in my StreamLanguage?  
As per the commit, I am trying to define myCustomTags in the tagSystem property of my StreamParser but it is not working. Also I cannot find this anywhere in the documentation or examples.  
Here is what I have done so far:

> const hissabTags = {  
> controllerToken: Tag.define(),  
> functionToken: Tag.define(),  
> }
> 
> const HissabHighlightStyle = HighlightStyle.define([  
> {tag: hissabTags.functionToken, color: “blue”},  
> {tag: hissabTags.controllerToken, color: “brown”},  
> ])
> 
> const hissabLang = {  
> token(stream) {  
> //some logic  
> return “functionToken”  
> // OR  
> return “controllerToken”  
> },  
> tagSystem: hissabTags  
> };

Getting error -

> Unknown highlighting tag functionToken

---

<div class="post-metadata">

### Author: ![prenx4x](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/prenx4x/32/1638_2.png) [@prenx4x](https://discuss.codemirror.net/u/prenx4x)
#### Post date: [December 22, 2021, 3:52pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/6 "2021-12-22T15:52:05Z")

</div>

Hello,  
Just checking if anyone can help me here. thanks!

---

<div class="post-metadata">

### Author: ![prenx4x](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/prenx4x/32/1638_2.png) [@prenx4x](https://discuss.codemirror.net/u/prenx4x)
#### Post date: [January 8, 2022, 7:43pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/7 "2022-01-08T19:43:59Z")

</div>

For anyone else looking for similar solution, I found one. I extended the tags object with my own custom tags and was able to use those successfully.

```auto
const customTags = {
  tagA: Tag.define(),
  tagB: Tag.define(),
};

Object.assign(tags, customTags);

export const customHighlightStyle = HighlightStyle.define([
  { tag: tags["tagA"], color: "#5C6E74" },
  { tag: tags["tagB"], color: "#DD4A68" },
]);

```

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [January 11, 2022, 10:34am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/8 "2022-01-11T10:34:37Z")

</div>

That’s assigning to library-internal objects though, which is a bit dodgy.

Does the patch below look like it’d address this use case for you?

> <https://github.com/codemirror/stream-parser/commit/6904902f37a9a8e84ee0890195d61d55cdd98b8c>
>
> FEATURE: Stream parser objects now support a \`tokenTable\` property that
> allows y…ou to use non-standard highlighting tags in the parser output.
> 
> See https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578

---

<div class="post-metadata">

### Author: ![prenx4x](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/prenx4x/32/1638_2.png) [@prenx4x](https://discuss.codemirror.net/u/prenx4x)
#### Post date: [January 15, 2022, 7:45pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/9 "2022-01-15T19:45:45Z")

</div>

Yes, this seems perfect and would work for my use case.  
Whats the status of this patch and release plans?

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [January 17, 2022, 7:37am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/10 "2022-01-17T07:37:38Z")

</div>

Glad to hear. I’ve tagged 0.19.4

---

<div class="post-metadata">

### Author: ![prenx4x](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/prenx4x/32/1638_2.png) [@prenx4x](https://discuss.codemirror.net/u/prenx4x)
#### Post date: [February 15, 2022, 9:59pm UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/11 "2022-02-15T21:59:55Z")

</div>

Hello @marijn  
I have been using custom tokens in my stream parser currently which is working fine during development. However during production there is no syntax highlighting.  
My suspect is that if the custom tag name is being used as is in your patch, then in the production build due to minification/uglification the names will change. This may be breaking the prod builds.

Is this an easy/quick fix and something you have bandwidth for? if not its fine, just lmk and I will revert to using builtin tags until there is a fix.

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [February 16, 2022, 7:13am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/12 "2022-02-16T07:13:12Z")

</div>

Minifiers do not, generally, change property names, unless they have enough type information to make sure it’s safe (I think Clojure does something like that?), so I’m not sure what might be going on here.

---

<div class="post-metadata">

### Author: ![tarwin](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/tarwin/32/2464_2.png) [@tarwin](https://discuss.codemirror.net/u/tarwin)
#### Post date: [September 12, 2022, 7:13am UTC](https://discuss.codemirror.net/t/codemirror-6-stream-syntax-with-custom-tags/2578/13 "2022-09-12T07:13:39Z")

</div>

For those who may be confused how to do this … I think I’ve got it:

```auto
import { Tag } from '@lezer/highlight'

... = {
  tokenTable: {
    'my-tag': Tag.define()
  }
}

```
