# Can tokens be zero or fractional length?

**URL:** https://discuss.codemirror.net/t/can-tokens-be-zero-or-fractional-length/7452
**Category:** Lezer
**Created:** [November 22, 2023, 7:23pm UTC](https://discuss.codemirror.net/t/can-tokens-be-zero-or-fractional-length/7452 "2023-11-22T19:23:56Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![shche](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/shche/32/1632_2.png) [@shche](https://discuss.codemirror.net/u/shche)
#### Post date: [November 22, 2023, 7:23pm UTC](https://discuss.codemirror.net/t/can-tokens-be-zero-or-fractional-length/7452/1 "2023-11-22T19:23:56Z")

</div>

Hello, can external tokenizer return zero length (or fractional length, like `0.5`) tokens? Or can it break the parser? Thanks!

---

<div class="post-metadata">

### Author: ![marijn](https://discuss.codemirror.net/user_avatar/discuss.codemirror.net/marijn/32/136_2.png) [@marijn](https://discuss.codemirror.net/u/marijn)
#### Post date: [November 22, 2023, 8:20pm UTC](https://discuss.codemirror.net/t/can-tokens-be-zero-or-fractional-length/7452/2 "2023-11-22T20:20:29Z")

</div>

They can, but you have to be really careful that your grammar doesn’t allow any zero-length thing to occur on repeat, or your parser will hang just running that production endlessly.