tokenizer

v1.0.0 AI & Deep Learning

Byte-Pair Encoding (BPE) text tokenizer for transformer models.

Publisher: Rohit Pathak (TEZZCORP PRIVATE LIMITED) • License: MIT • Downloads: 1,419

Installation & Inclusion

Run from your project root:

tezz install tokenizer

Or add directly to tezz.mod:

dep.tokenizer = 1.0.0

Example Usage in TezzNative

import tokenizer

fn main() -> int:
  say "Loaded TezzNative library: tokenizer v1.0.0"
  ret 0

Cryptographic Integrity (SHA-256)

80b6b9d5d29f0146be48a621c8ccdb7d89550253eab279942d1317f08369c5fb