tokenizer
v1.0.0 AI & Deep LearningByte-Pair Encoding (BPE) text tokenizer for transformer models.
Publisher: Rohit Pathak (TEZZCORP PRIVATE LIMITED)
•
License: MIT
•
Downloads: 1,419
Installation & Inclusion
Run from your project root:
tezz install tokenizer
Or add directly to tezz.mod:
dep.tokenizer = 1.0.0
Example Usage in TezzNative
import tokenizer
fn main() -> int:
say "Loaded TezzNative library: tokenizer v1.0.0"
ret 0
Cryptographic Integrity (SHA-256)
80b6b9d5d29f0146be48a621c8ccdb7d89550253eab279942d1317f08369c5fb
Package Metadata
- Version: v1.0.0
- Category: AI & Deep Learning
- License: MIT
- Downloads: 1,419
- Source: tokenizer.tn
- Company: TEZZCORP PRIVATE LIMITED