AI Blocks › Build and ship software
Tokenizer trainer
Give it your text. Get a custom word-splitting tokenizer trained on it, with padding and length limits set up.
Tokenizer trainer is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: advanced
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A tokenizer built from your own text, so a model sees your words, codes or product names as clean pieces instead of broken fragments. It is useful when general tokenizers split your specialist vocabulary badly. The skill uses the Hugging Face Tokenizers library. You pick a method: BPE, WordPiece or Unigram. It trains the vocabulary on your files, sets special tokens, and adds rules for padding and maximum length. It can track where each token came from in the original text, which helps with highlighting and entity work. It also loads ready-made tokenizers when you do not need a custom one. The library tokenizes about a gigabyte in under 20 seconds on a normal processor.
You give it
- Your text files
- Method preference: BPE, WordPiece or Unigram (optional)
- Vocabulary size
- Padding and length limits
You get
- A trained tokenizer file
- Special token setup
- Padding and truncation settings
- Token-to-text position mapping
- Test output on sample text
Use it when
- Your industry terms get chopped into odd pieces by standard tokenizers.
- You are training a model and need a vocabulary for your data.
- You need to map tokens back to positions in the original text.
- You must process a large text collection quickly.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code.