AI Blocks › Build and ship software
Small model trainer from a big one
Name your large model. Get a plan and code to teach a smaller, cheaper model to match it.
Small model trainer from a big one is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: expert · Thinking: high
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A plan and working code to teach a small, cheap model to copy a big one. You name the large teacher model and the small student, and you get a path to keep most of the quality at a fraction of the running cost. The skill covers the main methods. Soft targets with temperature scaling let the student learn from the teacher's full probability spread, not only its top answer. Logit distillation matches raw outputs. Response distillation trains on text the teacher wrote. A reverse KL method suits language models. It uses Python with transformers and torch, and training needs real GPU time. Expect to keep around 90 percent of performance when it goes well.
You give it
- The teacher model
- The student model you want to train
- Training data or prompts for the task
- GPU access and a Python setup
You get
- A distillation plan for your models
- Training code with the right loss
- Settings like temperature and loss weights
- A way to compare student and teacher
Use it when
- Your large model works but costs too much to run.
- You want an open model that handles one task as well as a paid one.
- You need a model small enough for limited hardware.
- You have teacher answers and want to train a student on them.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code on a computer with a GPU.