AI Blocks › Build and ship software
Model size reducer
Give it a large model. Get a 4-bit copy that is about 3 times faster and fits on a smaller GPU.
Model size reducer is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: advanced
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A large language model shrunk to 4-bit, running about three times faster and fitting on a smaller GPU, with only a small loss in answer quality. The skill uses AWQ, which looks at how the model behaves on sample text and protects the weights that matter most before shrinking the rest. You can load a ready-made small copy or make your own with a short Python script and a few hundred calibration samples. It compares AWQ with GPTQ and bitsandbytes so you pick the right one, and it covers serving with vLLM. Fast kernels need a recent Nvidia GPU.
You give it
- The model you want to shrink
- Your GPU type and memory
- A small set of sample text for calibration
- Your speed and quality targets (optional)
You get
- A 4-bit version of the model
- The script to make it
- Serving steps for vLLM
- Before and after speed and quality checks
- A method comparison
Use it when
- Your model does not fit in your GPU memory.
- Answers are too slow for a chat product.
- You want to serve a chat model cheaply with vLLM.
- You need to decide between AWQ, GPTQ and bitsandbytes.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code on a computer with a GPU.