AI Blocks › Build and ship software
Model preference tuning guide
Give it a model and examples of good and bad answers. Get a model tuned to answer the way you prefer.
Model preference tuning guide is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: expert · Thinking: high
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A language model tuned to answer the way you prefer, taught with examples of good and bad answers. You choose the method that fits your data and compute. The skill uses the TRL library. If you have pairs of preferred and rejected answers, it sets up DPO, the simplest route. If you want the classic path, it walks through three stages: teach basic instructions, train a reward model that scores answers, then tune with PPO. It also covers GRPO for tasks with checkable answers. It adds LoRA to cut memory use and ends by testing the tuned model against the original. You need Python and a GPU.
You give it
- A base model
- Pairs of chosen and rejected answers
- A GPU and a Python setup
- The behavior you want to see (optional)
You get
- Training code for the right method
- A dataset format checklist
- LoRA settings to save memory
- An evaluation against the original
- Advice on which method suits your data
Use it when
- Your model gives answers that are correct but the wrong tone.
- You have a pile of thumbs up and thumbs down examples.
- You want a small model to follow your house style.
- You need to pick between DPO, PPO and GRPO.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code on a computer with a GPU.