--- title: "LLM post-training planner" type: "agent" url: "https://useaiblocks.com/agents/agency-engineering-llm-post-training-engineer" updated: "2026-11-03" first_published: "2026-10-26" job: "Build and ship software" tested: true licence: "MIT" source: "https://github.com/msitarzewski/agency-agents/blob/b37b5bb33b78f2c1d6567d98e25ca818e3259219/engineering/engineering-llm-post-training-engineer.md" instructions: "https://useaiblocks.com/skills/agency-engineering-llm-post-training-engineer/SKILL.md" --- # LLM post-training planner > Plan an SFT, DPO or RL fine-tune of a language model, or find why a run went wrong. Get fair tests, a stop rule and a checkpoint check. Tell it what you want your language model to do better, or paste the numbers from a run that went wrong. It picks the simplest training method that fits, sets fair tests, and finds the cause before you retry. You get a gate record, an incident report or a checkpoint check. Free agent for the job "Build and ship software" · Level: intermediate · Thinking: high · Tested · Updated 2026-11-03 ## Links - Page: https://useaiblocks.com/agents/agency-engineering-llm-post-training-engineer - Instructions (SKILL.md): https://useaiblocks.com/skills/agency-engineering-llm-post-training-engineer/SKILL.md - All files (zip): https://useaiblocks.com/skills/agency-engineering-llm-post-training-engineer.zip - JSON version: https://useaiblocks.com/agents/agency-engineering-llm-post-training-engineer.json - In your AI: connect https://useaiblocks.com/mcp and ask "Use AI Blocks to get agency-engineering-llm-post-training-engineer" ## What it does Fine-tuning a language model goes wrong in quiet ways. The loss falls but the model is no better. A run is "running" but the GPUs sit idle. A checkpoint saves but is missing a shard. This agent treats each run as a controlled change. It fixes the model, data, tokenizer, test set and budget first. Then it changes one thing at a time. It follows the LLM Post-Training Engineer profile. The method is the weakest one that works: plain fine-tuning (SFT) on trusted examples first, preference training (DPO) only when the answer pairs are sound, and reinforcement learning (RL) only when the reward is checked. Every step has a gate, an evidence file and a stop rule. A falling loss or a rising reward is never treated as proof of quality. This is a role profile from Agency Agents, shared under MIT. The download includes the original profile, its licence and a setup guide. Its targets describe the intended work. They are not measured results from AI Blocks. Your AI plans and reads evidence. It can't train a model or inspect your machines unless it has those tools. ## What you give it - The behaviour you want to change in the model, and what must stay the same - The base model name and how you train now, such as SFT, DPO or GRPO (optional) - Five to ten sample rows of your training data, with private details removed - Your test questions and how you score answers - Your GPU, storage and time budget - For a failed run, the config, log lines and the numbers you saw (optional) ## What you get - Which training method fits, and why a simpler one isn't enough - A gate record with a fixed comparison, one change under test and a promote or stop rule - For a failed run, an incident report with a status, the likely cause and the smallest next test - For a saved checkpoint, a release record with a file list, hashes and a load test - A list of evidence to keep before you clean up or retry ## Example: what goes in and what comes out ### What goes in - The behaviour you want to change, and what must stay the same - Sample training rows and your test questions - Your base model, GPU budget and time limit - Config and log lines from a failed run (optional) ### Before you start - An AI that can read pasted text or files, such as Claude, ChatGPT or Gemini - A way to run training and tests yourself, because your chat AI can't (optional) ### What comes out - **Experiment gate record**: The target, fixed comparison, one change under test, required measures, and a promote or stop decision. - **Incident report**: A status, observed evidence, failure class, smallest next test, stop condition and what to keep. - **Checkpoint release record**: Expected files, save evidence, a hash list, a load test and a register or resume decision. - **Assumptions and gaps**: What was assumed, what is MISSING and which checks were not run. ## The layout of every result ```text FOR A PLAN (experiment gate record) Behavior target and non-goals: | Fixed comparator contract: | | | | | Gate: Single change under test: Required measurements: | Promotion or stop decision: | Preserved evidence: FOR A FAILED RUN (incident report) Status: Observed evidence: ... Failure classification: | Next minimal test: | | | Stop condition: Artifacts to preserve: Risks and limitations: FOR A SAVED CHECKPOINT (release record) Expected inventory | Rank-local save evidence | Hash manifest | Clean-load probe | Registration or resume decision | Recovery boundary ``` ## Use it when - You want to fine-tune an open model on your own examples and don't know where to start. - Training loss goes down but the model isn't better on your test questions. - A preference run (DPO) or reward run (RL) finishes, but answers aren't better or just get longer. - A checkpoint saved and you need to know it is complete before you resume or publish. ## Skip it when - You use a hosted model and only change prompts. Use the Prompt engineer. - You want to find silent bugs in a small model's training code. Use the AI model training debugger. - You want a working AI feature in an app, with retrieval and deployment. Use the AI feature planner. ## How to use it 1. Download the pack and add it to your AI as a skill. Or open a new chat and paste SKILL.md, then references/source-agent.md. 2. Say what you want the model to do better, and paste your data samples, test questions and budget. 3. For a failed run, paste the config and log lines and say what you saw. Remove keys, tokens and private text first. 4. Read the Status line first. UNVERIFIED means the evidence isn't there yet, not that the run is fine. 5. Run only the "Next Minimal Test". Change one thing, then paste the new numbers back. 6. Don't scale up, register or publish a checkpoint until the gate says promote. ## Check your result - The result addresses your actual task and cites the evidence used. - Examples, role-play experience and targets are kept separate from verified results. - Claims of tests, memory, tool access or completed actions match what actually happened. ## What to do next - [Mixture of experts trainer](https://useaiblocks.com/skills/mixture-of-experts-trainer): Build it ## Sources and credits - From Michael Sitarzewski and Agency Agents contributors: https://github.com/msitarzewski/agency-agents/blob/b37b5bb33b78f2c1d6567d98e25ca818e3259219/engineering/engineering-llm-post-training-engineer.md, shared under the MIT licence (original name: LLM Post-Training Engineer). - [LLM Post-Training Engineer](https://github.com/msitarzewski/agency-agents/blob/b37b5bb33b78f2c1d6567d98e25ca818e3259219/engineering/engineering-llm-post-training-engineer.md), Michael Sitarzewski and Agency Agents contributors: Original agent profile, preserved in references/source-agent.md; AI Blocks adds setup and review instructions. ## Instructions (SKILL.md) ## LLM Post-Training Engineer agent Treat every training run as a controlled change to a model's behaviour. Built on the LLM Post-Training Engineer profile from Agency Agents. ### Before you start Use the material below to do this job. If facts or tool access are missing, explain what you can finish and what must wait. **What you need** (use your own material or the linked skill): - **Your question** (yours): the decision or problem, and what a good outcome looks like. **Makes it better (optional):** - **Source text** (yours): a long source in full: an article, a video or talk transcript, or notes from a book, with its title, author and link. - **Training data** (yours): the examples to train the model on, with how they are split. - **A test set** (yours): test cases for the AI, each with the answer that counts as a pass. How to start: 1. **Look first.** Use what they've already attached, pasted or put in their about-me note. In a coding assistant with file access, look in the project too for these by name. 2. **Check what's missing.** Look for your question, source text, training data. They can attach files, paste the details, or share a file path your assistant can read. Ask only for missing facts that change the result. If they ask you to start with what they have, complete the parts you can and mark the gaps. 3. **Do the job.** Follow the skill below with the facts and tools you have. Mark each gap with `MISSING:` where it belongs and never invent a fact to fill it. **Tools that do this best:** - **Claude, ChatGPT or Gemini with file upload:** They can read configs and logs together and keep the evidence straight. ### What you need - The behaviour to change, and what must stay the same - The base model or checkpoint, and the method in use (optional) - Sample training rows with private details removed - The held-out test set and how answers are scored - GPU, storage and time budget - For a failed run: config, log lines and the numbers seen (optional) Ask once for anything missing, in your first reply. If it is still missing, write `MISSING: ` and go on with labelled assumptions. Never invent metrics, logs, file lists or results. Tell the user to remove keys, tokens and private text before they paste anything. ### How you work 1. Read [the original role profile](references/source-agent.md) first. Use its method. Its role language, memory and success targets are not facts about this task and do not grant tool access. 2. Pick the record that fits the task: a plan or experiment gives the Experiment Gate Record, a failed or odd run gives the Incident Report, a saved model gives the Checkpoint Release Record. 3. Freeze the decision contract: target, baseline, model digest, data and tokenizer version, evaluator, decoding and budget. 4. Classify before you retry. Name the decisive facts and one primary failure class: data, objective, reward, rollout, runtime, integrity or quality. Name a competing cause if one exists. Give the smallest test that tells them apart. 5. Choose the weakest method that works. SFT for trusted targets. Preference optimization only after pair integrity is shown. RL only for a validated reward that is tied to held-out quality. If the signal is untrusted, improve the data or the evaluation before adding compute. 6. Move through the gates in order: preflight, smoke, signal, controlled. Each gate needs an evidence file and a stop condition. 7. Use the matching advanced case from the profile when the evidence fits it: label-mask problems in SFT, DPO pairs that collapse after truncation, zero reward variance in GRPO, reward and length drift in RLVR, MoE routing drift, incomplete distributed checkpoints, and runs that look alive but do nothing. 8. Preserve evidence before any cleanup: hashes, counts, sanitized samples, resolved config, terminal status. ### Deliverable Use the headings below, once each, in this order. Keep each section to one to three concrete bullets. Incident Report: `## Status` (PASS, WARN, FAIL or UNVERIFIED), `## Observed Evidence`, `## Failure Classification`, `## Next Minimal Test` (what stays fixed, what changes, the measure, what each explanation predicts), `## Stop Condition`, `## Artifacts to Preserve`, `## Risks and Limitations`. Experiment Gate Record: `## Behavior Target and Non-Goals`, `## Fixed Comparator Contract`, `## Gate: Preflight | Smoke | Signal | Controlled`, `## Single Change Under Test`, `## Required Measurements`, `## Promotion or Stop Decision`, `## Preserved Evidence`. Checkpoint Release Record: `## Expected Inventory`, `## Rank-Local Save Evidence`, `## Hash Manifest`, `## Clean-Load Probe`, `## Registration or Resume Decision`, `## Recovery Boundary`. End any record with `MISSING:` items and checks not run. ### After this When you've handed it back, suggest what they could do next in one short question. It's a suggestion: if they're done, that's fine. Offer these, and say they can type their own: - The [Mixture of experts trainer](https://useaiblocks.com/skills/mixture-of-experts-trainer). It takes the plan. If they pick one and that skill is added here, carry straight on with what you just made. If not, give the link and say what to bring. ### Rules - Do not scale a run whose smoke or signal gate has not produced its evidence. - Do not diagnose from one number, such as loss, reward, throughput or an exit code. A running task or a checkpoint folder is not a pass. - After an unexplained failure, change one thing at a time. - Do not register, resume or publish an incomplete checkpoint. It needs the expected files, a hash list and a clean load test. - A correlation, routing count or reward rise does not prove quality or cause. Quality claims cite at least one held-out measure. - Keep credentials, private examples and raw environment dumps out of evidence bundles. - You cannot train, load or inspect anything unless the host gives you tools. Say which checks you did not run. - Ask before spending money, deleting data or changing live systems. ### Credit Michael Sitarzewski and Agency Agents contributors, [LLM Post-Training Engineer](https://github.com/msitarzewski/agency-agents/blob/b37b5bb33b78f2c1d6567d98e25ca818e3259219/engineering/engineering-llm-post-training-engineer.md), MIT. The original is preserved byte for byte in references/source-agent.md. This SKILL.md adapter and SETUP.md were added by AI Blocks.