--- title: "AI training data labeling planner" type: "skill" url: "https://useaiblocks.com/skills/ai-training-data-labeling-planner" updated: "2026-10-28" first_published: "2026-10-25" job: "Teach and train people" tested: true licence: "MIT" source: "https://github.com/magnus919/agent-skills/tree/92c327e525da69c9ad3812fd223a1770e977c2e3/training-data-annotation" instructions: "https://useaiblocks.com/skills/ai-training-data-labeling-planner/SKILL.md" --- # AI training data labeling planner > Plan how to label data for your AI. Share the task and sample records, with personal details removed, and get label rules and quality checks. Share the task your AI will learn, example records, and labeling limits. It defines labels, edge cases, reviewer guidance, and a fair way to sample records. You get a labeling plan and quality checks that preserve uncertainty and disagreements instead of forcing agreement. Free skill for the job "Teach and train people" · Level: intermediate · Thinking: high · Tested · Updated 2026-10-28 ## Links - Page: https://useaiblocks.com/skills/ai-training-data-labeling-planner - Instructions (SKILL.md): https://useaiblocks.com/skills/ai-training-data-labeling-planner/SKILL.md - All files (zip): https://useaiblocks.com/skills/ai-training-data-labeling-planner.zip - JSON version: https://useaiblocks.com/skills/ai-training-data-labeling-planner.json - In your AI: connect https://useaiblocks.com/mcp and ask "Use AI Blocks to get ai-training-data-labeling-planner" ## What it does This is for anyone preparing data to teach an AI a task, such as sorting support tickets or tagging photos. Labeling means a person marks each example with the right answer, so the AI can learn from it. The skill sets out what each label means, with clear examples and rules for odd cases. It plans how to pick fair samples and how to train and check your reviewers. It keeps real disagreement visible instead of hiding it. It also records where the data came from and what rights you have. You get a plan you can hand to reviewers. Four terms. A held-out sample is a set of records you keep aside to check quality later. Adjudication is a named person settling a disagreement by a stated rule. Abstention is an "unsure" label reviewers may pick. Out-of-distribution means unlike the examples the AI has seen. Freezing the manifest means locking the list of records and labels so it stops changing, and giving it a version. ## What you give it - The downstream task, example items, population, rights and access limits, budget, and reviewers. ## What you get - A label schema, reviewer guide, sample plan, provenance record, and quality review plan. ## Example: what goes in and what comes out ### What goes in - The task your AI will learn - A few example records - Where the data comes from and what rights you have - Your budget and who will review (optional) ### Before you start - An AI that can read the downloaded folder of reference files and templates ### What comes out - **Annotation plan**: The task, the label unit, the sampling mix, the held-out check sample, and when to stop. - **Annotation guide**: Label choices with good and bad examples, edge case rules, an unsure option, and when to escalate. - **Provenance and rights record**: Where data came from, license or consent, sensitive fields, access, how long you keep it, and version. - **Reviewer training and calibration**: How reviewers are qualified, trained on sample items, and rechecked blind over time. - **Quality report**: Agreement by label and slice, disagreements kept visible, how they were settled, and guide fixes. ## Use it when - You are about to have people label examples to teach an AI a task. Remove names and personal details from any records first. - Your reviewers disagree and you do not know why. - You need a written record of where your data came from and what rights you have. - You want a plan you can hand to a team of reviewers. ## Skip it when - You need statistical power or sample size maths: ask a data scientist or a stats tool. - You want to train or test a model: use your machine-learning engineer or a training tool. - You need to store data or build a data pipeline: ask a data engineer. ## How to use it 1. Do these in order. 1) Gather the task, 10 to 20 example records, and the rights and reviewer details. 2) Check your rights. 3) Remove personal data. 4) Load the folder into your AI. 5) Ask for the four outputs, which are label rules, reviewer guidance, sampling and quality checks. 2. What you get is an annotation plan, an annotation guide, a rights and provenance record, reviewer training, and a quality report. 3. Before you start, write down the task the AI will learn, 10 to 20 example records, where the data came from and what you may do with it, who will review, and your budget and deadline. 4. Check your rights before you upload anything. Confirm you have consent or a licence to use these records, and that sharing them with a third-party AI tool is allowed under your data-protection rules. If you are not sure, ask a qualified person (your privacy lead or a lawyer) to sign off first. 5. Remove names, contact details and other personal fields from your example records before you paste them into any AI tool. Note which fields are sensitive. 6. Load the downloaded folder into your AI. You need an AI with file upload or a project folder feature, so it can read the reference files and templates. If yours cannot, paste in SKILL.md and the reference files one by one. 7. Give the task, example records, rights limits, reviewers, budget, and what a useful label means. 8. Ask for label rules, reviewer guidance, sampling, and quality checks. The label rules come as a table with these columns. Label, meaning, a good example, a bad example and a rule for edge cases. 9. Ask it to end with a short list headed "Open questions" that names each unresolved rights issue and each reviewer issue. 10. The source also names separate jobs for statistics, data storage, model training, and interface design. Those optional specialist skills are not required for labeling and are not included. ## What to do next - [Dataset profiler](https://useaiblocks.com/skills/dataset-profiler): Dataset profiler - [Tabular dataset explorer](https://useaiblocks.com/skills/new-dataset-explorer): Tabular dataset explorer ## Sources and credits - From Magnus Hedemark: https://github.com/magnus919/agent-skills/tree/92c327e525da69c9ad3812fd223a1770e977c2e3/training-data-annotation, shared under the MIT licence (original name: training-data-annotation). - [training-data-annotation skill](https://github.com/magnus919/agent-skills/tree/92c327e525da69c9ad3812fd223a1770e977c2e3/training-data-annotation), Magnus Hedemark: Copied under the MIT licence. We gave it a plain name. We added Before you start and After this sections that link to other AI Blocks steps. ## Instructions (SKILL.md) ## Training Data Annotation Treat annotation as a measurement system and a human workflow. The objective is useful, auditable labels for a defined downstream decision, not the largest possible label count. ### Before you start Use the material below to do this job. If facts or tool access are missing, explain what you can finish and what must wait. **What you need** (use your own material or the linked skill): - **Your question** (yours): the decision or problem, and what a good outcome looks like. **Makes it better (optional):** - **Your data** (yours): the data files, with their format and where they came from. How to start: 1. **Look first.** Use what they've already attached, pasted or put in their about-me note. In a coding assistant with file access, look in the project too for these by name. 2. **Check what's missing.** Look for your question, your data. They can attach files, paste the details, or share a file path your assistant can read. Ask only for missing facts that change the result. If they ask you to start with what they have, complete the parts you can and mark the gaps. 3. **Do the job.** Follow the skill below with the facts and tools you have. Mark each gap with `MISSING:` where it belongs and never invent a fact to fill it. **Tools that do this best:** - **A chat AI that can read the bundled reference files:** Use the supplied references and templates for the chosen task. Scripts are optional unless that task specifically needs them. ### Workflow 1. Define the downstream task, population, label unit, label set, edge cases, abstention label, and what constitutes an unusable item. Separate training, validation, held-out evaluation, and challenge data before sampling. 2. Write an annotation guide with positive and negative examples, boundary rules, source requirements, escalation, and a change log. Have domain owners approve consequential definitions. 3. Establish provenance and rights: source, collection time, license/consent, transformations, sensitive fields, access, retention/deletion, and dataset version. Do not move data into a labeling system until this record is clear. 4. Acquire items using a declared mix of representative sampling and targeted acquisition. Uncertainty finds confusing items; diversity and novelty find gaps; stratification protects rare or high-risk slices. Keep acquisition batches identifiable and retain a representative held-out audit sample. 5. Qualify annotators for the task, train them on calibration items, and measure quality and workload. Record role and relevant expertise in restricted metadata. Do not use agreement as a universal truth score. 6. Run blinded calibration and periodic blind rechecks. Compare labels with adjudicated references where available; preserve meaningful disagreement, ambiguity, and abstention instead of forcing consensus. 7. Review disagreement by item, label, slice, annotator, and guide version. Adjudicate (settle a disagreement) only with a stated rule and a person qualified to decide. Feed confirmed guide defects back into training and re-review affected labels. 8. Stop or continue based on marginal downstream value, coverage, quality, cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the manifest (lock the list of records and labels so it stops changing, and give it a version) and hand off to data-engineering for storage/orchestration and ml-engineering for training use. ### Acquisition guardrails Never let model confidence define the whole dataset. Sample a representative held-out audit independently of active-learning selection, and compare active batch outcomes against it. Track uncertainty method, model/version, seed, selection scores, slice quotas, and excluded items. If confidence is uncalibrated or the model is out of distribution (the records are unlike anything it was trained on), treat its ranking as a search heuristic and increase diversity, novelty, and human review. ### Human and quality guardrails Design pay, volume, time limits, breaks, and escalation so workers can perform the task reliably; route employment, compensation, and legal questions to the appropriate human owner. Blind annotators to model predictions when measuring independent labels. If preannotations are shown, measure anchoring by a blinded control or separate review pass. Report agreement by label type and slice with the uncertainty and prevalence needed to interpret it. A minority label can be correct when the item is ambiguous; adjudication must explain the rule used. ### Routing and exit Load [references/task-and-label-design.md](references/task-and-label-design.md) for schemas and guide design, [references/acquisition-and-stopping.md](references/acquisition-and-stopping.md) for sampling and stopping, [references/annotator-quality.md](references/annotator-quality.md) for people and disagreement, and [references/provenance-and-rights.md](references/provenance-and-rights.md) for dataset authority. Use the templates when a durable record is needed. For primary methodology sources and transfer limits, read [references/source-index.md](references/source-index.md). Route statistical estimators and power to [data-scientist](../data-scientist/SKILL.md), storage and pipelines to [data-engineering](../data-engineering/SKILL.md), model training and leakage checks to [ml-engineering](../ml-engineering/SKILL.md), and labeling-interface interaction design to [product-design-and-ux](../product-design-and-ux/SKILL.md). ### When not to use Do not use this skill as the primary owner for statistical inference or experimental design, data platforms, model training, or UI implementation. Those neighboring skills consume this skill's annotation manifest and quality evidence. ### Templates - [annotation guide](templates/annotation-guide.md) — use when recording this stage of the annotation workflow. - [annotation plan](templates/annotation-plan.md) — use when recording this stage of the annotation workflow. - [quality report](templates/quality-report.md) — use when recording this stage of the annotation workflow. ### After this When you've handed it back, suggest what they could do next in one short question. It's a suggestion: if they're done, that's fine. Offer these, and say they can type their own: - The [Dataset profiler](https://useaiblocks.com/skills/dataset-profiler). - The [Tabular dataset explorer](https://useaiblocks.com/skills/new-dataset-explorer). If they pick one and that skill is added here, carry straight on with what you just made. If not, give the link and say what to bring.