AI training data labeling planner

What it does

This is for anyone preparing data to teach an AI a task, such as sorting support tickets or tagging photos. Labeling means a person marks each example with the right answer, so the AI can learn from it. The skill sets out what each label means, with clear examples and rules for odd cases. It plans how to pick fair samples and how to train and check your reviewers. It keeps real disagreement visible instead of hiding it. It also records where the data came from and what rights you have. You get a plan you can hand to reviewers. Four terms. A held-out sample is a set of records you keep aside to check quality later. Adjudication is a named person settling a disagreement by a stated rule. Abstention is an "unsure" label reviewers may pick. Out-of-distribution means unlike the examples the AI has seen. Freezing the manifest means…

What goes in and out

You give it

• The downstream task, example items, population, rights and access limits, budget, and reviewers.
• Your data (optional)

You get

• A label schema, reviewer guide, sample plan, provenance record, and quality review plan.

How the task works

• Define labels and rights. Set the unit, label choices, unclear cases, abstention option, source rights, and data access rules.
• Plan sampling and training. Keep a fair sample aside to check the results later. Train reviewers with examples before using their labels.
• Check quality and disagreement. Use blind rechecks and qualified review. Keep genuine ambiguity and disagreement visible.
• Set a stopping rule. Balance coverage, downstream value, time, cost, and quality before handing a versioned dataset to the next owner.

Full instructions

• Do these in order. 1) Gather the task, 10 to 20 example records, and the rights and reviewer details. 2) Check your rights. 3) Remove personal data. 4) Load the folder into your AI. 5) Ask for the four outputs, which are label rules, reviewer guidance, sampling and quality checks.
• What you get is an annotation plan, an annotation guide, a rights and provenance record, reviewer training, and a quality report.
• Before you start, write down the task the AI will learn, 10 to 20 example records, where the data came from and what you may do with it, who will review, and your budget and deadline.
• Check your rights before you upload anything. Confirm you have consent or a licence to use these records, and that sharing them with a third-party AI tool is allowed under your data-protection rules. If you are not sure, ask a qualified person (your privacy lead or a lawyer) to sign off first.
• Remove names, contact details and other personal fields from your example records before you paste them into any AI tool. Note which fields are sensitive.
• Load the downloaded folder into your AI. You need an AI with file upload or a project folder feature, so it can read the reference files and templates. If yours cannot, paste in SKILL.md and the reference files one by one.
• Give the task, example records, rights limits, reviewers, budget, and what a useful label means.
• Ask for label rules, reviewer guidance, sampling, and quality checks. The label rules come as a table with these columns. Label, meaning, a good example, a bad example and a rule for edge cases.
• Ask it to end with a short list headed "Open questions" that names each unresolved rights issue and each reviewer issue.
• The source also names separate jobs for statistics, data storage, model training, and interface design. Those optional specialist skills are not required for labeling and are not included.

Full instruction file

Training Data Annotation

Treat annotation as a measurement system and a human workflow. The objective is useful, auditable labels for a defined downstream decision, not the largest possible label count.

Before you start

Use the material below to do this job. If facts or tool access are missing, explain what you can finish and what must wait.

What you need (use your own material or the linked skill):

• Your question (yours): the decision or problem, and what a good outcome looks like.

Makes it better (optional):

• Your data (yours): the data files, with their format and where they came from.

How to start:

1. Look first. Use what they've already attached, pasted or put in their about-me note. In a coding assistant with file access, look in the project too for these by name.
2. Check what's missing. Look for your question, your data. They can attach files, paste the details, or share a file path your assistant can read. Ask only for missing facts that change the result. If they ask you to start with what they have, complete the parts you can and mark the gaps.
3. Do the job. Follow the skill below with the facts and tools you have. Mark each gap with MISSING: where it belongs and never invent a fact to fill it.

Tools that do this best:

• A chat AI that can read the bundled reference files: Use the supplied references and templates for the chosen task. Scripts are optional unless that task specifically needs them.

Workflow

1. Define the downstream task, population, label unit, label set, edge cases, abstention label, and what constitutes an unusable item. Separate training, validation, held-out evaluation, and challenge data before sampling.
2. Write an annotation guide with positive and negative examples, boundary rules, source requirements, escalation, and a change log. Have domain owners approve consequential definitions.
3. Establish provenance and rights: source, collection time, license/consent, transformations, sensitive fields, access, retention/deletion, and dataset version. Do not move data into a labeling system until this record is clear.
4. Acquire items using a declared mix of representative sampling and targeted acquisition. Uncertainty finds confusing items; diversity and novelty find gaps; stratification protects rare or high-risk slices. Keep acquisition batches identifiable and retain a representative held-out audit sample.
5. Qualify annotators for the task, train them on calibration items, and measure quality and workload. Record role and relevant expertise in restricted metadata. Do not use agreement as a universal truth score.
6. Run blinded calibration and periodic blind rechecks. Compare labels with adjudicated references where available; preserve meaningful disagreement, ambiguity, and abstention instead of forcing consensus.
7. Review disagreement by item, label, slice, annotator, and guide version. Adjudicate (settle a disagreement) only with a stated rule and a person qualified to decide. Feed confirmed guide defects back into training and re-review affected labels.
8. Stop or continue based on marginal downstream value, coverage, quality, cost, and time. An item quota may stop spending but cannot by itself establish sufficient quality. Freeze the manifest (lock the list of records and labels so it stops changing, and give it a version) and hand off to data-engineering for storage/orchestration and ml-engineering for training use.

Acquisition guardrails

Never let model confidence define the whole dataset. Sample a representative held-out audit independently of active-learning selection, and compare active batch outcomes against it. Track uncertainty method, model/version, seed, selection scores, slice quotas, and excluded items. If confidence is uncalibrated or the model is out of distribution (the records are unlike anything it was trained on), treat its ranking as a search heuristic and increase diversity, novelty, and human review.

Human and quality guardrails

Design pay, volume, time limits, breaks, and escalation so workers can perform the task reliably; route employment, compensation, and legal questions to the appropriate human owner. Blind annotators to model predictions when measuring independent labels. If preannotations are shown, measure anchoring by a blinded control or separate review pass. Report agreement by label type and slice with the uncertainty and prevalence needed to interpret it. A minority label can be correct when the item is ambiguous; adjudication must explain the rule used.

Routing and exit

Load references/task-and-label-design.md for schemas and guide design, references/acquisition-and-stopping.md for sampling and stopping, references/annotator-quality.md for people and disagreement, and references/provenance-and-rights.md for dataset authority. Use the templates when a durable record is needed. For primary methodology sources and transfer limits, read references/source-index.md.

Route statistical estimators and power to data-scientist (source reference: ../data-scientist/SKILL.md), storage and pipelines to data-engineering (source reference: ../data-engineering/SKILL.md), model training and leakage checks to ml-engineering (source reference: ../ml-engineering/SKILL.md), and labeling-interface interaction design to product-design-and-ux (source reference: ../product-design-and-ux/SKILL.md).

When not to use

Do not use this skill as the primary owner for statistical inference or experimental design, data platforms, model training, or UI implementation. Those neighboring skills consume this skill's annotation manifest and quality evidence.

Templates

• annotation guide — use when recording this stage of the annotation workflow.
• annotation plan — use when recording this stage of the annotation workflow.
• quality report — use when recording this stage of the annotation workflow.

After this

When you've handed it back, suggest what they could do next in one short question. It's a suggestion: if they're done, that's fine. Offer these, and say they can type their own:

• The Dataset profiler.
• The Tabular dataset explorer. If they pick one and that skill is added here, carry straight on with what you just made. If not, give the link and say what to bring.