Hire an AI machine learning model reviewer

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• An audit plan that maps each test to one of ten areas, with what result would worry it
• Findings, each with the observation, the evidence you gave, the impact and a fix, rated high, medium, low or info
• Example Python checks for stability (PSI), ranking power (AUC, Gini, KS), calibration and explanations, to run on your data
• An overall opinion of sound, sound with findings, unsound or not determined
• A list of evidence still missing and the checks not run

How it does the job well

• Every finding rests on evidence the user supplied. If there is none, the area goes under Not tested.
• Do not invent metrics, thresholds, drift numbers or results. Do not claim to have run code or replicated a model.
• Profile numbers (success rates, PSI bands, replication tolerance) are examples and goals, not standards you can quote as law. Label them as a guide.
• This is a review aid. It does not replace a qualified data scientist, model risk team or regulator-required validation. Say so in the opinion.
• Do not change, deploy or retire a model. Hand over the report for the user to act on.
• Read the existing code, its conventions and its tests before changing anything. Match the project's patterns.
• Make the smallest change that fully solves the request. Keep unrelated refactors out.

Mistakes it avoids

• Claiming code works without running it or its tests.
• Inventing APIs, package versions or settings instead of checking the installed ones.
• Changing behaviour outside the request, such as reformatting whole files or renaming public interfaces.

What a good result looks like

An example, not your task: A churn, credit, demand or ranking model is about to go live, and someone other than its builder should check it.

• Model and scope. What is under review, its use and label, and the evidence still missing.
• Audit plan. Ten areas, each with tests, the result that would worry it and what you supplied.
• Findings. Each finding with severity, observation, evidence, impact and fix.
• Checks to run. Example code for stability, ranking power, calibration and explanations, with what to paste back.
• Opinion. Sound, sound with findings, unsound or not determined, and the reasons.
• Not tested. Areas it could not test and what each needs.

How the work is checked

• The changed files build, and the relevant tests and linters pass. The commands and results are shown.
• Each change is explained in one line with its reason, and risky areas are named.
• Any check that could not run is reported as NOT_RUN with what it needs.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

Capabilities

• Scope the audit. It lists what is being reviewed, the documents it needs, and a test plan across ten areas.
• Check data and labels. It plans checks on the population, exclusions, label definition and whether future information leaks into training.
• Check features and stability. It plans per-feature drift checks, including the Population Stability Index, and checks that the features make sense.
• Rebuild and compare. It plans a replication from the documented method, and a simple challenger model, to compare scores with the original.
• Test calibration and performance. It plans checks that scores match real rates, and that ranking power holds across splits, groups and time.
• Explain and rate. It plans explanation and fairness checks, then rates each finding by severity and gives an opinion backed by evidence.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id specialized-model-qa only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/specialized-model-qa/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• What the model predicts, how it is used, and its type, such as a churn classifier
• How the target label is defined, and the time windows used for training and testing
• The data it uses, as a field list with summary numbers, or a sample (remove private details)
• The results you have, such as accuracy, AUC, calibration or the decision threshold (optional)
• Any documents, monitoring reports or earlier reviews (optional)

Helpful context (optional)

• The model's own documentation and approval records.
• Scores and outcomes for a holdout period, kept apart from training data.
• A list of the groups that matter for fairness, such as age band or region, where the law allows you to hold them.

Tools and access

• Python with pandas, scikit-learn and SHAP: The example checks use these free tools to compute drift, ranking power, calibration and feature effects.
• Claude Code or another AI that can run code on your data: It can run the checks itself and read the results. In a plain chat you run them and paste back the numbers.

Functions and deliverables

• Machine learning model reviewer: Audit a machine learning model you did not build. Get a test plan, rated findings and an opinion, backed only by the evidence you supply.
  • An audit plan that maps each test to one of ten areas, with what result would worry it
  • Findings, each with the observation, the evidence you gave, the impact and a fix, rated high, medium, low or info
  • Example Python checks for stability (PSI), ranking power (AUC, Gini, KS), calibration and explanations, to run on your data
  • An overall opinion of sound, sound with findings, unsound or not determined
  • A list of evidence still missing and the checks not run
  Source: Michael Sitarzewski and Agency Agents contributors.
• Machine learning pipeline builder: Share your data, model code and acceptance rules. Build a versioned pipeline with checks, promotion gates and recovery steps.
  • A stage-by-stage pipeline graph
  • Pipeline code in your existing tools
  • Input and output schemas for each stage
  • Pass or fail rules for release
  • Retry, backfill and rollback steps
  Source: Burhan Sebin.
• Machine learning research paper writer: Share research code, results and the target AI conference. Get a paper draft with source checks and clear notes on gaps.
  • A full paper draft in the conference style
  • Citations fetched from search, not memory. It cannot confirm them without web search, so check every one yourself
  • A list of unverified references
  • Feedback on the weak sections
  Source: Orchestra Research.
• Security audit evidence reviewer: Check a security audit before you trust it. Find findings with no proof, dropped findings and missing coverage, then get a pass or rework verdict.
  • A verdict of pass, rework or deliver with a flagged gap
  • A table that marks each finding as verified, unsupported or wrong location
  • A list of findings the report dropped or softened, with whether the reason holds
  • A list of areas the audit did not cover
  • A 1 to 10 score in each quality category
  • A short list of fixes before you share the report
  Source: GitHub Copilot community.
• AI project context organiser: Share your project and next task. Get a short rules file, a focused file list and a clear handoff for later sessions.
  • A ready rules file for your project
  • A list of which files to load for each task
  • Rules for when to reset or trim the session
  • A checklist for spotting drifting answers
  Source: Addy Osmani.

Tasks you can use it for

• A churn, credit, demand or ranking model is about to go live, and someone other than its builder should check it.
• A model that worked last year now gives odd scores, and you want to find out if the data shifted.
• A buyer, auditor or manager asks for proof that your model is calibrated and fair.
• You inherited a model with thin notes and want a list of what to test and in what order.

Limits to check before starting

• You want a model built or tuned. Use the AI feature planner.
• You want a final sign-off for a regulated use such as credit or hiring. A qualified data scientist or your model risk team must give that.
• You want to check an AI chat prompt or an LLM, not a scoring model. Use the Prompt engineer for prompts.
• You have no results, no data and no documents. Expect a plan and a list of what to collect, not an opinion.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Download the pack from this page and add it to your AI as a skill. Or paste SKILL.md into a new chat, then paste references/source-agent.md after it.
• Describe the model, its target label, its data and the results you have. Paste summary numbers, not private records. Say how the output is used and what happens if it is wrong.
• Read the audit plan first. Remove tests you cannot run, and add any rule your company already has.
• Run the example checks on your own data, in Python or your team's tools. Paste the results back, one area at a time.
• Ask for the findings table and the opinion. Check that each finding points to evidence you gave.
• Have a qualified data scientist or your model risk team review the report before you rely on it or share it.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/specialized-model-qa for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.