Hire an AI experiment plan and results reviewer

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• The one change, the one rate and what would prove the idea wrong, written before the run
• A readiness score out of 100 with the sample size and weeks needed
• A verdict with both versions, the gap, the confidence range and a ship call
• A second opinion on luck and whether it agrees
• What the test did not test
• A lesson written as a rule when the test was run badly

How it does the job well

• Start from the user problem and the evidence for it, then the solution.
• Write decisions down with the reason and the options you rejected.
• Keep scope small enough to ship and learn. Name what is out of scope.
• Define how success will be measured before building.

Mistakes it avoids

• Peeking before the read date.
• Calling a gap inside the confidence range a win.
• Running a test that changes two things because both seemed good.
• Shipping a side effect you did not look for.
• Treating opinions or one loud request as proven demand.
• Specs that skip edge cases, error states or dependencies.

What a good result looks like

An example, not your task: Your team runs A/B or split tests and the readouts depend on who reads them.

• The terms. The one change, the one rate, the hoped-for move and the result that would prove the idea wrong.
• readiness.md. Sample size, weeks needed, a readiness score out of 100 and a go or send back call.
• readout.md. Split check, both versions, the gap with its confidence range, side effects and a ship call.
• Second opinion. A second read of the same counts on luck, and whether it agrees with the verdict.
• Not tested. What this test cannot tell you, in plain words.
• lesson.md. The cause and a one-line rule, written only when the test was run badly.

How the work is checked

• The test changed one thing and named one rate before it ran, or it was sent back.
• No result was read before its date without a written yes.
• The verdict shows both versions, the gap and the confidence range.
• The two readers were compared and the difference, if any, is named.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

A second opinion

Before it hands back, your AI reviews the draft the way Isaac Newton and Demis Hassabis would. This is a way of reviewing the work. It does not mean they wrote or endorse it.

Capabilities

• Define the terms. Writes the one change, the one rate, the hoped-for move and what would prove the idea wrong, or sends the plan back.
• Check the plan. Confirms one change and a prediction written before the run.
• Check the traffic. Takes the checked plan and your baseline numbers, works out the sample size and weeks needed and scores readiness out of 100. Under the bar, the test goes back.
• Check the date. Refuses to read a result before its date unless you say so in writing, and then writes the lesson.
• Read the result. The split check, the result on the named rate with a confidence range, side effects and a ship call.
• Get a second opinion on luck. Takes the same end-of-test counts, not the first reader's report, and reads them a second way, names any disagreement and prefers the cautious call.
• Write the lesson when earned. For a peeked, stopped or muddled test, uses the calls and lessons from the two readings to find the root cause and write a rule for next time.
• Hand back. Updates the register and stops. You ship or revert.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id experiment-readout-desk only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/experiment-readout-desk/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• The test plan, with what changes, for whom, the rate to move, by how much and the read date
• Visits, conversions and revenue for the pages in the test, for the last few weeks
• The results per version when the test ends, by day if you have them
• What else happened during the test, with dates (advised)
• The test register, once it exists

Helpful context (optional)

• Release notes and campaign dates during the test
• The last three tests the team ran, so the register starts with history

Tools and access

• Claude Code: It can keep one folder per test and the register between tests.
• Your testing tool: Export results per version by day, with the rate named in the plan and the main business number.
• Any good chat AI: Listing factors and mapping results is text work any strong chat AI does well with this skill.
• A spreadsheet: Keeps the held-fixed list and the results side by side for each run.
• Any good chat AI: The sums are simple, and the main chat AIs all show their working when asked.
• Google Sheets: Keeps each month's numbers in one place, so you can check every sum and compare with your own past.
• Evan Miller's sample size calculator: Free, and a quick second check on the visitors per version.
• A spreadsheet: Holds the counts and gives the audience and reply sums.
• Your email tool or ad platform: Exports counts by version key.
• Any good chat AI: Structuring decisions and writing plans is text work any strong chat AI does well with this skill.
• A notes app or document: Gives you one place to keep rules and the dates to check them.

Functions and deliverables

• Experiment plan and results reviewer: Check a test before it runs. Read the results against its plan, flag weak evidence and save useful lessons.
  • The one change, the one rate and what would prove the idea wrong, written before the run
  • A readiness score out of 100 with the sample size and weeks needed
  • A verdict with both versions, the gap, the confidence range and a ship call
  • A second opinion on luck and whether it agrees
  • What the test did not test
  • A lesson written as a rule when the test was run badly
• One-change experiment planner: Plan a one-change experiment. Tell it what you want to learn and get a test plan that changes one thing and says what each result means.
  • The question rewritten in plain terms, with each term defined
  • The one change to test and the list of things held fixed
  • A second check, the no-effect result and what each outcome means
• A/B test traffic checker: Check if your site has enough visitors for an A/B test. Paste visits, orders and revenue to get the weeks needed and a readiness score out of 100.
  • A numbers table with conversion rate, average order value and revenue per visitor, each sum shown
  • A short read of what moved and what the traffic mix explains
  • The test sizing written out, with visitors and weeks needed
  • A verdict (can test, test a bolder change, or ship and watch) and three next actions
  • A test readiness score out of 100, showing how much of the needed sample your traffic reaches in four weeks
• A/B test results analyst: Share your A/B test data. Get a split check, results with confidence ranges and a ship call.
  • A traffic split check
  • Results with 95 percent ranges
  • Guard metric checks
  • A ship, stop or extend call
  • A written test report
  Source: Nimrod Fisher.
• A/B test results reader: Read your split-run test results: see how much better one version is, whether luck explains it, and get a keep, rerun or stop call.
  • A table of both versions and the gap as a percentage
  • A luck check with the sum shown
  • Sales per hundred replies and cost per reply and per sale
  • A keep, rerun or stop call with its reason
  • A record of what the test taught, and the next test to run
• Lessons learned principle writer: Turn a mistake into a rule you can check. Give it what happened and your goal. You get the root cause, a rule and a check date.
  • A list of the problems without softening
  • A root cause about a choice or habit
  • A rule with the situation it applies to
  • Two past cases tested, a critic named and a check date
• Product context file writer: Share product facts, audience and positioning. Get a reviewed context file that other marketing skills can reference.
  • A product marketing context file
  • A summary of what the file already holds
  • A change log of updates
  • Gaps you still need to fill
  Source: Corey Haines.
• Isaac Newton: Untangle a pattern or test a claim through Isaac Newton’s recorded approach. Define terms, isolate a change and keep findings separate from guesses.
  • Your problem worked on the way Isaac Newton would: their questions, their method, their push-back
  • The answer or the work, in Isaac Newton's voice, using their ideas by name
  • Anything that goes beyond their public record, marked
  • One suggested next step
• Demis Hassabis: Plan an AI research test using Demis Hassabis’s recorded approach. Choose the root problem, define a score and learn in a small test world.
  • Your problem worked on the way Demis Hassabis would: their questions, their method, their push-back
  • The answer or the work, in Demis Hassabis's voice, using their ideas by name
  • Anything that goes beyond their public record, marked
  • One suggested next step

Tasks you can use it for

• Your team runs A/B or split tests and the readouts depend on who reads them.
• A test shipped last quarter and nobody can say now whether it worked.
• You want to know, before you build a test, whether you have the traffic to read it.

Limits to check before starting

• Your site has too little traffic to read a test: talk to ten customers or run a simple survey instead, and use the readiness score to confirm.
• You want ideas for what to test: use the Product funnel steps and experiment review, which proposes the experiments this desk reads.
• You want a test started, stopped or shipped for you: do that in your own testing tool, because this agent never does it.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Start a chat with the agent. In Claude Code, add it as a skill or save it in .claude/agents, then ask it to check a test plan.
• Paste the plan and the traffic numbers. Read the readiness score. Fix the plan or drop the test if it says so.
• Start the test in your own tool. Tell the agent the start date.
• On the read date, export the results per version and give them to the agent. Read the verdict.
• Ship or revert yourself. The register keeps the team's hit rate over the year.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/experiment-readout-desk for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.