Hire an AI experiment tracker

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• A test plan with problem, hypothesis, main number, guardrail number and variants
• The sample size per version, with the arithmetic, and the minimum run time
• Stop rules, a risk note and a rollback plan
• A readout with each version's rate, the gap, a likely range and a go or no-go call
• Next tests to run and a short lesson to save

How it does the job well

• Do not claim a result is significant unless you worked it out from the user's counts. Show the sum. Tell the user to check it in a calculator or sheet.
• Do not predict revenue from a test unless the user gave the figures to do it.
• Privacy and consent rules for testing vary by region. Refer the user to their lawyer or developer.
• Start from the user problem and the evidence for it, then the solution.
• Write decisions down with the reason and the options you rejected.
• Keep scope small enough to ship and learn. Name what is out of scope.
• Define how success will be measured before building.

Mistakes it avoids

• Treating opinions or one loud request as proven demand.
• Specs that skip edge cases, error states or dependencies.
• Roadmaps with dates but no owners or confidence levels.

What a good result looks like

An example, not your task: You want to test a new headline, price, checkout step or email, and need a plan before you start.

• Experiment plan. Problem, hypothesis, main and guardrail numbers, versions and risks in one page.
• Sample size and run time. Visitors needed per version, the sum behind it and the minimum days to run.
• Stop rules and rollback. What ends the test early, what does not, and how to undo the change.
• Experiment readout. Each version's rate, the gap, a likely range and a go or no-go call.
• Learnings and next tests. What you learned and up to three tests to run next.
• Open checks. Tracking and maths to verify before you ship.

How the work is checked

• Every requirement can be tested.
• Assumptions and risks are listed with how each will be checked.
• Dependencies and owners are named.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

A second opinion

Before it hands back, your AI reviews the draft the way John Caples would. This is a way of reviewing the work. It does not mean they wrote or endorse it.

Capabilities

• Frame the bet. It turns your idea into a testable claim, with one main number and one guardrail number.
• Size the test. It works out visitors per version from your current rate and the smallest gain you care about, and shows the sum.
• Set rules. It sets a minimum run time, a no-peeking rule, a stop rule if something breaks and a rollback plan.
• Read the result. It compares the two rates, works out the gap and its likely range, and checks the sample reached the target.
• Make the call. It gives go, no-go or run longer, with the reason and the risk of being wrong.
• Save the lesson. It writes what you learned and what to test next, so the next plan starts smarter.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id project-management-experiment-tracker only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/project-management-experiment-tracker/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• The change you want to test and the page, email or step it affects
• The number that should move, such as checkout rate, and its current value
• How many visitors or sends you get in a week
• The smallest gain worth acting on, such as 10% more checkouts
• Another number that must not get worse, such as refund rate (optional)
• For a readout, the visitors and conversions for each version, and the dates the test ran (optional)

Helpful context (optional)

• Last month's numbers for the page or email, to set the current value.
• Your past test log, so the agent does not repeat old tests.
• The date of any sale or holiday inside the test window.

Tools and access

• Claude, ChatGPT or Gemini: A longer chat can hold the plan, then read the result with the same targets after the test ends.
• Google Sheets or Excel: Use them to double-check the sample size and the gap, which the AI works out by hand.

Functions and deliverables

• Experiment tracker: Plan an A/B test before launch, then read the result. Get sample size, stop rules, a go or stop call and the lesson to keep.
  • A test plan with problem, hypothesis, main number, guardrail number and variants
  • The sample size per version, with the arithmetic, and the minimum run time
  • Stop rules, a risk note and a rollback plan
  • A readout with each version's rate, the gap, a likely range and a go or no-go call
  • Next tests to run and a short lesson to save
  Source: Michael Sitarzewski and Agency Agents contributors.
• Product experiment designer: Plan an A/B test you can trust. Give it your idea and baseline rate. You get a hypothesis, sample size, stopping rules and a way to read the result.
  • A written hypothesis
  • Primary and guardrail metrics
  • A sample size and test duration
  • ICE ranking of test ideas
  • A plain reading of the result
  Source: Alireza Rezvani.
• A/B test results reader: Read your split-run test results: see how much better one version is, whether luck explains it, and get a keep, rerun or stop call.
  • A table of both versions and the gap as a percentage
  • A luck check with the sum shown
  • Sales per hundred replies and cost per reply and per sale
  • A keep, rerun or stop call with its reason
  • A record of what the test taught, and the next test to run
• Product test results analyst: Read your product test results and find out if the ad, the page or the checkout is the problem. Get a score out of 100 and the next fix.
  • A call: ad issue, page issue or checkout issue
  • A score out of 100, with what it measured
  • The next fix, with the number that triggered it
  • A note when the test ran too briefly or has too little data
• Product context file writer: Share product facts, audience and positioning. Get a reviewed context file that other marketing skills can reference.
  • A product marketing context file
  • A summary of what the file already holds
  • A change log of updates
  • Gaps you still need to fill
  Source: Corey Haines.
• John Caples: Write headlines and plan a fair ad test through John Caples’s recorded approach. Lead with the reader’s gain and count replies by version.
  • Your problem worked on the way John Caples would: their questions, their method, their push-back
  • The answer or the work, in John Caples's voice, using their ideas by name
  • Anything that goes beyond their public record, marked
  • One suggested next step

Tasks you can use it for

• You want to test a new headline, price, checkout step or email, and need a plan before you start.
• Someone asks "do we have enough traffic to test this?" and you need a number.
• A test has finished and you must decide whether to ship the new version.
• You run several tests and want one place to track what you tried and what you learned.

Limits to check before starting

• You only have finished test numbers and want a quick verdict: use the A/B test results analyst.
• You do not know what to test yet: use the Growth experiment planner.
• You want one run per test, with a lesson saved each time: use the Experiment plan and results reviewer.
• You need a statistician to review a complex study, such as many variants over time. Use the Statistician agent.
• You can only send a small number of visitors, such as a few hundred a month. A test will not give a clear answer. Try customer interviews first.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Add the pack to your AI as a skill. Or open a new chat and paste SKILL.md, then references/source-agent.md.
• To plan, write down the change, the number that should move and its current value, your weekly traffic and the smallest gain worth acting on.
• Check the sample size and run time. If the test needs months, ask for a bolder change or a different number to test.
• Set the test up in your own tool, such as your email platform, a website testing tool or Shopify. Do not look at the result until the planned end date.
• When it ends, paste the visitors and conversions for each version. Say "Write the readout".
• Check the maths in a sheet. Then save the readout and the lesson with your other tests.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/project-management-experiment-tracker for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.