AI agent tester
Describe your AI agent. Get a test set, scoring rules and a release decision memo you can trust.
AI agent tester is a free AI skill for the job "Test and fix code". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Burhan Sebin · Level: advanced · Thinking: high
Made by Burhan Sebin. See the original on GitHub; we copied it as it is, with its licence.
What it does
A test set, scoring rules and a release memo for your AI agent, built so a decision-maker can trust the result. It avoids cherry-picked demos. First it states what is being tested and what decision the test supports. Then it builds cases from real use, plus odd ones: bad input, long-tail requests, broken tools and attacks. A holdout set stays hidden while you tweak prompts. Each check uses the plainest fair judge: exact checks for facts, a rubric-driven AI judge for quality, and blind human review for risky or disputed cases. Everything is frozen and run against a baseline, repeated where results vary. Failures are read case by case before any average is trusted.
You give it
- What the agent is for and who uses it
- Outcomes that must never happen
- The current version to compare against
- Examples from real use (optional)
- Codebase (optional)
You get
- An evaluation brief
- A test set with notes on where cases came from
- Scoring rules and pass marks
- Results with a comparison to the baseline
- A decision memo for the release owner
Use it when
- You must decide if an agent is ready to ship and want evidence.
- You want to compare two prompts, models or tool set-ups fairly.
- A production failure should become a test that never goes away.
- You need a repeatable gate every time the agent changes.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.