Prompt change tester
Give it a prompt change and a fixed set of cases. Get a clear report of what got better or worse.
Prompt change tester is a free AI skill for the job "Test and fix code". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · CC BY 4.0 licence · Made by sickn33 · Level: advanced
Made by sickn33. See the original on GitHub; we copied it as it is, with its licence.
What it does
A fair test of whether your new prompt is better than the old one. You get case-by-case evidence, so a change ships on proof, not a hunch. You freeze a set of test cases, then run the old and new prompts under the same model settings and budget. The skill records the prompt fingerprints, model and settings used. It decides pass rules in advance: exact checks for structured output, behaviour checks for tools, a clear rubric for open text. A timeout or missing answer counts as incomplete, never a pass. When an AI judges the results, it notes the judge's limits and compares against human labels. The report lists wins and losses per case.
You give it
- Old and new prompts
- A fixed set of test cases
- Rules for what counts as a pass
- Model and settings used
- Human-scored examples (optional)
You get
- Side-by-side results per case
- List of regressions
- Notes on missing or noisy results
- A ship or hold recommendation
Use it when
- You rewrote a prompt and want to know if it got worse anywhere.
- Your team argues over which prompt is better and needs evidence.
- You are changing models and want to compare answers on the same cases.
- You need a record of why you approved a change.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.