Hire an AI app quality reviewer

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• Evaluation criteria and cases
• Feedback loop design
• Quality findings and next experiment

How it does the job well

• Clarify the purpose, audience and limits, such as length, format and deadline, first.
• Use the user's materials and voice. Ask for samples when tone matters.
• Be specific and concrete. Cut what does not serve the purpose.
• Say what you are unsure of and where to check it.

Mistakes it avoids

• Generic output that ignores the user's details.
• Inventing facts, quotes or sources.
• Long answers when a short one was asked for.

What a good result looks like

An example, not your task: You plan a change to a Python app that calls an AI model and want proof it helps.

• Evaluation criteria and cases. Test cases, scoring rules and a repeatable pass or fail run, from the AI app evaluation planner.
• Feedback loop design. A rewritten task prompt, a rubric, critique and refine prompts and stop rules, from the Agent answer feedback designer.
• Quality findings. Five scores out of 5, the worst problems ranked, and a fix for each, from the AI workflow quality investigator.
• Next experiment. The one change to try next, with source coverage, unresolved findings and missing facts.

How the work is checked

• The output meets the stated length and format.
• Every fact comes from the user's material or is marked to check.
• Suggested changes explain why.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

A second opinion

Before it hands back, your AI reviews the draft the way Kent Beck would. This is a way of reviewing the work. It does not mean they wrote or endorse it.

Capabilities

• AI app evaluation planner. Share your AI app and goal. Get test cases and a way to compare changes with real results. Hands step 2 the success criteria and cases.
• Agent answer feedback designer. Share an agent task. Uses the success criteria from step 1 as the standard for a good answer. Get checks and feedback loops to improve the quality of its answers.
• AI workflow quality investigator. Reads your own workflow files and uses the test cases and checks from steps 1 and 2 to show where it disappoints. Get its weak points ranked, with causes and practical fixes.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id check-an-ai-app-improvement only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/check-an-ai-app-improvement/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• Your AI app's code (required, step 1 runs it)
• The change you want to make, and what a good answer looks like
• A spending limit for model calls, plus one weak agent task if you have it

Tools and access

• Any good chat AI: Claude, ChatGPT or Gemini can run the check and fix by hand in one chat.
• A second AI as judge: A different model grades more fairly than one marking its own work.

Functions and deliverables

• AI app quality reviewer: Set useful test cases for an AI app change, plan feedback checks and rank the next fixes from real evidence.
  • Evaluation criteria and cases
  • Feedback loop design
  • Quality findings and next experiment
• Python LLM app evaluation planner: Share a Python LLM app and goal. Get test cases and a plan to compare changes against results.
  • Eval criteria: what your app must do well, written as checks you can score
  • Your app code marked so each run records its data and answers
  • One recorded run that proves the setup works
  • Scoring rules and a test set with inputs and the data for each case
  • A pass or fail result and a score for each case
  • An action plan for the weak spots the scores show
  Source: GitHub Copilot community.
• AI agent feedback designer: Make your AI agent check and fix its own answers. Get a pass or fail rubric, a rewritten task prompt and a stop rule for one task.
  • A rewritten task prompt with the success criteria built in
  • A rubric table with weights and a pass line for each check
  • A check prompt and a fix prompt, ready to paste
  • Stop rules for the loop
  • Steps to run it in a chat, or the code pattern for it
  Source: GitHub Copilot community.
• AI workflow quality investigator: Check an AI workflow for weak spots before it goes live. Get five scores, the worst problems ranked, and clear fixes for each.
  • A score from 1 to 5 in five areas, and a total out of 25.
  • Findings for each area, based on what you shared.
  • The three most serious problems, ranked.
  • One specific fix for each serious problem.
  Source: GitHub Copilot community.
• me.md builder: Answer a few questions, or let your AI use what it remembers. Get one private file that tells any AI who you are.
  • One me.md file with up to ten dated sections
  • A list of facts taken from memory, for you to check
  • A short note of what it removed for privacy
  • The two next steps, to save it and set it up
• Kent Beck: Plan a software change using Kent Beck’s recorded approach. List the cases, take small steps and keep code tidying separate from new behaviour.
  • Your problem worked on the way Kent Beck would: their questions, their method, their push-back
  • The answer or the work, in Kent Beck's voice, using their ideas by name
  • Anything that goes beyond their public record, marked
  • One suggested next step

Tasks you can use it for

• You plan a change to a Python app that calls an AI model and want proof it helps.
• You want test cases you can rerun after every change.

Limits to check before starting

• You want to compare an old prompt with a new one on the same test cases. Use the Prompt change tester.
• Your app is not Python code, or your AI cannot run code. Use the AI agent release tester for a test set and a ship or hold memo.
• You want one agent to check and fix its own answers. Use the AI agent feedback designer.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Add the complete skill pack to your AI.
• Share your goal and the files listed above.
• Ask it to follow the steps and mark any missing facts.
• Review the result before approving external actions.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/check-an-ai-app-improvement for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.