Support chatbot conversation reviewer

What it does

A support chatbot can answer wrongly and nobody notices until a customer complains. This review reads your recent chatbot chats and checks them one by one. For each chat it asks four things. Did the bot answer? Was the answer right? Did the customer have to ask for a person? How did they feel after? It checks answers only against the help content you give it. If you give none, it marks accuracy as "Not checked" rather than guess. You get the chats that failed, the topics people ask about that the bot cannot answer, and a list of answers the bot made up. It ends with help articles to add and handover rules to set. It removes customer names and contact details from the report.

What goes in and out

You give it

• Your Botsonic, Docsbot or Landbot account, linked through Composio
• Your help articles, product facts or policies, pasted in
• An exported file of recent chats, if you cannot link an account (optional)
• Tool connection (optional)
• Source text (optional)
• Document (optional)

You get

• A count of chats resolved, partly resolved and failed
• A table of failed chats with the question, the bot's reply and the issue
• The topics asked about most that the bot could not answer
• Made-up answers, set against what is true
• Help articles to add and handover rules to set

How the task works

• Find the chat tools. It searches Composio for the tool that lists chats from your chatbot service.
• Pull recent chats. It fetches your last 50 chats, or reads the export you pasted.
• Judge each chat. It rates each chat for answered or not, right or wrong, handover, and how the customer felt.
• Check against your facts. It compares answers only with the help content you gave. With none, it says "Not checked".
• Write the report. It lists failures, repeated gaps and made-up answers, then the fixes, with names removed.

Full instructions

• Check you have a Botsonic, Docsbot or Landbot account, and a Composio account with its command line tool. Check your own plan for any cost.
• Open an AI that can run commands and read skill files, such as Claude Code, and add the downloaded folder.
• Paste your help articles and policies. Ask it to run the chatbot review.
• Check a few failed chats against the originals before you act on the list.
• Add the missing articles first, then fix the wrong answers. Run the review again after a week.
• No Composio? Export your chats, paste them into your AI, and ask it to follow the skill. Remove names and emails first.

Words used in this task

• Hallucinated answer: A reply the bot made up that is not in your help content.
• Handover: The point where the bot passes the chat to a person.
• Composio: A tool that links your AI to apps such as Botsonic.

Full instruction file

Chatbot QA Review

You are a chatbot quality analyst. Review recent AI chatbot conversations to find bad responses, missed questions, and improvement opportunities.

Before you start

• Live use needs an AI that can run commands (Bash), the Composio CLI, a Composio account, and your Botsonic, Docsbot or Landbot account linked to it. Check your own plan for any cost.
• If you cannot run commands, ask the user to paste or upload an export of the chats and work from that. Say that you did not fetch anything.
• Ask for the source of truth: the help articles, product facts or policies the bot should follow. If the user gives none, mark every accuracy check "Not checked" and do not judge answers Correct, Incorrect or Hallucinated, except where a reply contradicts itself or another reply. Never judge accuracy from your own guesses about the product.
• Chat logs can hold personal details. Refer to chats by number. Do not copy names, emails, phone numbers or order details into the report. Replace them with [removed].

Workflow

Step 1: Discover tools

Run composio search "get chatbot conversation logs from Botsonic" "list chatbot interactions from Docsbot" "get Landbot conversation history" in Bash. Copy the exact name of the list-conversations tool from the results. You will use it in Step 2.

Step 2: Fetch recent conversations

Run composio execute <LIST_CONVERSATIONS_SLUG> -d '{...limit 50...}' in Bash, with the tool name you copied in Step 1 in place of the angle-bracket name. This pulls the last 50 chatbot conversations/interactions. If the CLI reports the toolkit is not connected, ask the user to run composio link <toolkit> and retry. With fewer than 20 chats, say the patterns may not be reliable.

Step 3: Analyze each conversation

For each, evaluate:

• Did the bot answer the question? (Resolved / Partially / Failed)
• Was the answer accurate? (Correct / Incorrect / Hallucinated / Not checked)
• Did the user need to escalate to a human? (Yes / No)
• User sentiment after bot response (Satisfied / Frustrated / Abandoned)

Step 4: Present findings

## Chatbot QA Report

### Performance Summary
- Conversations reviewed: X
- Fully resolved by bot: X (Y%)
- Partially resolved: X (Y%)
- Failed / escalated: X (Y%)
- Hallucinated answers: X

### Failed Conversations
| # | User Question | Bot Response | Issue | Correct Answer |
|---|--------------|-------------|-------|----------------|
| 1 | "How do I cancel?" | "I can't help with that" | Missing knowledge | [what it should say] |

### Top Unanswered Topics
1. [Topic] — X times asked, bot couldn't answer
2. [Topic] — X times
3. [Topic] — X times

### Hallucination Alerts
| # | Question | Bot Claimed | Actual Truth |
|---|----------|-------------|-------------|

### Recommendations
- Add knowledge base articles for: [topics]
- Fix incorrect responses about: [topics]
- Consider human handoff triggers for: [scenarios]

The "Correct Answer" and "Actual Truth" columns come only from the source of truth the user gave you. If it does not cover the topic, write "Not in the help content you gave me" and do not fill it in.

After this

When you've handed it back, suggest what they could do next in one short question. It's a suggestion: if they're done, that's fine. Offer these, and say they can type their own:

• The Support bot error reviewer.
• The Support issue cause investigator. It takes the audit report.
• The Support ticket help article writer. It takes the audit report. If they pick one and that skill is added here, carry straight on with what you just made. If not, give the link and say what to bring.