--- title: "Support chatbot conversation reviewer" type: "skill" url: "https://useaiblocks.com/skills/support-chatbot-conversation-review" updated: "2026-11-03" first_published: "2026-10-25" job: "Help your customers" tested: true licence: "MIT" source: "https://github.com/composio-community/support-skills/tree/83382c4b055fc4e5b055044adb5bfb7585484c9a/chatbot-review" instructions: "https://useaiblocks.com/skills/support-chatbot-conversation-review/SKILL.md" --- # Support chatbot conversation reviewer > Find where your support chatbot fails. Review recent chats to get missed questions, wrong answers and help articles to add or fix. Link your Botsonic, Docsbot or Landbot account and give it your help content. It reads your last 50 chatbot chats and checks if each reply answered the question. You get failed chats, repeated gaps and fixes to make. Free skill for the job "Help your customers" · Level: intermediate · Thinking: medium · Safety checked · Updated 2026-11-03 ## Links - Page: https://useaiblocks.com/skills/support-chatbot-conversation-review - Instructions (SKILL.md): https://useaiblocks.com/skills/support-chatbot-conversation-review/SKILL.md - All files (zip): https://useaiblocks.com/skills/support-chatbot-conversation-review.zip - JSON version: https://useaiblocks.com/skills/support-chatbot-conversation-review.json - In your AI: connect https://useaiblocks.com/mcp and ask "Use AI Blocks to get support-chatbot-conversation-review" ## What it does A support chatbot can answer wrongly and nobody notices until a customer complains. This review reads your recent chatbot chats and checks them one by one. For each chat it asks four things. Did the bot answer? Was the answer right? Did the customer have to ask for a person? How did they feel after? It checks answers only against the help content you give it. If you give none, it marks accuracy as "Not checked" rather than guess. You get the chats that failed, the topics people ask about that the bot cannot answer, and a list of answers the bot made up. It ends with help articles to add and handover rules to set. It removes customer names and contact details from the report. ## What you give it - Your Botsonic, Docsbot or Landbot account, linked through Composio - Your help articles, product facts or policies, pasted in - An exported file of recent chats, if you cannot link an account (optional) ## What you get - A count of chats resolved, partly resolved and failed - A table of failed chats with the question, the bot's reply and the issue - The topics asked about most that the bot could not answer - Made-up answers, set against what is true - Help articles to add and handover rules to set ## Example: what goes in and what comes out ### What goes in - Your chatbot account, linked through Composio, or an exported file of chats - Your help articles, product facts or policies - How many chats to review, 50 by default (optional) ### Before you start - An AI that can run commands, with Composio CLI installed, or a pasted export - A Botsonic, Docsbot or Landbot account - Chats may hold personal details, so check this suits your privacy rules ### What comes out - **Performance Summary**: How many chats were reviewed, resolved, partly resolved or failed, and how many answers were made up. - **Failed Conversations**: A table of each failed chat with the question, the bot's reply, the issue and the right answer. - **Top Unanswered Topics**: The topics people asked about most that the bot could not answer. - **Hallucination Alerts**: Answers the bot made up, set against what your help content says. - **Recommendations**: Help articles to add, wrong answers to fix and handover rules to set. ## Use it when - Customers keep saying your chatbot is no help, and you want to see why. - You changed your help content and want to check how the bot handles it. - You want to know which questions the bot cannot answer, so you can add articles. - You suspect the bot is making up policies and want to find examples. ## Skip it when - You are building a bot. Use the Chatbot flow planner. - You want to check one written claim against your facts. Use the Unsupported claim spotter. - You have no help content or product facts to check against. Gather them first, or the accuracy check will say "Not checked". - You want a report on survey scores or comments. Use the Customer survey feedback analyst. ## How to use it 1. Check you have a Botsonic, Docsbot or Landbot account, and a Composio account with its command line tool. Check your own plan for any cost. 2. Open an AI that can run commands and read skill files, such as Claude Code, and add the downloaded folder. 3. Paste your help articles and policies. Ask it to run the chatbot review. 4. Check a few failed chats against the originals before you act on the list. 5. Add the missing articles first, then fix the wrong answers. Run the review again after a week. 6. No Composio? Export your chats, paste them into your AI, and ask it to follow the skill. Remove names and emails first. ## What to do next - [Support bot error reviewer](https://useaiblocks.com/skills/review-support-bot-errors-with-retest-cases): Support bot error reviewer - [Support issue cause investigator](https://useaiblocks.com/skills/support-issue-cause-investigator): Support issue cause investigator - [Support ticket help article writer](https://useaiblocks.com/skills/solved-ticket-help-page-writer): Support ticket help article writer ## Sources and credits - From Composio: https://github.com/composio-community/support-skills/tree/83382c4b055fc4e5b055044adb5bfb7585484c9a/chatbot-review, shared under the MIT licence (original name: chatbot-review). - [chatbot-review skill](https://github.com/composio-community/support-skills/tree/83382c4b055fc4e5b055044adb5bfb7585484c9a/chatbot-review), Composio: Copied under the MIT licence. We gave it a plain name. We added Before you start and After this sections that link to other AI Blocks steps. ## Instructions (SKILL.md) ## Chatbot QA Review You are a chatbot quality analyst. Review recent AI chatbot conversations to find bad responses, missed questions, and improvement opportunities. ### Before you start - Live use needs an AI that can run commands (Bash), the Composio CLI, a Composio account, and your Botsonic, Docsbot or Landbot account linked to it. Check your own plan for any cost. - If you cannot run commands, ask the user to paste or upload an export of the chats and work from that. Say that you did not fetch anything. - Ask for the source of truth: the help articles, product facts or policies the bot should follow. If the user gives none, mark every accuracy check "Not checked" and do not judge answers Correct, Incorrect or Hallucinated, except where a reply contradicts itself or another reply. Never judge accuracy from your own guesses about the product. - Chat logs can hold personal details. Refer to chats by number. Do not copy names, emails, phone numbers or order details into the report. Replace them with [removed]. ### Workflow #### Step 1: Discover tools Run `composio search "get chatbot conversation logs from Botsonic" "list chatbot interactions from Docsbot" "get Landbot conversation history"` in Bash. Copy the exact name of the list-conversations tool from the results. You will use it in Step 2. #### Step 2: Fetch recent conversations Run `composio execute -d '{...limit 50...}'` in Bash, with the tool name you copied in Step 1 in place of the angle-bracket name. This pulls the last 50 chatbot conversations/interactions. If the CLI reports the toolkit is not connected, ask the user to run `composio link ` and retry. With fewer than 20 chats, say the patterns may not be reliable. #### Step 3: Analyze each conversation For each, evaluate: - **Did the bot answer the question?** (Resolved / Partially / Failed) - **Was the answer accurate?** (Correct / Incorrect / Hallucinated / Not checked) - **Did the user need to escalate to a human?** (Yes / No) - **User sentiment after bot response** (Satisfied / Frustrated / Abandoned) #### Step 4: Present findings ``` ## Chatbot QA Report ### Performance Summary - Conversations reviewed: X - Fully resolved by bot: X (Y%) - Partially resolved: X (Y%) - Failed / escalated: X (Y%) - Hallucinated answers: X ### Failed Conversations | # | User Question | Bot Response | Issue | Correct Answer | |---|--------------|-------------|-------|----------------| | 1 | "How do I cancel?" | "I can't help with that" | Missing knowledge | [what it should say] | ### Top Unanswered Topics 1. [Topic] — X times asked, bot couldn't answer 2. [Topic] — X times 3. [Topic] — X times ### Hallucination Alerts | # | Question | Bot Claimed | Actual Truth | |---|----------|-------------|-------------| ### Recommendations - Add knowledge base articles for: [topics] - Fix incorrect responses about: [topics] - Consider human handoff triggers for: [scenarios] ``` The "Correct Answer" and "Actual Truth" columns come only from the source of truth the user gave you. If it does not cover the topic, write "Not in the help content you gave me" and do not fill it in. ### After this When you've handed it back, suggest what they could do next in one short question. It's a suggestion: if they're done, that's fine. Offer these, and say they can type their own: - The [Support bot error reviewer](https://useaiblocks.com/skills/review-support-bot-errors-with-retest-cases). - The [Support issue cause investigator](https://useaiblocks.com/skills/support-issue-cause-investigator). It takes the audit report. - The [Support ticket help article writer](https://useaiblocks.com/skills/solved-ticket-help-page-writer). It takes the audit report. If they pick one and that skill is added here, carry straight on with what you just made. If not, give the link and say what to bring.