AI Blocks › Build and ship software
LangSmith AI app monitor
Give it your AI app. Get traces, test sets and scores so you can find bad answers before users do.
LangSmith AI app monitor is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: advanced
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A way to see what your AI app really did, so bad answers show up in a report before a user complains. Add LangSmith tracing to your app and every model call is recorded with its input, output, delay and token cost. You can then turn good and bad traces into a test set. Run that set whenever you change a prompt or model, and score the answers with built-in or custom checkers. A drop in score means you broke something. In production, dashboards track errors, speed and spend. It works with OpenAI, Anthropic, LangChain and LlamaIndex. You need Python, a LangSmith account and an API key.
You give it
- Your AI app code
- A LangSmith account and key
- Example questions and good answers (optional)
- What counts as a good answer
You get
- Tracing set up in your code
- A test set from real traces
- Scoring checks you can rerun
- Monitoring for errors, speed and cost
Use it when
- Your chatbot gave a wrong answer and you cannot see which step failed.
- You changed a prompt and want proof it did not make things worse.
- You want to track cost and response time of every model call.
- You want a test set built from real user questions.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code and use a LangSmith account (paid beyond the free plan).