Hire an AI search relevance engineer

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• A table that links each bad search to its likely cause and the first check to run
• A plan for the query list and the scores to record before you change anything
• Ordered fixes for missed matches, then for ranking
• A view on whether to add meaning-based search, with the reason
• A safe rollout plan, with a test run and a way to roll back
• Assumptions and the answers you still need from your search server

How it does the job well

• Never tune by anecdote. One stakeholder's pet query is not a relevance strategy. Changes are evaluated against a judgment list sampled from real query logs — head, torso, and tail — or they don't ship.
• Recall before precision. If the right document can't match, no boost will save it. Diagnose with the explain API and zero-results analysis before touching scoring.
• Analyzers are a contract between index time and query time. A stemmer added only at index time, or synonyms only at query time, silently breaks matching. Test both sides with the analyze API on real vocabulary.
• Version indices, alias everything, reindex sideways. Mappings are immutable in the ways that matter. products_v7 behind the products alias, reindex, verify, flip — downtime zero, rollback instant.
• Read the existing code, its conventions and its tests before changing anything. Match the project's patterns.
• Make the smallest change that fully solves the request. Keep unrelated refactors out.
• Reproduce a bug before fixing it, and add a test that fails without the fix.

Mistakes it avoids

• Claiming code works without running it or its tests.
• Inventing APIs, package versions or settings instead of checking the installed ones.
• Changing behaviour outside the request, such as reformatting whole files or renaming public interfaces.

What a good result looks like

An example, not your task: Shoppers or staff search your site and get no results for words that should match.

• Symptom table. Each bad search with its likely cause, the first check to run and a proposed fix.
• Baseline to record. The scores and speed numbers to note before any change, and how to get them.
• Query list plan. How to sample common, middle and rare searches and judge their results.
• Fixes in order. Matching fixes first, then ranking fixes, each with how to score it.
• Hybrid search view. Where meaning-based search helps, where it does not, and how to prove it.
• Rollout and rollback. The test run, the safe index swap and the way back.

How the work is checked

• The changed files build, and the relevant tests and linters pass. The commands and results are shown.
• Each change is explained in one line with its reason, and risky areas are named.
• Any check that could not run is reported as NOT_RUN with what it needs.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

Capabilities

• Sort the symptoms. It groups your bad searches into no results, wrong order, broken product codes and weak long-tail searches, each with a likely cause.
• Plan the baseline. It lists the real queries to sample and the scores to record first, such as nDCG, MRR and the no-results rate.
• Fix matching first. It checks word splitting, stemming, synonyms and typo handling on both the index side and the search side.
• Then fix ranking. It plans field weights, freshness and popularity signals, scored one at a time.
• Decide on hybrid search. It says where meaning-based search adds value, and keeps word matching for exact terms and filters.
• Plan the rollout. It plans a test run, a new index behind an alias so you can roll back, and the dashboards to watch.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id engineering-search-relevance-engineer only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/engineering-search-relevance-engineer/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• Which search tool you use, such as Elasticsearch or OpenSearch, and its version
• 5 to 10 searches that go wrong, with what you expected to see
• Your index mapping and one real search query as JSON (optional)
• An export of your top searches and searches with no results (optional)
• A few sample documents, such as product records (optional)
• Your speed limit, such as how many milliseconds a search may take (optional)

Helpful context (optional)

• A search log export with the query text, result clicks and no-result flags.
• Ratings from a few teammates on what the top 3 results for 20 searches should be.

Tools and access

• Claude, ChatGPT or Gemini: A chat can write the plan. A tool that reads files can also read a long search log export.

Functions and deliverables

• Search relevance engineer agent: Fix poor site search with a tested plan for Elasticsearch or OpenSearch, so each change is scored on real queries before it goes live.
  • A table that links each bad search to its likely cause and the first check to run
  • A plan for the query list and the scores to record before you change anything
  • Ordered fixes for missed matches, then for ranking
  • A view on whether to add meaning-based search, with the reason
  • A safe rollout plan, with a test run and a way to roll back
  • Assumptions and the answers you still need from your search server
  Source: Michael Sitarzewski and Agency Agents contributors.
• Qdrant search relevance investigator: Find why Qdrant search returns wrong or missing results. Get ranked causes, one test for each and the fixes to try in order.
  • A ranked list of likely causes for your case
  • One test per cause, with what each result would mean
  • Fixes in the order to try them
  • A plan to measure quality before and after each change
  Source: GitHub Copilot community.
• Slow query fixer: Share a slow SQL query and plan. Get likely causes, query or index changes, and checks for speed and identical results.
  • The cause of the slowdown
  • A rewritten query
  • Index statements to add
  • A before and after plan comparison
  Source: Burhan Sebin.
• Live code debugger: Share code, bug steps and expected result. With a supported debugger, inspect values, find a likely cause and plan a fix to test.
  • A debug session with chosen breakpoints
  • Live variable values at each stop
  • The root cause
  • A fix suggestion
  Source: sickn33.
• AI project context organiser: Share your project and next task. Get a short rules file, a focused file list and a clear handoff for later sessions.
  • A ready rules file for your project
  • A list of which files to load for each task
  • Rules for when to reset or trim the session
  • A checklist for spotting drifting answers
  Source: Addy Osmani.

Tasks you can use it for

• Shoppers or staff search your site and get no results for words that should match.
• The right product exists but ranks below weaker matches.
• Someone asks "can we add AI search?" and you need to know if it will help.
• You changed synonyms or boosts and need a way to know if search got better or worse.

Limits to check before starting

• You want your pages to rank higher on Google. That is website SEO, not site search. Use the Search Console quick-win finder.
• You want an AI to answer questions from your documents. Use the RAG pipeline implementation planner.
• Your database queries are slow and search is not the cause. Use the Database performance analyst.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Download the pack from this page and add it to your AI as a skill. If you cannot, paste SKILL.md into a new chat, then references/source-agent.md.
• Paste your bad searches, what you expected, and your mapping and sample query if you have them.
• Say "Write the relevance plan. Ask me once if anything is missing."
• Run the first checks it lists on your own search server, such as the analyze and explain tools, and paste the answers back.
• Build the query list and record the baseline scores before you apply any fix.
• Apply one fix at a time and score each. Keep a change only if the score goes up and speed stays inside your limit.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/engineering-search-relevance-engineer for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.