Hire an AI data pipeline designer

Everything it can hand back

Your task sets the actual outputs. The specialist picks the methods that cover them.

• A source list with the expected shape of each source and who owns it.
• A choice between batch, changes-only and streaming loads, with the reason.
• A design for the raw, cleaned and business-ready layers.
• A set of quality checks, with what each does when it fails.
• A freshness target for each output and the alerts that guard it.
• A recovery plan for reruns, backfills and source changes, and a short runbook for each pipeline.

How it does the job well

• Give no cost, speed or saving figure you did not get from the user or from a source you name.
• Do not build, run or change a live pipeline. Give a design and let the engineer act.
• Agree the question and the decision it supports before analysing.
• Check the data first: date range, grain, missing values, duplicates and definitions.
• Compare like with like: same period length, definitions and segments.
• Separate what the data shows from what it might mean, and give confidence levels.

Mistakes it avoids

• Reporting a change without checking whether tracking or definitions changed.
• Treating correlation as cause.
• Averages that hide important segments or outliers.

What a good result looks like

An example, not your task: You are moving data from several systems, such as a shop, a CRM and ad accounts, into one warehouse.

• Sources and contracts. Each source with its owner, volume, change rate, expected fields and any private data.
• Load choice. Full, changes-only or streaming for each source, with the reason and a cost note.
• Layer design. What the raw, cleaned and business-ready layers hold, their rules and who may read them.
• Quality checks. Each check, where it runs and what happens when it fails.
• Freshness and alerts. A freshness target for each output, the alert that guards it and its owner.
• Recovery plan. Safe reruns, backfills and the runbook note for each kind of failure.

How the work is checked

• Numbers reconcile to the source totals, and the reconciliation is shown.
• Every chart and number states its source, period and definition.
• Limits and next checks are listed.
• States which checks passed, failed or could not run, and names missing inputs.

These checks have not been run against a model yet. Your AI uses them to check its own work.

A second opinion

Before it hands back, your AI reviews the draft the way Margaret Hamilton would. This is a way of reviewing the work. It does not mean they wrote or endorse it.

Capabilities

• Profile the sources. It lists each source with its volume, how often it changes, what is missing or empty, and who owns it.
• Set data contracts. It writes the expected fields and types for each source and each output. A change that breaks them raises an alert.
• Choose how to load. It picks full loads, changes-only loads or streaming for each source, weighing cost against freshness.
• Design the layers. Raw data is kept unchanged. Cleaned data is deduplicated and consistent. Business-ready data is shaped for reports and never read from the raw layer.
• Add quality checks. It adds checks for empty fields, duplicates, value ranges, links between tables and freshness, and says what stops a load.
• Plan monitoring and alerts. It sets alerts for failed loads, late data, odd row counts and changed schemas, and an owner for each alert.
• Plan recovery. It makes every step safe to rerun, plans backfills, and writes a runbook of what breaks and how to fix it.

How the role works

1. Checks your task, context and available tools.
2. Picks the methods needed and prepares a work plan.
3. Carries out supported tasks, with each method's limits and review gates.
4. Checks the real deliverables and corrects supported faults.
5. Hands back files, evidence and unresolved gaps.

Your specialist's working setup

Download the base specialist without a task form or account. README.md and SETUP.md explain how to use the folder. SOUL.md defines its scope. INTAKE.md asks missing questions inside your AI. PLAYBOOK.md, SKILLS.md, BEST_PRACTICES.md, CHECKS.md and HANDOFF.md guide the work and its checked deliverables.

Complete associated skills, supporting files and licences are included. Choose only the methods needed for the actual task. Keep context, working files and accepted feedback in your AI project.

Use this specialist through MCP or the API

Call prepare_specialist with role_id engineering-data-engineer only. No task is required to load it. Your AI asks what job you need done, what you want back and which relevant context it can use.

For task-led hiring, call find_specialist, explain the suggested specialists and let the user accept or change the team.

Complete ZIP: Download base specialist. API: GET /api/v1/specialists/engineering-data-engineer/pack. The host AI supplies its own model, tools and execution; a pack does not connect accounts or run a task.

Inputs and helpful context

The specialist starts from the task brief. Only active methods require their inputs.

For the primary method, when used

• Your data sources, such as apps, databases, files or event feeds, and roughly how much data each holds.
• How often each source changes and whether it can tell you what changed.
• Who uses the data and the questions they need answered.
• How fresh the data must be, such as daily or within 15 minutes.
• Where the data will live and the tools you use now (optional).
• Anything private in the data, such as customer emails or card details.

Helpful context (optional)

• A small sample of each source, with private details removed.
• The list of reports that depend on the data.
• Your last incident, if a pipeline has failed before.

Tools and access

• Claude, ChatGPT or Gemini with file upload: You can attach a sample of each source with private details removed, and the design will match the real fields.
• Your warehouse or lakehouse docs: Tool names in the profile are examples, and your platform's own limits shape the design.

Functions and deliverables

• Data pipeline designer: Get a data pipeline design with raw, cleaned and report-ready layers, quality checks, freshness targets, alerts and a plan for when loads fail.
  • A source list with the expected shape of each source and who owns it.
  • A choice between batch, changes-only and streaming loads, with the reason.
  • A design for the raw, cleaned and business-ready layers.
  • A set of quality checks, with what each does when it fails.
  • A freshness target for each output and the alerts that guard it.
  • A recovery plan for reruns, backfills and source changes, and a short runbook for each pipeline.
  Source: Michael Sitarzewski and Agency Agents contributors.
• Data pipeline builder: Describe where your data starts and who uses it. Get a data pipeline design, load code, quality checks and a run schedule.
  • A written pipeline design with the reasons for each choice
  • Extract and load code for each source
  • A reporting data model, such as a star schema
  • Data quality checks for gaps, repeats and late records
  • A schedule and fixes for common faults
  Source: Alireza Rezvani.
• Data extraction pipeline builder: Turn a website, file, or API source and field list into extraction code with validation checks.
  • Extraction script
  • Clean CSV or JSON file
  • Validation result
  • Cost and quota estimate
  Source: Alireza Rezvani.
• Agent role loader builder: Plan a safer loader for your AI agent's role files, with checked fields, trimming that keeps the rules, clear failures and tests. Draft code only.
  • A list of gaps in the loader, each with a proposed fix
  • A validation spec for each role file field and section
  • A trim order that never cuts the rules
  • Draft code changes and the tests to write
  • A list of what is still missing before you can trust it
• Metric definition writer: Share a metric, calculation and business use. Get a YAML definition, validation results and questions for the data owner.
  • A filled metric, dimension or entity YAML
  • A validation report
  • dbt-ready field names
  • Notes on edge cases
  Source: Nimrod Fisher.
• Margaret Hamilton: Review a system failure plan using Margaret Hamilton’s recorded approach. Check handoffs, priorities, warnings and what happens when people make mistakes.
  • Your problem worked on the way Margaret Hamilton would: their questions, their method, their push-back
  • The answer or the work, in Margaret Hamilton's voice, using their ideas by name
  • Anything that goes beyond their public record, marked
  • One suggested next step

Tasks you can use it for

• You are moving data from several systems, such as a shop, a CRM and ad accounts, into one warehouse.
• A report is wrong or late and nobody knows which step broke.
• You must choose between a nightly batch and a near-live feed.
• You want a standard for quality checks and alerts before more teams depend on the data.

Limits to check before starting

• You want working pipeline code for a smaller job. Use the Data pipeline builder.
• You need dashboards or charts. Use the Data visualization engineer agent.
• You are reviewing a business question or a SQL query on a warehouse. Use the Data warehouse query reviewer.
• One query is slow. Use the Database performance analyst.
• You have a single messy file to clean once. Ask your AI to clean it directly. A full pipeline design is more than you need.

Best way to use this role

Give it one clear outcome, the actual files and a definition of done. Keep the work in one AI project so it can reuse checked context and earlier files.

• Download the pack from this page and add it to your AI as a skill. Or paste SKILL.md into a new chat, then paste references/source-agent.md after it.
• List your sources and describe each in a line. Add the questions your team asks of the data and how fresh answers must be.
• Say "Design the pipeline" for a new setup, or "Review this pipeline" if you paste an existing design or code.
• Your AI asks once for anything missing. Unknown volumes stay marked as assumptions.
• Read the load choice and the quality checks first. Make sure each check matches a real way your data goes wrong.
• Hand the design to the engineer who will build it. Code in it is an example to adapt, and it needs testing on your data.

Read it as data

Read the complete role setup as JSON. Connect your AI to the AI Blocks MCP library and read aiblocks://roles/engineering-data-engineer for its role instructions and pinned sources.

This is a prepared AI role. No task has run. Your AI carries out authorized work using its available tools and reports the actual deliverables and checks.