AI Blocks › Build and ship software
Data pipeline builder
Describe your data sources and goal. Get a working pipeline with checks, scheduling and fixes for common faults.
Data pipeline builder is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Alireza Rezvani · Level: advanced
Made by Alireza Rezvani. See the original on GitHub; we copied it as it is, with its licence.
What it does
A plan and working code for moving data from where it starts to where people use it, with checks along the way. You also get advice on choices such as batch versus streaming. You describe your sources, volume and goal. The skill designs the flow using tools like Python, SQL, Spark, Airflow, dbt and Kafka, and picks between ETL and ELT. It builds the data model, such as a star schema, sets up scheduling, and adds data quality checks. It deals with late arriving records and weighs designs like Lambda against Kappa. A troubleshooting section covers common faults such as stalled jobs and duplicates. Expect to run code on your own systems and need access to your sources.
You give it
- Data sources and their format
- Where the data should end up
- How fresh it must be
- Tools you already use (optional)
You get
- Pipeline design
- Working extraction and load code
- Data quality checks
- Scheduling setup and fixes
Use it when
- You need to load data from an app and a spreadsheet into one warehouse.
- You are deciding between batch and streaming for a new feature.
- A nightly job keeps failing and you need a fix.
- You want a star schema for reporting.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run code and read your project files.