Spark job tuner
Give it a slow Spark job. Get changes to partitions, caching and memory that speed it up.
Spark job tuner is a free AI skill for the job "Test and fix code". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · CC BY 4.0 licence · Made by sickn33 · Level: advanced
Made by sickn33. See the original on GitHub; we copied it as it is, with its licence.
What it does
A slow Spark job made faster, with each change explained. You share the job code, its settings and where it spends time. The skill looks at the usual causes in order: how data is split into partitions, shuffles between steps, uneven data that overloads a few tasks, memory pressure, and slow serialisation. It suggests fixes such as adaptive query settings, broadcast joins, salting skewed keys, caching the right data, Kryo serialisation and better file formats. You get advice on executor memory and partition counts, plus ways to confirm the job improved. It assumes you can run Spark code.
You give it
- The Spark job code
- Cluster size and current settings
- Job run times or the stage view (optional)
- Rough data size
You get
- Partition and shuffle changes
- Memory and executor settings
- Fixes for skewed data
- Caching advice
- Ways to check the speed gain
Use it when
- A nightly Spark job takes far too long.
- A few tasks run much longer than the rest.
- Your executors run out of memory or spill to disk.
- You are scaling a pipeline to a much bigger dataset.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run code and read your project files.