AI Blocks › Build and ship software
Big data tool chooser
Describe your large files. Get the right MATLAB tool picked for data too big for memory.
Big data tool chooser is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · BSD 2 Clause licence · Made by MathWorks
Made by MathWorks. See the original on GitHub; we copied it as it is, with its licence.
What it does
One clear MATLAB path for working with files too big for memory, not a menu of options. Describe your data: CSV, Parquet, text or spreadsheet files, how many and how big, and what you want to do with them. The AI runs a pre-flight check and follows a decision flowchart. Continuous data across files points to a datastore with tall arrays. Files to process one at a time point to a datastore with transform. Parquet can be handled per row group. It also covers running reads in parallel, when mapreduce is actually needed, formats with no built-in datastore, and how to write a custom datastore. You need MATLAB.
You give it
- File type, count and total size
- What you want to compute
- Your MATLAB setup and available memory
- Whether files are one dataset or independent
- Codebase (optional)
You get
- One recommended approach with reasons
- Working MATLAB code for it
- Parallel settings if they help
- A custom datastore outline (if needed)
Use it when
- readtable crashes with an out-of-memory error.
- You have a folder of two hundred CSV files to treat as one dataset.
- You want to speed up a loop over many files.
- Your data is in a format MATLAB can't read in chunks.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run MATLAB.