AI Blocks › Build and ship software
Language model benchmark runner
Give it a model. Get scores on 60 or more standard tests, so you can compare it with others.
Language model benchmark runner is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: advanced · Thinking: low
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
Scores for your model on the same tests everyone else uses, so you can compare it with other models fairly. Point the skill at a model, such as one from Hugging Face, and pick tasks from the 60 or more in EleutherAI's open test suite. These include MMLU for knowledge, GSM8K for maths and HumanEval for code. It writes the command, sets batch size and device, runs it and gathers the results into one table. Because the prompts and metrics are fixed, a score from your run lines up with published ones. You need Python, the lm-eval package and a GPU for larger models.
You give it
- The model name or path
- Which benchmarks to run
- Access to a GPU machine
- A past results file to compare with (optional)
You get
- The exact run command
- Scores per benchmark
- A results table
- Notes on how to compare across models
Use it when
- You fine-tuned a model and want to know if it got better or worse.
- You are choosing between two open models for a project.
- You want to compare your model with published leaderboard numbers.
- You need a repeatable evaluation to run after every training change.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run Python code on a computer with a GPU.