AI Blocks › Build and ship software
vLLM model server setup
Name your model and GPU. Get a fast model server running, with settings for speed and memory.
vLLM model server setup is a free AI skill for the job "Build and ship software". Give the instructions to an AI assistant that supports the required files and tools, then review its output.
Free · No account · MIT licence · Made by Orchestra Research · Level: expert · Thinking: high
Made by Orchestra Research. See the original on GitHub; we copied it as it is, with its licence.
What it does
A running model server that answers requests fast, with settings picked to fit your GPU and model size. It speaks the same API as OpenAI, so your existing code points at it with a one-line change. The skill starts from a quick install and a test call. It then sets memory use, context length and port for your model, and for large models it splits work across several GPUs. It covers shrinking models with quantization to fit smaller cards, testing with light traffic, turning on monitoring and checking speed numbers before go-live. It needs Python, a supported GPU and the vLLM package. You download the model yourself.
You give it
- The model you want to run
- Your GPU type and how many you have
- Expected traffic or response time target
- Whether a smaller, quantized model is acceptable (optional)
You get
- Install and launch commands
- Tuned settings for memory and context
- A test call using the OpenAI client
- A launch check list with monitoring
Use it when
- You want to host an open model on your own GPU instead of paying per call.
- Your current setup is slow or runs out of memory under load.
- You need an OpenAI-style endpoint for an existing app.
- A large model must be split across two or more GPUs.
How to use it
- Add it to your AI once: download it and add the folder to Claude, ChatGPT or Codex as a skill.
- Ask for the job in your own words. Your AI opens the skill when it fits.
- It works best when your AI can run code on a machine with an NVIDIA GPU.