Lanta LLM Hosting
Built for a hackathon. My team needed open models and had no GPU host, so I served them from the LANTA supercomputer and gave everyone one OpenAI-compatible API and a browser chat.
- Built
- June 2026
- Role
- Built and ran it end to end
- Serves
- One model at a time, swappable without touching clients
- Stack
vLLM · LiteLLM · Slurm · Docker Compose · FastAPI · Grafana
- vLLM on Slurm (LANTA)
- SSH tunnel
- LiteLLM gateway
- OpenWebUI
One model name that never changes
Clients and chat users always ask for the same model, active-lanta-model. Behind it I can swap the model vLLM is serving on LANTA without anyone changing their settings.
| Layer | Value |
|---|---|
| Client model name | active-lanta-model |
| LiteLLM backend | openai/<served-model-name> |
| vLLM served model | The selected LANTA preset |
How it works
- vLLM serves the chosen Hugging Face model as a Slurm job on a LANTA GPU node.
- A watchdog keeps an SSH tunnel open from the host machine and follows the job when Slurm moves it to another compute node.
- LiteLLM exposes the stable model name with virtual keys, budgets and usage records stored in PostgreSQL.
- Open WebUI gives the team accounts and a browser chat on top of the gateway.
- Prometheus, Grafana and a custom exporter track request volume, token usage, latency and errors, and a small FastAPI dashboard shows whether every service is reachable.
Models it can serve
| Model | Context |
|---|---|
| Qwen3.6-27B | 131,072 |
| Qwen3.6-35B-A3B | 131,072 |
| Qwen3-Coder-30B-A3B-Instruct | 32,768 |
| Qwen2.5-Coder-32B-Instruct | 32,768 |
| DeepSeek-Coder-V2-Lite-Instruct | 32,768 |
Only one preset runs at a time. Submitting a new one cancels the model job that is running.
Running it
- Watchdog scripts check the Slurm serving job and resubmit it if it stops.
- PowerShell scripts start, check and stop the tunnel and verify the whole platform's health.
- Runbooks cover setup, key management and model swaps.
- Automated tests cover the dashboard, the exporter, the default-model policy and the watchdog scripts.
