Skip to content

Lanta LLM Hosting

Built for a hackathon. My team needed open models and had no GPU host, so I served them from the LANTA supercomputer and gave everyone one OpenAI-compatible API and a browser chat.

Built
June 2026
Role
Built and ran it end to end
Serves
One model at a time, swappable without touching clients
Stack

vLLM · LiteLLM · Slurm · Docker Compose · FastAPI · Grafana

  1. vLLM on Slurm (LANTA)
  2. SSH tunnel
  3. LiteLLM gateway
  4. OpenWebUI

One model name that never changes

Clients and chat users always ask for the same model, active-lanta-model. Behind it I can swap the model vLLM is serving on LANTA without anyone changing their settings.

How the stable name maps to the backend
LayerValue
Client model nameactive-lanta-model
LiteLLM backendopenai/<served-model-name>
vLLM served modelThe selected LANTA preset

How it works

  • vLLM serves the chosen Hugging Face model as a Slurm job on a LANTA GPU node.
  • A watchdog keeps an SSH tunnel open from the host machine and follows the job when Slurm moves it to another compute node.
  • LiteLLM exposes the stable model name with virtual keys, budgets and usage records stored in PostgreSQL.
  • Open WebUI gives the team accounts and a browser chat on top of the gateway.
  • Prometheus, Grafana and a custom exporter track request volume, token usage, latency and errors, and a small FastAPI dashboard shows whether every service is reachable.

Models it can serve

Presets, one served at a time
ModelContext
Qwen3.6-27B131,072
Qwen3.6-35B-A3B131,072
Qwen3-Coder-30B-A3B-Instruct32,768
Qwen2.5-Coder-32B-Instruct32,768
DeepSeek-Coder-V2-Lite-Instruct32,768

Only one preset runs at a time. Submitting a new one cancels the model job that is running.

Running it

  • Watchdog scripts check the Slurm serving job and resubmit it if it stops.
  • PowerShell scripts start, check and stop the tunnel and verify the whole platform's health.
  • Runbooks cover setup, key management and model swaps.
  • Automated tests cover the dashboard, the exporter, the default-model policy and the watchdog scripts.

All projects