Jul 16, 2026

Self-Hosting LLMs in Kubernetes with vLLM: The Data Residency Path

When to self-host: Data residency obligations, high-volume workloads where per-token API costs compound quickly, or latency budgets that cloud API round-trips can’t meet. Managed APIs win on day one; the math shifts as request volume scales. This stack covers the Kubernetes deployment pattern using vLLM with LINSTOR persistent storage — model weights cache on-cluster after the first download, pod restarts are fast, and the endpoint is OpenAI-compatible so your existing LangChain or LlamaIndex integrations work with a single config change.

The Framework

Three Kubernetes resources do the work: a PersistentVolumeClaim to cache model weights, a Secret holding the Hugging Face token, and a Deployment running vLLM behind a ClusterIP Service. The LINSTOR CSI driver provides thin-provisioned, two-replica block storage — any Kubernetes-compatible block storage works as a substitute.

Pre-requisites

Kubernetes cluster (any distribution)
LINSTOR CSI driver installed or any RWO block StorageClass
Hugging Face account with model access approved and a read-permission API token
kubectl access to the target namespace

Step 1 — Create a PVC for Model Cache

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-model-cache
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: linstor-csi-lvm-thin-r2
  resources:
    requests:
      storage: 50Gi

Fifty gigabytes accommodates most 7B–13B models. Adjust for larger checkpoints. LINSTOR’s thin provisioning keeps actual disk usage aligned to what is written rather than the full allocation.

Step 2 — Store the Hugging Face Token

kubectl create secret generic hf-token \
  --from-literal=token=<YOUR_HF_READ_TOKEN>

The token is mounted into the pod at runtime; it never lands in a ConfigMap or image layer.

Step 3 — Deploy vLLM

apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm
spec:
  replicas: 1
  selector:
    matchLabels: { app: vllm }
  template:
    metadata:
      labels: { app: vllm }
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
          - --model
          - meta-llama/Llama-3.2-1B-Instruct
          - --gpu-memory-utilization
          - "0.80"
        env:
        - name: HF_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        volumeMounts:
        - name: model-cache
          mountPath: /root/.cache/huggingface
      volumes:
      - name: model-cache
        persistentVolumeClaim:
          claimName: vllm-model-cache
---
apiVersion: v1
kind: Service
metadata:
  name: vllm
spec:
  selector: { app: vllm }
  ports:
  - port: 8000
    targetPort: 8000

vLLM exposes an OpenAI-compatible REST API on port 8000. Point any existing client at http://vllm:8000/v1 and the integration works without changes.

Step 4 — Verify the Endpoint

curl http://vllm:8000/v1/models

The response lists the loaded model. Once the weights are cached on the PVC, pod restarts skip the download and come up in seconds.

Step 5 — Swap Your Client Endpoint

# LangChain example — one line changes
llm = OpenAI(base_url="http://vllm:8000/v1", api_key="not-needed")

The Trap

CPU-only inference is demonstration speed, not production speed. The Llama-3.2-1B-Instruct model used in the CNCF reference post was chosen specifically because it runs on CPU. Any real workload needs GPU nodes. Add a node with GPU support, switch to a GPU-capable vLLM image, and update the deployment’s resource requests with nvidia.com/gpu: 1. The OpenAI-compatible endpoint remains identical — only the pod spec and underlying node change. Running the 1B model on CPU is valid for validating the plumbing; it is not a realistic inference benchmark.

Cited References

Source: CNCF Blog

Content created with AI assistance and reviewed for accuracy.