How Caktus Runs and Uses Local Models

As a research project this year at Caktus, we’ve been experimenting with serving open-weight language models on our own hardware. In this post, we’ll walk through the hardware, the Docker Compose stack that serves our models, the Gemma and Qwen configurations we tested, our benchmarks, and the configurations we use in pi.dev and VS Code.

In conversations at DjangoCon US 2026, we found that other developers are either running local models for client work or curious about how to do it. We hadn’t seen much written about a working setup end to end, so we’re sharing what we’ve done so far in the hopes it’s helpful. This is the latest in our ongoing series documenting what we’ve learned working with LLMs.

Why we did this

When a coding agent works on a codebase, some or all of that code may be sent to a cloud provider. We build data-intensive Django applications, so keeping client code and data under our own control matters to us (and some clients rule out third-party cloud AI entirely). Running the models ourselves fits how we prefer to work.

Building a workstation with two high-end GPUs, plus the time to configure, benchmark, and maintain the stack, adds overhead, so this isn’t the right fit for everyone. For our consulting practice, four factors made it worthwhile:

1. Client code and data stay on our network. The models are reachable only over our Tailscale private network, with no public endpoints. Agents running with unrestricted permissions never run where they have access to client code, data, or production infrastructure; those sessions are confined to isolated VMs.
2. Prototyping client-facing local AI features. Some clients need a model embedded directly in their own application for sensitive-data workloads. Dedicated in-house hardware let us prototype, benchmark, and validate the architecture under strict privacy constraints before deploying to their infrastructure.
3. Less dependence on third-party policies, rate limits, or pricing changes. The models are open-weight checkpoints we host ourselves under their respective licenses; we update them on our own schedule.
4. The models are fast enough. At 200–250 tokens per second decode, the experience is close enough to a hosted model’s that we’ve stopped thinking about it.

The hardware

Everything runs on one machine we call geforce, reachable from our laptops and CI over Tailscale. We bought it in May 2026; the full parts list is on PCPartPicker:

The two GPUs are each dedicated to one model:

We rate both cards down from their default power draw, which keeps the machine quiet, cool, and within its power envelope. The benchmarks below were run at those limits.

The two models run on different physical GPUs, not two containers sharing one card. Each Docker Compose service reserves its GPU explicitly via device_ids, which makes the setup deterministic.

The two models

Both models are Qwen checkpoints in the 262,144-token context class, served as OpenAI-compatible APIs by vLLM.

The dense FP8 model is the higher-fidelity option: official Qwen weights, vision support, and room for many concurrent agent sessions. The MoE activates only 3B parameters per token, so on a 32 GB card it decodes at more than double the dense model’s speed. We use the MoE for everyday work and the dense model for planning, multi-step reasoning, and long sessions.

Two community resources did a lot of the heavy lifting here:

• noonghunna/club-3090. Community recipes for serving LLMs on RTX 3090/4090/5090 cards, and where we first saw that these Qwen checkpoints run well on consumer Blackwell hardware.
• froggeric/Qwen-Fixed-Chat-Templates. Fixed Jinja chat templates for the Qwen 3.5/3.6/3.8 family. We pin specific revisions per model; the current revision defaults reasoning effort to medium and fixes rendering bugs in the official templates.

We also ran the Gemma 4 family on this same machine in June and July: the 31B temporarily took the 96 GB card’s slot from the Qwen dense model for testing. What we found is in the next section.

The Gemma 4 config we tested

We ran gemma-4-31B-it in FP8 block quantization on the RTX PRO 6000. The full model definition, rendered from our Ansible templates, is in the Serving section below.

It served cleanly: the 31B ran the full 262,144-token context with tool-calling enabled, pinned to the official Google chat template at a revision that includes Google’s 2026-07-09 template fix.

Two things kept us from serving it day-to-day:

1. Decode speed without MTP. The benchmarks below put the 31B at about 43 tokens per second on the same card and power limit — roughly 44% of the Qwen dense model’s 98 t/s. That’s fine for long-running, batch-style work, but the waits add up in an interactive coding session.
2. MTP tool-calling bug. Gemma 4 ships a ~0.5B multi-token-prediction drafter (the -assistant checkpoint), which the club-3090 recipes rate at roughly 2× throughput at low batch. But on the vLLM releases of the time, enabling MTP broke streaming tool calls — the upstream issue is vllm-project/vllm#39043, and a follow-up streaming multi-tool-call patch (#42006) was closed unmerged. Club-3090 A/B-verified it on an RTX 5090: with MTP off, tool calls stream clean; with MTP on, they leak <pad> and <|tool_call> tokens. Since a coding agent depends on clean tool calls, we ran Gemma with MTP off, which is also why we never got the drafter’s speed benefit.

Gemma 4 is a viable alternative for this kind of setup — on the 96 GB card in particular — and it’s on our re-test list. Since #39043 is now fixed, we can turn the MTP drafter back on, and we plan to re-test soon.

Serving: vLLM in Docker Compose

We serve both models with vLLM, pinned at v0.32.0, using the official vllm/vllm-openai image. We pin deliberately and re-benchmark on every bump. Most of our flag choices came from community recipes and a lot of re-benchmarking, and some may be wrong.

Ansible automates the deployment. Model definitions live in one group_vars file, and Jinja2 templates render a single compose file for both models. Here is the rendered output (Docker Compose GPU support handles the per-GPU reservation):

# ~/vllm-containers/docker-compose.yml, rendered from our Ansible templates
services:
  # Dense 27B FP8 on the RTX PRO 6000 (96 GB), host GPU 0
  vllm-qwen3.8-27b-fp8:
    image: "vllm/vllm-openai:v0.32.0"
    container_name: "vllm-qwen3.8-27b-fp8"
    restart: unless-stopped
    shm_size: "16g"
    ports:
      - "8014:8000"
    volumes:
      - "/home/vllm/models:/home/vllm/models:ro"
    env_file:
      - qwen3.8-27b-fp8.env
    # $$ stops Compose from expanding VLLM_ARGS; bash + eval re-parses the
    # single-quoted words inside the container.
    entrypoint: ["/bin/bash", "-c"]
    command: ["eval vllm serve $$VLLM_ARGS"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["0"]
              capabilities: [gpu]

  # MoE 35B-A3B NVFP4 on the RTX 5090 (32 GB), host GPU 1
  vllm-qwen3.6-35b-a3b-nvfp4:
    image: "vllm/vllm-openai:v0.32.0"
    container_name: "vllm-qwen3.6-35b-a3b-nvfp4"
    restart: unless-stopped
    shm_size: "16g"
    ports:
      - "8012:8000"
    volumes:
      - "/home/vllm/models:/home/vllm/models:ro"
    env_file:
      - qwen3.6-35b-a3b-nvfp4.env
    entrypoint: ["/bin/bash", "-c"]
    command: ["eval vllm serve $$VLLM_ARGS"]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ["1"]
              capabilities: [gpu]

Model weights are pre-downloaded (a Hugging Face token is only needed for that initial download, not for serving) and bind-mounted read-only. Each service gets an .env with its full vllm serve command line:

# qwen3.8-27b-fp8.env, dense 27B FP8 on the RTX PRO 6000
MODEL_ID=qwen3.8-27b-fp8
MODEL_PORT=8014
VLLM_IMAGE=vllm/vllm-openai:v0.32.0
GPU_DEVICE=0
MODELS_DIR=/home/vllm/models
CUDA_DEVICE_ORDER="PCI_BUS_ID"
CUDA_VISIBLE_DEVICES="0"
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS="1"
VLLM_WORKER_MULTIPROC_METHOD="spawn"
VLLM_USE_FLASHINFER_SAMPLER="1"
VLLM_NO_USAGE_STATS="1"
PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"
OMP_NUM_THREADS="1"
VLLM_ARGS="/home/vllm/models/qwen3.8-27b-fp8 --served-model-name 'Qwen/Qwen3.8-27B-FP8' --dtype 'auto' --tensor-parallel-size '1' --pipeline-parallel-size '1' --gpu-memory-utilization '0.9298' --max-num-seqs '16' --max-model-len '262144' --host '0.0.0.0' --port '8000' --enable-auto-tool-choice --tool-call-parser 'qwen3_xml' --override-generation-config '{\"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"repetition_penalty\": 1.0, \"max_tokens\": 131072}' --speculative-config '{\"method\":\"mtp\",\"num_speculative_tokens\":3}' --kv-cache-dtype 'fp8' --reasoning-parser 'qwen3' --chat-template '/home/vllm/models/.chat-templates/qwen3.8-27b-fp8-src/4ea21db90694e60d002500dae85ebff26e4b23ad/chat_template.jinja' --enable-prefix-caching --max-num-batched-tokens '16384' --limit-mm-per-prompt '{\"image\":2,\"audio\":0}' --attention-backend 'flashinfer' --default-chat-template-kwargs '{\"preserve_thinking\": true}' --enable-prompt-tokens-details --trust-remote-code --generation-config 'vllm'"

# qwen3.6-35b-a3b-nvfp4.env, MoE 35B-A3B NVFP4 on the RTX 5090
MODEL_ID=qwen3.6-35b-a3b-nvfp4
MODEL_PORT=8012
VLLM_IMAGE=vllm/vllm-openai:v0.32.0
GPU_DEVICE=1
MODELS_DIR=/home/vllm/models
CUDA_DEVICE_ORDER="PCI_BUS_ID"
CUDA_VISIBLE_DEVICES="0"  # index inside the container; the host GPU is GPU_DEVICE
VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS="1"
VLLM_WORKER_MULTIPROC_METHOD="spawn"
NCCL_CUMEM_ENABLE="0"
NCCL_P2P_DISABLE="1"
VLLM_NO_USAGE_STATS="1"
VLLM_USE_FLASHINFER_SAMPLER="1"
PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"
OMP_NUM_THREADS="1"
VLLM_ARGS="/home/vllm/models/qwen3.6-35b-a3b-nvfp4 --served-model-name 'unsloth/Qwen3.6-35B-A3B-NVFP4' --dtype 'auto' --tensor-parallel-size '1' --pipeline-parallel-size '1' --gpu-memory-utilization '0.93' --max-num-seqs '1' --max-model-len '262144' --host '0.0.0.0' --port '8000' --enable-auto-tool-choice --tool-call-parser 'qwen3_coder' --override-generation-config '{\"temperature\": 0.6, \"top_p\": 0.95, \"top_k\": 20, \"min_p\": 0.0, \"presence_penalty\": 0.0, \"repetition_penalty\": 1.0}' --kv-cache-dtype 'fp8_e4m3' --reasoning-parser 'qwen3' --chat-template '/home/vllm/models/.chat-templates/qwen3.6-35b-a3b-nvfp4-src/b225c1d63f84c65af7dd6af2ba384eb037aa885e/chat_template.jinja' --enable-prefix-caching --enable-chunked-prefill --max-num-batched-tokens '8192' --limit-mm-per-prompt '{\"image\":0,\"audio\":0}' --trust-remote-code"

The Gemma 4 31B definition from the July test rendered the same way:

# gemma-4-31b-it-fp8-block.env, dense 31B FP8 block on the RTX PRO 6000 (July test)
MODEL_ID=gemma-4-31b-it-fp8-block
MODEL_PORT=8015
GPU_DEVICE=0
CUDA_DEVICE_ORDER="PCI_BUS_ID"
CUDA_VISIBLE_DEVICES="0"
VLLM_WORKER_MULTIPROC_METHOD="spawn"
NCCL_CUMEM_ENABLE="0"
NCCL_P2P_DISABLE="1"
VLLM_NO_USAGE_STATS="1"
VLLM_USE_FLASHINFER_SAMPLER="1"
VLLM_USE_DEEP_GEMM="0"
VLLM_ALLOW_LONG_MAX_MODEL_LEN="1"
PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True,max_split_size_mb:512"
OMP_NUM_THREADS="1"
VLLM_ARGS="'RedHatAI/gemma-4-31B-it-FP8-block' --served-model-name 'RedHatAI/gemma-4-31B-it-FP8-block' --dtype 'auto' --tensor-parallel-size '1' --pipeline-parallel-size '1' --gpu-memory-utilization '0.92' --max-num-seqs '1' --max-model-len '262144' --host '0.0.0.0' --port '8000' --enable-auto-tool-choice --tool-call-parser 'gemma4' --override-generation-config '{\"temperature\": 1.0, \"top_p\": 0.95, \"top_k\": 64, \"min_p\": 0.0, \"repetition_penalty\": 1.0}' --reasoning-parser 'gemma4' --chat-template '/home/vllm/models/.chat-templates/gemma-4-31b-it-fp8-block-src/842da3794eaa0b77d5f08bae87a17459d91ff475/chat_template.jinja' --enable-prefix-caching --enable-chunked-prefill --limit-mm-per-prompt '{\"image\":0,\"audio\":0}' --trust-remote-code"

One caveat on that one: the July test ran before our September move to Docker, when vLLM served from a systemd unit on the then-current release instead of the pinned image, and the model argument pointed at the Hugging Face repo name so the weights downloaded on first start. The model-level flags are what we actually ran.

A few flags are worth pointing out:

• Quantization is auto-detected. None of these entries passes --quantization; vLLM reads FP8 and compressed-tensors NVFP4 straight from the model configs.
• MTP speculative decoding (--speculative-config '{"method":"mtp","num_speculative_tokens":3}') uses each Qwen model’s own multi-token-prediction module as its draft model. On the dense model it buys roughly 15% more tokens per second; we’ve left it off the MoE for now, since community reports on mixed 5090 hardware are inconsistent.
• FP8 KV cache (--kv-cache-dtype fp8) saves about half the VRAM versus bfloat16, which is what lets the 96 GB card hold the full 262k context and 16 concurrent sequences.
• Do not force the Marlin backend on these checkpoints. Per the Unsloth NVFP4 docs it is roughly 2× slower. vLLM picks the right native backend on its own.
• max_num_seqs: 1 on the Qwen MoE is deliberate. It’s a single-user card; queuing one long agent session beats interleaving several slow ones.
• Chunked prefill is off on the Qwen dense model. We hit Xid 31 memory faults under long-context concurrent load, a known issue from the older vLLM engine, and the 96 GB card has enough KV headroom that we don’t need it anyway. We’ll re-test on future vLLM pins.
• The Gemma entry is the same shape with two differences: VLLM_ALLOW_LONG_MAX_MODEL_LEN=1, which silences vLLM’s long-context safety check at 262,144 tokens, and VLLM_USE_DEEP_GEMM=0, carried over from the club-3090 recipe for this checkpoint. Its override-generation-config also follows Google’s recommended sampling parameters (temperature 1.0, top-p 0.95, top-k 64).

Benchmarks

We benchmark with llama-benchy at several context depths. The numbers below are from 2026-10-30 on vLLM v0.32.0, at the power limits:

The MoE decodes roughly 2.5× faster at every depth we tested, and prefills about 15× faster than the dense model. Its decode speed barely degrades as context grows. We think this is partly because the linear-attention layers keep the KV cache small. For interactive coding, that decode speed is fast enough that the model rarely gets in the way.

For comparison, here’s what the Gemma 4 31B produced on the same card and power limit during our July tests (2026-07-20, same llama-benchy depths, MTP off):

That decode speed is the main reason we left it out of daily service: it works, but it’s about 44% of the Qwen dense model’s pace, and the difference is very noticeable turn after turn.

Note: these figures are for our specific checkpoints, vLLM pin, and power limits, measured while we were still tuning the stack. Treat them as reference points, not promises, and re-run your own benchmarks before buying hardware on the strength of them.

Using the models

Both endpoints speak the OpenAI chat-completions API, so any client that accepts a base URL works. These are the two we use daily.

pi.dev

pi.dev reads local model definitions from ~/.pi/agent/models.json. Ours declares one provider per endpoint, with compatibility flags and thinking-level mappings for each model (pi.dev docs on models, custom providers):

{
  "providers": {
    "geforce-8014": {
      "baseUrl": "http://geforce:8014/v1",
      "api": "openai-completions",
      "apiKey": "local",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false,
        "thinkingFormat": "chat-template",
        "chatTemplateKwargs": {
          "enable_thinking": { "$var": "thinking.enabled" },
          "preserve_thinking": true,
          "reasoning_effort": { "$var": "thinking.effort", "omitWhenOff": true }
        }
      },
      "models": [
        {
          "id": "Qwen/Qwen3.8-27B-FP8",
          "name": "Qwen3.8-27B FP8 (official), 262k, fp8 KV, vision, tools",
          "reasoning": true,
          "input": ["text", "image"],
          "contextWindow": 262144,
          "maxTokens": 131072,
          "thinkingLevelMap": {
            "minimal": "low",
            "low": "low",
            "medium": "medium",
            "high": "xhigh",
            "xhigh": "xhigh"
          },
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    },
    "geforce-8012": {
      "baseUrl": "http://geforce:8012/v1",
      "api": "openai-completions",
      "apiKey": "local",
      "compat": {
        "supportsDeveloperRole": false,
        "supportsReasoningEffort": false
      },
      "models": [
        {
          "id": "unsloth/Qwen3.6-35B-A3B-NVFP4",
          "name": "Qwen3.6-35B-A3B NVFP4 (Unsloth), 262k, fp8_e4m3 KV, text-only, tools",
          "reasoning": true,
          "input": ["text"],
          "contextWindow": 262144,
          "maxTokens": 32768,
          "cost": { "input": 0, "output": 0, "cacheRead": 0, "cacheWrite": 0 }
        }
      ]
    }
  }
}

geforce here is the Tailscale hostname of the GPU box. On the box itself you’d use http://localhost:8014/v1 and http://localhost:8012/v1. The "apiKey": "local" value is a placeholder. vLLM doesn’t check it, but the API expects a header.

Then ~/.pi/agent/settings.json picks the default model and routes our fast subagents to the faster MoE:

{
  "defaultProvider": "geforce-8014",
  "defaultModel": "Qwen/Qwen3.8-27B-FP8",
  "defaultThinkingLevel": "medium",
  "subagents": {
    "agentOverrides": {
      "scout": { "model": "geforce-8012/unsloth/Qwen3.6-35B-A3B-NVFP4" },
      "delegate": { "model": "geforce-8012/unsloth/Qwen3.6-35B-A3B-NVFP4" }
    }
  }
}

We run the dense model at a lowered thinking level. Qwen 3.8 defaults to xhigh reasoning effort, and it overthinks at that setting. It can burn tens of thousands of reasoning tokens on a routine question before answering. So defaultThinkingLevel is set to medium for day-to-day work. We bump it up for the harder planning sessions, where the extra thinking pays for itself.

VS Code

VS Code’s built-in Custom Endpoint provider is enough. No third-party extension needed. The file is ~/.config/Code/User/chatLanguageModels.json (Linux):

[
  {
    "name": "GeForce",
    "vendor": "customendpoint",
    "apiKey": "sk-no-key-required",
    "apiType": "chat-completions",
    "models": [
      {
        "id": "Qwen/Qwen3.8-27B-FP8",
        "name": "Qwen3.8-27B FP8 (official), 262k, fp8 KV, vision, tools",
        "url": "http://geforce:8014/v1",
        "toolCalling": true,
        "vision": true,
        "maxInputTokens": 262144,
        "maxOutputTokens": 131072,
        "thinking": true,
        "supportsReasoningEffort": ["low", "medium", "high"],
        "reasoningEffortFormat": "chat-completions"
      },
      {
        "id": "unsloth/Qwen3.6-35B-A3B-NVFP4",
        "name": "Qwen3.6-35B-A3B NVFP4 (Unsloth), 262k, fp8_e4m3 KV, text-only, tools",
        "url": "http://geforce:8012/v1",
        "toolCalling": true,
        "vision": false,
        "maxInputTokens": 262144,
        "maxOutputTokens": 32768
      }
    ]
  }
]

For our small trusted team, direct endpoints work well enough. Nobody needs an API token, and we track usage with Grafana dashboards built on top of the monitoring stack on geforce (more on those in a future post). If we ever need to issue API tokens or meter usage per person, LiteLLM is the obvious next stop. We haven’t put it in front of these models yet, but it’s the first thing we’d add as the team grows.

Where this leaves us

The main payoff for us is that client code stays on our network. A model reads, analyzes, and edits a Django codebase, and the farthest that code travels is a Docker container on our own tailnet.

The rest is still in progress. We’ve already hit cases where a hosted model does the job better, and our configuration choices (the pinned vLLM, the community chat templates, the per-model concurrency) are best guesses we expect to keep revising as we learn more.

If your team is exploring local AI deployments, looking to establish secure developer workflows, or needs experienced Python and Django consulting, get in touch with us.