How Caktus Runs and Uses Local Models

Fri, 30 Oct 2026 09:00:00 -0400

As a research project this year at Caktus, we’ve been experimenting with serving open-weight language models on our own hardware. In this post, we’ll walk through the hardware, the Docker Compose stack that serves our models, the Gemma and Qwen configurations we tested, our benchmarks, and the configurations we use in pi.dev and VS Code.

Vllm on Caktus Group

How Caktus Runs and Uses Local Models