Sometimes a Software Upgrade Is Enough
We’ve been doing machine learning for about a decade. Like a lot of teams from that era, we still have some legacy GPU inventory sitting around. Instead of sending our old Tesla K80s to the landfill, we gave them a second life doing in-house LLM inference.
The result has been surprisingly useful. The hardware is already fully capitalized and lives inside infrastructure that is already in the operational budget. That means the incremental cost of this LLM capacity is effectively zero. For a company that values “always fail safe” and careful capital allocation, that is a meaningful advantage.
We published the full, reproducible build recipe here:
The Tesla K80 is a 2014-era dual-GPU board (Kepler, compute capability 3.7). Each half of the card gives you roughly 11.3–11.4 GiB of usable VRAM. It is not a modern accelerator. It has no Tensor Cores, limited memory bandwidth by today’s standards, and modern CUDA toolkits have dropped support for it.
Yet it remains highly effective for a specific class of work: lights-out, batch-oriented, high-quality inference where the answer can take tens of seconds or a few minutes. Overnight summarization, asynchronous agent queues, offline RAG pipelines, bulk classification, document processing, and similar jobs sit squarely in this category.
In our lab we run a mix of models on a Dell PowerEdge R730 populated with two K80s (four logical devices). Current measured performance with Q4_K_M GGUF models and full GPU offload looks like this:
These numbers are steady and predictable. Generation speed is essentially flat across short-to-medium output lengths; the limiting factor is memory bandwidth, not compute. Multi-GPU layer splitting helps fit larger models and improves long-context prefill, but it does not meaningfully raise generation throughput on this architecture. We therefore prefer independent processes—one llama-server per GPU—rather than aggressive model splitting.
Kepler architecture support (including the K80 / CC 3.7) was removed from CUDA 12.0 in December 2022, and modern toolkits and drivers, including out-of-the-box llama.cpp builds, no longer support it. Getting full CUDA acceleration required three specific configurations:
-DCMAKE_CUDA_ARCHITECTURES=37 at configure time.Once those pieces are in place, the rest of llama.cpp works normally: full layer offload, Flash Attention, OpenAI-compatible llama-server, and multi-model fleets. We pinned a specific llama.cpp tag for reproducibility and documented every step, including the exact CMake flags, driver/CUDA install sequence, and runtime recommendations.
The repository also contains lab notes from the R730 deployment—power draw under load (typically 85–95 W per active GPU half), temperature behavior, VRAM footprints for the models we actually use, and guidance on context length versus concurrency. On this hardware we generally keep context in the 8k–16k range for serving and reserve very long contexts for single-slot batch jobs.
We treat the K80s as a reliable, zero-marginal-cost backend for work that does not need to be interactive:
Anything latency-sensitive simply goes to Grok (or another modern API). The combination is pragmatic: high-quality local capacity for the long tail of work, and a fast external service for the interactive path. The local fleet never becomes a bottleneck because it was never intended to be the primary interactive path.
The cards cost almost nothing on the secondary market ($50–75 each as of mid-2026). More importantly for us, they already sit in powered, cooled, monitored racks. Power draw under load is modest by modern standards, and idle cards drop into low-power states. There is no CapEx, no new rack space, and no new operational process to invent.
This approach aligns with how we think about infrastructure in general. We prefer systems that fail safe, remain understandable years later, and extract residual value from assets we already own. A carefully engineered software path on decade-old hardware can still deliver real production value when the workload profile matches the hardware’s strengths.
If you have a closet (or a rack) full of old datacenter cards and have been wondering whether they can still earn their keep, the recipe is public and deliberately conservative. It is not the fastest path, and it is not intended for real-time chat (although it's surprisingly good at that too). But for batch and offline work it is quiet, predictable, and free once the cards are already on the floor.
Sometimes the highest-leverage move is not buying the newest hardware. Sometimes it is simply teaching the hardware you already own a new trick.
Happy Computing!
Jesse
Pages