I'm running a 284-billion-parameter model locally, and it competes with cloud AI

Running a large language model on your own hardware has always come with asterisks. You can do it, and both the models and the tooling have got genuinely good, but somewhere in the back of your mind you know you're running with a compromise. Models are smaller, they're usually slower, and even when they're competitive, they're not going to have the same broad level of knowledge as a cloud model from the likes of Anthropic or OpenAI.

DeepSeek V4 Flash is a model that you can run locally with the right hardware, despite being a 284-billion-parameter model. I've been running it on a Lenovo ThinkStation PGX, the little GB10 box I ran a 200-billion-parameter model on not long ago, and it's capable of doing work that most would seem to think would only be possible with the cloud. The engine making it possible isn't Ollama or llama.cpp, though. It's ds4, a from-scratch inference engine written by antirez, also known as Salvatore Sanfilippo, the creator of Redis, and it does a handful of clever things that no general-purpose runner does.

The catch of all this, though, is speed. I'm getting somewhere between 10 and 14 tokens per second of output depending on how much context is in use. On top of that, the software is only a few weeks old and openly beta, and you need a fairly serious machine to run it at all. But it works, and it works well, and once you stop expecting it to behave like a real-time chat window it turns into something I didn't think a local setup could really be just yet.

DeepSeek V4 Flash is huge on paper and light in practice

It has a unique architecture

DeepSeek released the V4 preview family back in April under an MIT license, and it comes in two sizes. V4 Pro is the giant of the pair, at 1.6 trillion parameters, whereas V4 Flash is the smaller sibling. "Smaller," though, is doing a lot of work, given that it's still 284 billion parameters. The trick is that it's a mixture-of-experts model, so only about 13 billion of those parameters are active for any given token. It's expensive to hold in memory but cheap to actually run, which is exactly what you want if you're trying to fit something closer to the frontier-class onto a single machine.

The other thing V4 brought is a reworked attention design (called Compressed Sparse Attention, paired with Heavily Compressed Attention) that compresses the KV cache, the running memory a model keeps as it works through a conversation. Both V4 models ship with a one-million-token context window as standard, which would be a pipe dream on many local setups. DeepSeek's compression is what makes it realistic, and as it turns out, it's also the thing ds4 makes use of the hardest.

None of that helps if you can't load the weights, though. No matter how much you optimize, 284 billion parameters at full precision is far more than a 128 GB machine can hold, so something has to give. And ds4 tries to make use of DeepSeek V4's unique architecture as much as possible in order to get it running on machines with 128 GB of VRAM.

ds4 only knows how to run one model family

It's for DeepSeek V4, and DeepSeek V4 only

Most local inference runs through Ollama or llama.cpp, as both are built to run almost anything. That broad approach is the whole appeal of both systems, but it's also the compromise, since a runner that supports hundreds of architectures can't really hand-tune itself to any single one of them. ds4, which Sanfilippo calls DwarfStar, goes the other way entirely. It runs DeepSeek V4 and nothing else; it doesn't link against GGML, it doesn't load arbitrary GGUF files, and it isn't a wrapper around another runtime. It's a self-contained lump of C written to serve exactly one model family as well as it possibly can.

This project is in beta and the code was, per Sanfilippo's own words, written with heavy assistance from GPT 5.5. So, if AI-written code bothers you, then this simply isn't for you. However, the payoff of that narrowness is that every layer, from the quantization method to the tool-calling to the way it handles memory, is built around the quirks of one model instead of the average of many.

The engine ships builds for Apple's Metal, Nvidia's CUDA, and AMD's ROCm, and rather than pointing it at a folder full of models, you pull one of a small set of quantizations Sanfilippo has provided himself. On the PGX I built the CUDA version with a single make cuda-spark, since the GB10 chip inside it is the same silicon as Nvidia's DGX Spark, then grabbed the Q2 quantization meant for machines with 96 to 128GB of memory. There's no menu of a thousand models to get lost in, which is either a limitation or a relief depending on how you like to spend your evenings.

Fitting that many parameters into 128GB means squeezing the weights down to roughly 2 bits each, and that kind of aggressive quantization normally does visible damage. Tool calling tends to be the first thing to break, since it depends on the model reliably producing exactly-formatted output, and a model that's been compressed too far starts fumbling the syntax. But there's a neat trick here to overcome that.

ds4 gets around this by not quantizing everything equally. Only the routed expert layers, which are the bulk of the parameters, get taken all the way down to 2 bits. The shared components that every token passes through; that is the attention, routing, and projections layers, are left at higher precision. Those are the parts where a heavy quantization would rear itself, so keeping them intact holds the model together and, crucially, keeps its tool calls well-formed even after the rest has been crushed.

In practice, it holds up. I've been using DeepSeek V4 Flash paired with Claude Code and watching it tear through long chains of tool calls without anything falling apart, and that's not something I'd have expected from a 2-bit model even just a year ago.

The KV cache doesn't have to live in RAM

It can live on the SSD

ds4 has another neat trick when it comes to the KV cache, which stores the model's internal representation of the context window, handling it more aggressively than any other runner I've used. Normally the cache lives in RAM or VRAM, full stop, and it grows with your context until it doesn't fit and everything falls over. Sanfilippo's argument is that DeepSeek's compression, combined with how fast modern SSDs have got, means that the assumption your context needs to stay in VRAM is outdated. In ds4, checkpoints of the KV cache are persisted to the disk. It gets written out, checkpointed, and loaded back in as it's needed.

That changes two things. A huge context window is no longer a hard memory wall and, instead, simply becomes a question of disk space and patience instead. And because the cache is sitting on disk rather than only in memory, sessions survive, so you can stop one and pick it back up later without re-processing the whole conversation from the very beginning.

I kept an eye on the server logs and could see this play out in action. On one turn, ds4 restored just under 25,000 tokens of cached context from disk in 197 milliseconds, then only had to process the couple of hundred new tokens I'd actually added. Prefilling those same 25,000 tokens from cold, which I could see happening on other turns, takes around 73 seconds on this machine. So the disk cache turned what would have been a minute-and-a-bit wait into essentially nothing.

It isn't a clean win all the time, though. When an agent reshuffles its own reasoning between requests, the cached prefix stops matching and ds4 has to re-process it anyway, and you can see those misses in the logs, too. The cache is doing a lot of work, but it's a spectrum of savings rather than a guarantee, and it's at its best when used in a cold stop-and-resume situation rather than in the middle of a busy agent loop.

On the ThinkStation PGX, it prefills slow and generates slower

It's not exactly real-time

The PGX is a good match for this on paper. It's a GB10 Grace Blackwell machine with 128GB of unified memory, roughly the size of a Mac Mini, and it draws a maximum of 240W. ds4 has a build target for this exact kind of hardware, and the Q2 quantization is sized to fit this much memory, so there was very little fighting involved in getting it up and running.

Once it was started, I pointed Claude Code at it using my local Claude Code script, gave it a context window of a few hundred thousand tokens, and let it loose on a large codebase. Prefill, the phase where the model reads through everything you've handed it, runs at around 330 to 350 tokens per second early on, which is fast enough but still, overall, quite slow. Generation, where it writes its answer one token at a time, starts at around 12 to 14 tokens per second. That gap isn't a fault in ds4, it's just physics, because writing tokens is limited by memory bandwidth, which GB10-class machines don't have a lot of compared to most mid-to-high-end GPUs.

Unfortunately, both of those numbers are best-case scenarios, and they start to fall sharply as the context fills. By the time I'd fed it well over 100,000 tokens of codebase, prefill had dropped to around 200 tokens per second and generation was down under 10, since every new token now has to attend over a much bigger cache, and each checkpoint the engine wrote out to disk had grown past a gigabyte. It's all in line with exepctations, but it's still painful that a long agent loop gets slower the deeper it goes.

The tool calling held up nicely under real use. Watching the logs, I could see DeepSeek V4 Flash firing off batches of file reads and web searches. It was doing it in the text-based format DeepSeek uses instead of JSON, and ds4 mapped each of them cleanly back to the tools Claude Code expected on the other end.

The reason I used Claude Code was simply because it's one of the harnesses Sanfilippo shows off in the repo as something that just works out of the box. Pi also works, but Claude Code is a known quantity, so I figured it was the best chance of having a consistently good experience. However, the first request Claude Code sends is enormous, since it bundles the entire system prompt and every tool definition into one payload. On my setup that came to a little over 26,000 tokens before I'd typed a single word, which took the best part of a hundred seconds to work through. After that first hit, the disk cache keeps it saved, so the processing of those tokens is mostly a one-off at the start.

Then there's the power draw, which I was greatly impressed by. Under sustained load, reading a whole codebase and generating from it, the entire machine sat between 120 and 140W. That's a 284-billion-parameter model running flat out and pulling less than a gaming PC does under load, on a box you could tuck behind a monitor and forget about.

What it's actually good for

Long runs or overnight workflows

So what is a setup like this for? Not real-time chat, that's for sure. At around 10 tokens per second, and slower the bigger the job gets, sitting and watching it work through a large agentic task is like watching paint dry, and if you need an answer right now, the cloud is still right there and still faster. Where it comes into its own is the exact opposite of interactive. You hand it something substantial, a big refactor, or a codebase to read through and document, and you walk away from it.

For example, I ran it overnight, building a CS2 demo parser that would give me the ability to highlight events from the demo file and, in future, show a 2D replay, and I have no problem with it taking its time. I'm not in a rush, and there's something great about waking up to a finished task that never even left my network, didn't cost me any API fees, and didn't even draw that much power. In this context, you wouldn't think of it as a chatbot, but rather a worker you can point at a problem where an answer will be prepared for you in an hour's time or so.

That's the biggest trade you make here; you give up speed, but in return you get privacy, incredibly cheap running cost, and a model capable enough to do the job without a subscription or a network connection. It only makes sense if you've already got the hardware, since a GB10 box or a 128GB Mac isn't a casual purchase, but if you have, DeepSeek V4 Flash on ds4 is the first time local inference has felt in any way like a cloud model does without needing to offload work to cloud-hosted MCP servers. For the work you're not in a rush to have completed, it's a lot more powerful than I expected.