There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
Yes! One method is KV cache compression to 4-bits - llama.cpp does have hadamard, but it's not very accurate - we were researching if 4-bit KV cache can be used with a sliding window of 1024 for example of 8-bit KV cache
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
As an experiment I took, not this current quant, of Qwen 3.8 27B and had it analyze some of my writing in CPU-only mode with llamacpp. Tried a Q4 and Q5. It was dreadfully slow, but you fire that off before bed, and you can get results the next day. This was a ~6000 word markdown file, took maybe 10 minutes before it started generating anything at all, but then it just cranks away. I would need to tune down the default thinking intensity, but it otherwise did the job.
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
FWIW depending on what exactly you want to review something smaller like Gemma 4 E4B would be much faster. On my 7yo 3700X, llama.cpp runs Gemma 4 E4B UD-Q4_K_XL with a 131k context (context size doesn't seem to make much of a difference in speed) at ~10-15 tps (using MTP draft).
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
I’ve been using Bonsai 2 (also based off of Qwen3.8 27B) on my mac mini in this way (using pizza-bot for the harness). In my case I’ve been using more for deep web search type stuff. It’s been a nice workflow to kick off a request in the evening and come back to a 1 page report on it the next morning.
It’s fun, but it’s just so tough to see the value when the same research report via ChatGPT is done in 5 minutes and likely higher quality, for less than a dollar a day.
This would depend hugely on the price of electricity, however I'm pretty sure where I am (at roughly 25c/kwh in usd) the price of running qwen costs more than 6.1-luna per token. So other than privacy (which doesn't matter for my work) there isn't any justification for running locally. Is privacy purely what people justify local models on?
Bonsai 2 does some additional tuning in addition to just quantizing (mostly around the fp16 block weights per 32 params), so they’re able to pack more in to just so few GB.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Depends on how unique the needles are. If they're fairly unique then even Qwen 3.5 27B @ Q5 with 128k context and Q4 KV cache has perfect recall of at least up to 5 needles, according to my tests.
But I'm sure if you make them less unique they'll start to struggle.
Every local and reasonable model is bad at that. It’s OpenAI’s own benchmark, so take it with a grain of salt, but “GPT-6 Astra achieved 100.0% accuracy on contexts from 256K to 512K tokens and 96.3% accuracy from 512K to 1M tokens on OpenAI's 8-needle long-context benchmark”.
Just installed Qwen 3.8 Splash tonight (doesn’t appear on linked page). Super impressed. Easily averaging 45 tps, even with thinking on. Totally acceptable. Running M4 Mac mini, 48GB, LM Studio.
We also updated https://huggingface.co/unsloth/DeepSeek-V4.1-Flash-1017-GGUF with UD3.1 quants
And also https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF as well!
We provide our benchmarking methods like what graders we use, infra fixes and issues and more in https://unsloth.ai/docs/basics/dynamic-3.0-ggufs/dynamic-3.1...
what would you recommend re: software stack for best pp/tg overall performance?
So we can have a 4bit KV cache, but the most recent N tokens can be 8bit - we found accuracy improves 2x!
Not compatible with vibe coding, but if you want a local tool to review content, it is possible today.
I'm mainly using it when i need some simple stuff and Qwen 3.8 27B is already occupied in the background :-P
Less important parameters should get quantized more, seems odd that we would quantize every parameter the same. At least for storage.
The tradeoff with unsloth seems to be that it’s more directly applicable for other off the shelf open models, whereas with Bonsai you’d have to wait for them to release another one
Bonsai's technology would be better suited for 100s of B models.
But I'm sure if you make them less unique they'll start to struggle.