Rendered at 19:36:00 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
SwellJoe 23 hours ago [-]
What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.
janalsncm 21 hours ago [-]
Based on their numbers and cross referencing with the Gemma numbers, this model crushes Gemma 4 12b on math and coding, is slightly worse on knowledge and tool calling, and is significantly worse on vision tasks.
recursivegirth 20 hours ago [-]
I think this is where leveraging classifier models will become important. The frontier LLM models do "everything", while we've known for a while that to truly scale this we will need to distill models into their individual functions. I don't see this as necessarily a bad thing and hope more is done in this space. Very promising.
bigmadshoe 20 hours ago [-]
Bitter lesson is knocking. Mixture of experts is essentially what you’re describing but free from unnecessary inductive biases.
hovering_nox 10 hours ago [-]
Mixture of Experts is absolutely not what they're describing. MoE has to be one of the most misleadingly named things ever. It's completely confusing as to what it actually is.
bigmadshoe 6 hours ago [-]
Note the “but free from unnecessary inductive biases” part of my comment. By that I meant a decision to make each expert good at a human-defined thing.
ralusek 9 hours ago [-]
MoE is literally exactly what they're describing. The classifier being described is baked into the model and is the thing that makes it MoE.
Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.
eamag 6 hours ago [-]
only there's no [a model that was good at math], [model that was good at code]... in the MoE
kennywinker 17 hours ago [-]
There is value in splitting things. If all I ever do is local app automations, i don’t need model that knows how to code. If all I ever do is coding, i don’t need a model that translates english to slovakian.
ArcHound 15 hours ago [-]
Slovakia mentioned, let's gooo. Ehm, exactly, we can achieve better smaller models for specialized tasks rather than using compute to improve a big model that does everything. There's a lingering philosophical question if better language processing capabilities translate to better image processing capabilities (i.e. having the vocabulary and experience to properly describe an image), but I still think that identifying tasks and splitting responsibilities saves a lot of effort.
14 hours ago [-]
bigmadshoe 14 hours ago [-]
Good point! I thought you meant splitting them and then doing inference with some kind of learned router while keeping all the split models loaded at once. What you're suggesting is pretty sensible.
Zababa 12 hours ago [-]
There is value in splitting things but there is also a cost. You have to train the specialized model, for that you have to know your use case, you have to hope the use case is going to be stable over time, you then have to see if you can remove english -> slovakian or coding from a model without affecting the useful parts.
graysonk 8 hours ago [-]
I enjoy doing local image generation and this is one thing that the community around that has really optimized.
In some workflows you might have 20 different models doing their specialized tasks. Pose detection, hand/eye/face detailers, classifiers, refiners, up scalers, taggers, etc can all use their own models and that’s not even the including the model(s) used for the actual image generation part.
I’m interested to see the optimization when this concept gets applied to other general ai tasks.
dofm 19 hours ago [-]
FWIW my tests on my little puzzles suggest that it is not better than the Gemma 4 12B on SQL. It really does seem to get quite tangled up on stuff.
PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops.
Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.
sosodev 17 hours ago [-]
Qwen3.6-27B is the best model in that range that I’ve used for agentic coding by far. I think it’s kinda mid at everything else.
larodi 12 hours ago [-]
Surely not that good at vision. TBH none of these 14-27b models come close to even the cheapest Gemma 2.5 flash.
If these buddies are similarly bad on text, then they definitely don’t get anywhere close to big boys, no matter what the synthetic stats claim upon release.
tharkun__ 19 hours ago [-]
From my own experiments with local, low VRAM model use vs. what I'm used to from using Claude at work is that being good at "coding" is of no use, if you're worse at "tool calling" as coding in an agentic way requires quite a bit of tool calling.
If you can "hide" different models of 8GB VRAM requirements each that have those specialties and mix and match them for me without having to manage it manually, I'll be impressed. Until then I will keep using my Claude, because "remarkably good _for their size_" models I've tried so far just sucked at trying to use them the way I code at work with Claude.
hypercube33 20 hours ago [-]
More to the argument that we need a model of models - one general one that calls specialists in to do what they are good at and handles that like a foreman for you.
kaycey2022 16 hours ago [-]
Is that different from mixture of experts?
int_19h 12 hours ago [-]
Yes. A mixture of experts is a single model that activates different routes though the same weights, with the route possibly changing literally on every token. It's not experts as in a bunch of standalone models that are good at specific high-level tasks.
dofm 19 hours ago [-]
This is somewhat akin to the "one-expert-per-query" solution Apple are using in their small foundation models I think?
SwellJoe 20 hours ago [-]
To be fair, everything (roughly within an order of magnitude in size) is worse on vision. 12b is a beast for vision tasks, better than its bigger siblings, even.
CuriouslyC 20 hours ago [-]
The things it loses are all the things that google models are historically excellent at, so that's a reasonable performance. I think the take home here is that the 1 bit models are probably better, but it's not a slam dunk given advanced quantization techniques.
sosodev 20 hours ago [-]
The Gemma models are so good at vision. It seems particularly important for phones. Also, they write in a much more pleasant manner than Qwen imo.
dofm 19 hours ago [-]
I absolutely agree that Gemma 4 writes well out of the box. Free of a lot of the standard American model blog spam writing style but a little more fluid than Qwen.
moffkalast 13 hours ago [-]
> slightly worse on knowledge and tool calling
Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.
SwellJoe 11 hours ago [-]
I think that depends on how you run it. Llama.cpp has several fixes for the somewhat unusual tool call semantics in Gemma 4. I don't think I have noticed any issues.
xienze 9 hours ago [-]
I think it's extremely quantization and engine specific. I run Gemma4-31B at FP8 on vLLM and it's fantastic, no issues anymore[*].
* I will say that early on there were a LOT of issues with the chat template, across all engines. I dunno who decided using crappy Jinja templates was a good idea, but clearly it has its limitations. In the latest version of vLLM (0.25) they've ditched the Jinja templates for an in-engine parser and I've seen no issues.
verdverm 22 hours ago [-]
4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn.
I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.
dofm 21 hours ago [-]
The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better).
The 12B QAT model is indeed sort of mindblowing.
novaRom 21 hours ago [-]
Gemma 4 12B QAT is amazing - agents run very fast, and it's really very smart, at least in my agent's harness domain which is GNU software development - on par with frontiers like GPT Sol, DeepSeek, or Claude - Why to buy those expensive tokens if a local tiny model performs so well?
gwerbin 18 hours ago [-]
What's your harness setup? I haven't had this kind of on-par success with any local LLM yet.
sosodev 17 hours ago [-]
They’re exaggerating or have a very simple way of using these models. The Gemma 4 series, even at 31B, is nowhere near the frontier. They’re great models, but you will notice a huge difference for complex tasks.
The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.
xienze 9 hours ago [-]
> They’re great models, but you will notice a huge difference for complex tasks.
Yes and no. I think where frontier models really blow small models away is in how thorough they are in order to infer your intentions and how best to accomplish them. So you can tell Claude "change this code to make it do X", whereas a Qwen3.6-27B or Gemma4-31B can do the same, but you have to be a lot more thorough, i.e., "change this code to make it do X, but first, let me explain the concept of X as I see it and some notes about things to avoid or pay attention to while you're doing it." So for best success with small models you really need a big toolbelt of skills and MCPs.
verdverm 17 hours ago [-]
same with opencode, next on my evals.list are the fine tunes from big model traces, and then my own if ftuning looks reasonable cost/time wise
I have two main tasks I want to see if I can improve, coding and doc understanding/summary
verdverm 21 hours ago [-]
I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant.
Hopefully more of the lab releases are trained under QAT so we can all benefit.
dofm 21 hours ago [-]
I think they did Gemma 3 QAT models and there are QAT versions of essentially all the Gemma 4 models (including DiffusionGemma).
ckluis 20 hours ago [-]
I'd like to see them do a 1-bit binary version of Gemma 4 12B ;)
SwellJoe 17 hours ago [-]
I want 31B. The 12B 4-bit QAT is already small enough to run well enough on every device I use regularly, including phone and tablet, I don't need a 1-bit or ternary version of the 12B.
But, what I really want is for Google to release bigger Gemma 4 models, particularly a bigger MoE, like a ~70B or ~120B. Gemma 4 is the best all-rounder among the models I can self-host even though I've got a 128GB Strix Halo. A 4-bit QAT version of a 70B MoE would probably be the sweet spot.
A bigger Qwen 3.6 with a 4-bit QAT version would also be welcome, as the prior bigger versions aren't notably better than 3.6 27B, but I guess Qwen is done doing larger open weights models. They did release AgentWorld recently, a post-train of the 3.6 MoE, so they're still doing some open things.
I think I want to see more third-party testing of this ternary Qwen to know if crushing it to 1.56 bits kills it; there are tons of benchmarks of Qwen 3.6 27B, so it's an ideal candidate to figure out what the extreme compression does to it.
motbus3 23 hours ago [-]
I need help understanding this.
I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models.
How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?
I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem.
Anyone care to explain this poor silly fellow some of those points?
jedbrooke 22 hours ago [-]
from what I understand prismml isn’t doing a quant like normal models where you take a model trained at fp16 and then chop off some bits to reduce vram, but rather they’re training the model natively with 1 bit weights. It’s explained more in the article. They’re also doing some other tricks like a fp16 weight per block of 128 1bit weights to get some more data out of 1 bit weights
parsimo2010 19 hours ago [-]
They aren’t training at all. They are quantizing existing models, it’s just that the process is different. The 27B uses Qwen3.6 27B as the base model.
0c3ca83 23 hours ago [-]
[flagged]
23 hours ago [-]
mehmetkose 12 hours ago [-]
1-bit weights are not pointers so cpus can process them, storing them takes less space etc. tons of gains
kbart 8 hours ago [-]
Excuse my likely stupid question, but has anybody had some success using Claude Code with frontier agents (or Junie or anything else) to invoke local LLMs for specific sub-tasks or wrapped as skills? In other words, is there a way to use expensive, frontier models as orchestrators that manage local models to do the specialised coding tasks?
docheinestages 7 hours ago [-]
Look into FastContext by Microsoft. Not extraordinary but specifically designed for paired usage with a larger LLM to save tokens [1].
The system as a whole is meant to support that use case, where each task (ticket in its jargon) can be tackled using a custom workflow that can each use a different agent/llm (so, it should support local LLMs if you have configured your coding agent to use them).
Sidenote: it's still not where I want, but getting there...
diseasedyak 6 hours ago [-]
This may not be exactly what you're asking, but I've been running Hermes using different frontier models (like DeepseekV4-Pro, GLM-5.2, and others) as the heavy lifters but who also spawn agents to run my local models (mainly Qwen3.6-IQ4) for offloading tasks that are a good fit for a quantized model that's local.
Works really well.
stiltzkin 2 hours ago [-]
[dead]
adam_patarino 7 hours ago [-]
What for? Cost savings? Something else?
newswasboring 7 hours ago [-]
That is my entire workflow in opencode through delegates. I have an "orchestrator" which uses matt pocock skills like grilling and writes specs in gh tickets. Once that is done it delegates individual tickets to deepseek v4 flash based developer subagent which executes the code pretty fast. This agent can only delegate one subagent which is a code-reviewer which uses a better model like qwen 3.7 for review. So dev does its own review loop before returning.
It's going amazingly. The orchestrator holds the big knowledge from grilling and also enables me to do more grilling to refine the specs. The trick was to check and re-align after each phase was developed. Also carefully defining each workflow step explicitly otherwise it makes mistakes like trying to self review the code. Also needed to define very explicit contracts. I chose the same surfaces as the matt skill set.
rtcoms 5 hours ago [-]
Do you have source code or config for this setup ?
newswasboring 5 hours ago [-]
It is just basic agent setup. When I get home in the evening I can isolate my agent files and share GitHub link. In a couple of hours.
Edit: So life got in the way and I couldn't isolate it. But I have made this project public and committed the dev and code review agents here.
The delegate builder is too ugly to share yet. But to be honest just use the normal builder with a frontier model and instruct it to delegate to developer any developer task. That works well.
Not able to access the link, may be it is a private repo.
NamlchakKhandro 7 hours ago [-]
why would you want to use claude code?
networked 12 hours ago [-]
I have benchmarked Bonsai 27B CPU inference on my computer (a Ryzen 7 5700X desktop with 48G RAM running Ubuntu 24.04) using the latest 62061f910 build of PrismML's llama.cpp fork.
Binary: 9 t/s prompt, 6 t/s generation. Ternary: 0.8 t/s prompt, 0.7 t/s generation. It looks like CPU inference for ternary isn't optimized yet.
Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.
Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).
dofm 21 hours ago [-]
I would be slightly surprised if anything Apple wanted to do with Gemma they couldn't do with Gemini, which they have the right to make various derivatives of.
They could more or less redistribute Gemma as-is in the developer program; they are unlikely to be troubled by any of the licence terms.
segmondy 22 hours ago [-]
Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.
conception 22 hours ago [-]
You’re right, Apple only has 68 billion dollars in cash, up 40% since last year. Definitely on their last legs.
mgambati 21 hours ago [-]
Free cash? Crazy. I think I can live with the interest of that in the bank.
nxtfari 21 hours ago [-]
you could live (100k a year) with the interest of 0.004% of that in the bank.
brcmthrowaway 21 hours ago [-]
I remember the infographics of 5-10 years ago. They had 200 billion. What happened? Sinking ship?
malshe 21 hours ago [-]
They never had $200 billion in cash in any quarter. Not at least since 2014. Besides, how much cash to hold is an executive decision. If the cash balance went down, it doesn't mean their business is not doing well. Cash can be used for Capex, like what the hyperscalers are doing right now. It can be used for buying back equity.
mandeepj 20 hours ago [-]
They spent close to $100 billion in stock buybacks during the last 10ish years :-)
brcmthrowaway 19 hours ago [-]
How does that compare to other companies?
mandeepj 5 hours ago [-]
What do you think?
mandeepj 21 hours ago [-]
> Apple is too desperate to be making demands
They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?
dofm 21 hours ago [-]
They didn't buy Perplexity and it indeed rather seems like Perplexity may have damaged the deal by blurting.
Whether things are different now I don't know, but Prism would potentially be a good acquisition.
This is not the first Bonsai model from Prism using this technology, and they've also applied this technology to an image model.
The model aspect isn't that significant (even in this news), because it is Qwen, under the hood. The encoding efficiency is.
Prism's technology might be valuable, and their team could well be.
And they very evidently do care a lot about efficient on-device AI; they just don't care about developing frontier cloud models.
segmondy 21 hours ago [-]
They do, they are low key panicking.
dofm 21 hours ago [-]
I do not believe they are panicking, not least because I don't think they've finished adjusting Apple Silicon for the task; it will be very interesting to see what happens in the M6 and M7.
I think their strategy is broadly correct, actually; I think there's still a bit of scope for "sit and wait and do it right" here. But acquiring more edge AI tech and edge AI people would potentially be in their interest.
2 hours ago [-]
trollbridge 21 hours ago [-]
And if they aren't, they should be. They ship some of the cheapest, most capable, and easiest to buy/set up AI hardware out there, and then don't ship any AI software worth speaking of. Apple needs to fix this.
literalAardvark 11 hours ago [-]
"Apple needs to get into gold prospecting, this funny business of selling the best shovels just can't work out"
thakoppno 22 hours ago [-]
either way, maybe a portent for the times.
apple’s secrecy agenda has been defeated to an extent by the practicalities of ubiquitous technology?
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
bansaltushar 24 hours ago [-]
Depending on which model you're running, you might need to use the custom forks.
I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :)
I am happy to see the message. Thanks!
anon373839 17 hours ago [-]
I can report that it's working in oMLX. I've been experimenting with the ternary one; it is quite an impressive model! I've been grilling it on some deep learning/computer vision stuff and it's aced everything so far. Responses are thorough, accurate, sophisticated. General knowledge outside of CS doesn't seem as robust, which I expected. Honestly, I don't think the examples in the blog post do it justice.
SwellJoe 21 hours ago [-]
I got their previous model working in their custom fork of llama.cpp (https://github.com/PrismML-Eng/llama.cpp). I haven't tried this one yet, but will find some time to benchmark it sometime this week.
Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.
dofm 21 hours ago [-]
Not sure if there's any way to run Prism's fork of llama.cpp inside LM Studio.
The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high.
On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.
trollbridge 24 hours ago [-]
Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.
dofm 19 hours ago [-]
Have Prism-ML upstreamed their forked code?
lta 7 hours ago [-]
Yeah it's a llama.cpp fork and it's linked everywhere on their repo
dofm 2 hours ago [-]
Of course — I compiled it and ran it yesterday. Not aware if they have a PR in place with the main project, though as I recall I ran the last Bonsai model without needing it so I guess there might be plans.
try-working 20 hours ago [-]
I downloaded two of the official ones in LM Studio, both 3.6gb, and neither loaded.
davedx 12 hours ago [-]
I find it super interesting that we're now in an era where we have LLM's that are quantized to binary weights - 1's and 0's. So effectively they're digital neural networks.
I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?
jboss10 12 hours ago [-]
Most of the time, the speed of these models are constrained by memory bandwidth. GPUs normally have much more memory bandwidth.
davedx 10 hours ago [-]
I'd expect the memory bandwidth to be the same for the CPU and GPU under a unified memory architecture like Apple silicon uses?
kmacdough 10 hours ago [-]
Its not about geometry, it's a parallel compute thing thing. CPUs typically don't have more than 10 or 20 cores. GPU have 100s to 1000s.
Matmul is very well parallelized. More, lower power cores will always pay off handsomely.
Havoc 7 hours ago [-]
Got this running on my phone. Unfortunately like other small models it hallucinates quite easily.
eg asked it what Signoz is. It reckoned it is a woocommerce/shopify competitor aimed at India market
Arcuru 23 hours ago [-]
Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.
Same here. I’m excited to have a model that might be usable on a 16 GB laptop.
erwan577 24 hours ago [-]
The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
verdverm 23 hours ago [-]
quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.
RugnirViking 21 hours ago [-]
maybe its nitpicking here but the demo shows them asking the model what to cook and its recipie sounds like it wouldn't be very good and also it totally gets the macronutrients wrong. 25g protein for "spaghetti, carrots, peppers, garlic and herbs"?
purerandomness 20 hours ago [-]
Good spaghetti has 15% protein, 15g per 100g.
Even the lowest quality one has 12g/100g.
gnabgib 20 hours ago [-]
Pasta should be made of whole durum wheat.. which is 12g/100g. What's this "good" you speak of? There's certainly worse (soft white/red flour with germ/bran removed), but where do we find the +3g?
May be going out on a limb here but I think it might be the eggs?
kennywinker 17 hours ago [-]
What LLM model are you using to figure that out? The reasoning mode must be frontier level.
/satire, obviously ;)
stamps 19 hours ago [-]
I know felcetti has a few varieties that exceed that number. ~112g for 16g protein.
gnabgib 19 hours ago [-]
The brand that uses khorasan wheat in some versions? Yeah they're a little higher (14.7g/100g). We're stretching the definition of pasta now (it's certainly the highest protein grain you can find)
stamps 19 hours ago [-]
Yeah I’m not 100% sure, they claim “special durum wheat variety.”
For the price, from a protein perspective, it would be cheaper to get it elsewhere.
They do, but they have to use the standard numbers for durum (like everyone else, even growers - individual berries vary, of course). The only one I can find with these high numbers is the khorasan wheat versions[0].
I personally don't like carrots much, but it doesn't sound bad to me – could definitely use a tomato sauce but that doesn't seem to be an option from the image it was given.
> 25g protein for "spaghetti, carrots, peppers, garlic and herbs"?
Maybe it assumed the pasta was some kind of protein chickpea pasta? =P it definitely seems wrong.
SamBam 20 hours ago [-]
Many people don't realize that regular spaghetti actually is pretty high in protein. All that gluten.
sandworm101 21 hours ago [-]
And why would i want such mundane questions to be handled by an AI on my phone? That sort of thing doesnt need AI, let alone a local one. Basic google search was answering those questions long ago. My point: phone-sized AI is only useful if it can do things that only AI can do. Can it ingest a document scanned by the phones camera? Can it translate in real time? I dont see how or why i would ever ask it for recipe advice. That need is met elsewhere x10.
codebje 20 hours ago [-]
If it can give me the recipe without 14 pages of backstory about how Nonna used to make it, it'd be satisfying a real need.
foolfoolz 20 hours ago [-]
every single recipe blog site builder has a “jump to recipe” button at the top
tenuousemphasis 15 hours ago [-]
Paprika does that for any recipe.
timfsu 13 hours ago [-]
I had exactly this use case - in a grocery store in the Alps, no internet, fired up a local LLM on my phone to figure out what to cook and what to buy
recursivegirth 20 hours ago [-]
Awwwe, cmon. You are thinking about the problem space incorrectly. This is an opportunity to create a unicorn company that develops the first AI tongue.
alvatech 1 days ago [-]
TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1
NitpickLawyer 1 days ago [-]
There's two variants of this (or, as the joke goes, for very big values of bit):
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
PcChip 24 hours ago [-]
this is a really dumb question, but how is -1 represented?
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
edflsafoiewq 23 hours ago [-]
It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.
Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.
It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
snthpy 12 hours ago [-]
Thanks. This relates to some questions I've got. I was playing around with the previous generation smaller models and found that i wasn't getting any speedups from the T1 and T2 binary/ternary models compared to standard Q4 quants of straight qwen3.6 models. I was wondering whether unpacking of the ternary encoding was impacting inference speed?
If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.
You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59).
At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.
lioeters 23 hours ago [-]
> never heard of a bit ever having more than two possible values
It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.
zawaideh 24 hours ago [-]
It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.
throwawayffffas 23 hours ago [-]
I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer.
The way they do it is packing like the other comment says.
Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.
Dwedit 22 hours ago [-]
1.6 bits if you want the most practical way to pack five 3-state numbers into a single byte. But even then, they usually pack four 4-state numbers instead.
bensyverson 1 days ago [-]
Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.
thomasjb 1 days ago [-]
I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)
kennywinker 16 hours ago [-]
Toss the rtx into a cheapo optiplex or thinkcenter, and run it headless - the load on your machine is gonna make doing other stuff while it’s running painful. Plus that frees up the rest of your vram.
HalfCrimp 8 hours ago [-]
Running an llm on a PC running a desktop environment really isn't that bad. You lose a bit of ram & vram but that only matters if you _reaaaally_ want to push to the max model size your hardware can handle.
The biggest issue I've found is absent mindedly opening YouTube or the like that spike ram requirements and freezing the system up. But that's a me problem
xyzsparetimexyz 13 hours ago [-]
Why would they do that when they already have a perfectly good PC?
kennywinker 12 hours ago [-]
…? I said why.
> the load on your machine is gonna make doing other stuff while it’s running painful
Is your question about something else?
thomasjb 11 hours ago [-]
I aspire to someday move up to an AM5 system, but for now, the Dell T3610 has to do everything. Might get a second GPU for it though.
dakolli 23 hours ago [-]
start saving your money.
thomasjb 21 hours ago [-]
Or watch and wait as models get denser
liuliu 1 days ago [-]
The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.
liuliu 1 days ago [-]
You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.
h14h 23 hours ago [-]
I'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom:
This is accelerant #3 and #4 from our article converging in one release: a 27B-class model, built on Qwen (already one of our examples of local models "good enough to matter"), now running on an iPhone. The hardware layer and the local-model layer aren't just going to converge in the future, they're doing it right now! https://news.ycombinator.com/item?id=48892559
syntaxing 1 days ago [-]
For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.
luckystarr 1 days ago [-]
Tried it on Android and got "!!!!!!!!!!!!!" for answers.
gunalx 23 hours ago [-]
The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.
aesthesia 18 hours ago [-]
Oh, interesting. "!" is token id 0 in the Qwen tokenizer; I wonder if there's some tokenizer shenanigans either in inference or training that end up causing this specific behavior.
amelius 22 hours ago [-]
Wait in a regular sentence, what is the probability of "!!!" being followed by "!"?
Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.
jldugger 22 hours ago [-]
After the third !, the probability of a fourth probably skyrockets =)
verdverm 23 hours ago [-]
That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.
When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.
kamranjon 23 hours ago [-]
After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.
digdugdirk 23 hours ago [-]
What model? And what hardware do you run it on?
I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.
How do you find the overall experience? And do you have any special sauce or recommendations for going this route?
kamranjon 22 hours ago [-]
I’m using DeepSeek V4 Flash on 128gb mbp - it’s a bit different using a 200b+ param model. It’s MoE so performance is acceptable. It will still malform a tool call every now and then, but the capabilities are so far ahead anything else that the majority of the time it works really well and solves really complex problems.
apitman 15 hours ago [-]
I've got dual 3060s as well. What's the best models you've found for this setup?
The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.
pylotlight 18 hours ago [-]
But isn't it running basically 1 request at a time? This would make agentic coding difficult right? Compared to running as many sequential tests as you want via api?
kamranjon 15 hours ago [-]
Yea I don’t use sub-agent style workflows. I often use planning patterns and generate markdown for really complex tasks, but never got into the sub-agent thing. I am likely closer to AI-assisted, though I am heavily using pi coding agent and my editor is basically just for viewing files and changes now.
trvz 22 hours ago [-]
I still don’t see the point of this. In my testing, it’s worse than Qwen 3.5 4B and even 0.8B.
kamranjon 15 hours ago [-]
When new models are released (I realized this is qwen 3.6 but the quant is novel) - it takes a few days for the kinks to get worked out - you’ll likely have better luck if you give it a few days and try again.
rvz 21 hours ago [-]
It is still worth experimenting. Although we do need independent evaluation of these models instead of labs posting such biased and skewed results.
sigbottle 1 days ago [-]
What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?
trollbridge 24 hours ago [-]
If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.
doctoboggan 23 hours ago [-]
Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.
ndrwdvvs 22 hours ago [-]
or smart fridge
try-working 20 hours ago [-]
open source is a GTM strategy
comandillos 23 hours ago [-]
Quite weird that heavy quantization method on a dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.
At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.
LeBit 21 hours ago [-]
26B-A4B?
snthpy 12 hours ago [-]
What is the best way to deploy these on CPUs, arm64 ones in particular?
I'm interested in the CPU inference application of these models with things like the FairyFuse kernels.
I've tried trillim previously but was disappointed that i got higher tok/s just with similar sized models through ollama using just Q4_K_M quants.
I see there is bitnet.cpp and litespark-inference. What else should i look at?
MrDrMcCoy 11 hours ago [-]
Llama.cpp should be rolling out support for this soon if they haven't already. Cactus is a but more targeted for efficient ARM execution, but I haven't been keeping up with what all they support. Would be worth a feature request if they aren't working on it yet.
athrowaway3z 22 hours ago [-]
So first off, phenomenal stuff to see a 1bit model at 90% capability.
However, this is the 5th product post in 2 weeks that proclaims that AI use is shifting, and why [insert tradeoffs] are the perfect fit.
Paradigms shift don't happen in the release announcements.
I suspect this is an AI-ism, making all the release posts sound so paradigmshiftery.
jrflo 22 hours ago [-]
That is true of all tech announcements. Marketers will do their thing regardless of reality.
est 21 hours ago [-]
Maybe Taalas could cook this as their AI-on-chip next
Bonsai 8B and 1.7B were on Qwen3.5 the benchmark is from a few months ago. However I'll add Qwen3.6 to the benchmark too.
nostrebored 14 hours ago [-]
qwen3.6 starts at 27B
all2 20 hours ago [-]
Tried this on an old 4 core i5 and got about 1tps.
OS: WSL2 on Windows 10
RandyOrion 16 hours ago [-]
Thanks Bonsai team.
Now open weight LLMs/VLMs/LMMs are becoming even larger to the extent that consumer-grade hardware are no longer able to run these models. In contrast, quantization and pruning make the model better at the size-performance pareto and provide people with strictly more possibilities.
syntaxing 1 days ago [-]
I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.
dofm 8 hours ago [-]
It’s still a dense model so all 27B parameters are active. 35B-A3B activates only 3B at any time.
pulse7 1 days ago [-]
Most probably not optimized yet for this model...
wy35 23 hours ago [-]
Entire blog post seems to be AI-generated :/
wmf 23 hours ago [-]
Do you think people who work on AI for a living are not going to use it?
wy35 23 hours ago [-]
Of course not, personally almost all of my code these days is generated.
The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.
arjie 23 hours ago [-]
The text is mostly content-free. Headline + charts are enough for most HN stories.
aesthesia 18 hours ago [-]
IDK, Anthropic's public posts are mostly free of Claude-isms. (Their documentation less so...)
Heliodex 22 hours ago [-]
Nice to see a larger model in their lineup, I've been using Ternary 8B and it seems to get higher TPS than most other similarly sized models on my hardware.
ExxKA 19 hours ago [-]
From an investors perspective, this is truly a paradigm shift - this will kill a whole range of startups in Europe which were packaging privacy and wrapping around large hosted models. There's absolutely no reason to use a "Privacy GPT tm" provider, then I have it all on my own laptop - There is also no need for banks or other regulated institutions to rely on those providers when they can selfhost with this much intelligence on tap.
dofm 19 hours ago [-]
It will when they can get the performance up a bit.
My brief experiments with the ternary version suggest that it broadly meets their claim to be a 27B model that fits in much less RAM, that is for sure. It is about as fast as the underlying Qwen 27B but it gets stuck in reasoning loops quite easily.
xyzsparetimexyz 1 days ago [-]
That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?
Catloafdev 1 days ago [-]
Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.
kroaton 22 hours ago [-]
Which would be very interesting to test, as larger models (such as Deepseek V4 Flash or Qwen 397B) seem to compress better. Their Q2 quants are usable as is, even without the ternary compression.
drob518 22 hours ago [-]
Yep, that’s the question. I asked just that when Bonsai’s first models got released. Super interesting if we can push the parameter count over 100B with 1.125 bit quantization and still keep pretty good performance versus 16-bit 100B models. That’s a definite sweet spot.
Luker88 23 hours ago [-]
Nice!
Do they have plans to bring even bigger models down to ~16GB VRAM so that more consumer hardware might be useful?
verdverm 22 hours ago [-]
bigger quant'd harder is not always better than a model of more modest size and quant
bilsbie 22 hours ago [-]
What data type stores one bit? Does it offer opportunity for more efficient matmuls?
raylad 21 hours ago [-]
Not impressed. It fails the "Jabberwocky" test.
raylad 19 hours ago [-]
This got a downvote and I understand why: because I didn't describe the test, which is to ask it "Please recite Jabberwocky".
This is actually difficult because there are so many invented words in the poem which have extremely low frequencies in the training data. So a model that can do it properly is likely to be very good in other ways. Qwen-3.6-27B can do this until it gets overly quantized.
kamranjon 15 hours ago [-]
Curious why you find that’s a useful test since it seems to be solely measuring training data memorization, something you’d expect to degrade from quantization.
raylad 14 hours ago [-]
[dead]
15 hours ago [-]
lifesucks1 12 hours ago [-]
When Bonsai GLM 5.2 2bit
diddid 20 hours ago [-]
They should do this to GLM 5.2
drob518 23 hours ago [-]
This is going in a good direction.
anshumankmr 12 hours ago [-]
More and more it seems the iPhone 16 was the worst deal in history cause I don't think mine will support the upcoming foundation model from Apple or this one, does it?
rvba 12 hours ago [-]
Does anyone know how to disable the 10 minute timeout in Android studio when using a self hosted model on local machine?
It always auto disconnects.
luciana1u 20 hours ago [-]
phone manufacturers are about to add '27B-capable' right next to '5G' on the spec sheet and honestly it's the first spec bump I've cared about in years
goofy_lemur 16 hours ago [-]
I tried this on M1 Pro today with 16GB ram and it worked!!!
I was using vscode and it seemed to interperet the system prompt right and then started actually inspecting and doing stuff.
Unfortunately the vscode system prompt is 24000 tokens, and I was getting 100 at beginning, 69 by the end of it, but honestly I'm super impressed. Great work team
1
verdverm 23 hours ago [-]
Preliminary analysis via lm-evaluation-harness + vllm
Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope)
Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
kamranjon 15 hours ago [-]
Doesn’t this suggest you aren’t properly running the model?
verdverm 3 hours ago [-]
Its unclear if its in vllm, the eval harness, or the weights. I'm running it the same as other qwen3.6 derived models and it appears to work, but other comments speak of waiting for support in their tool. Could be a chat template or could be legit because I'm using a different quant / bit selection from their hugging face and using their primary model as the comparison point for my huh? They may have put more effort into the flagship model than the 4bit because they are focused on on-device model running.
Software was already complex, now we are adding highly non deterministic elements into the mix
0xbadcafebee 23 hours ago [-]
27B is way more than you need for a phone. Doesn't matter how much you try to compress it, it's the wrong application of the wrong tool. There are already useful tiny models that fit on phones and do basic things really well. Dumb down a big model too much and it becomes worse than a small fine-tuned model.
13 hours ago [-]
latexr 8 hours ago [-]
The meal demo is hilarious.
— Hey, model, see this fake-ass stock photo of a variety of spices, vegetables, and spaghetti? What meal can I make with this?
— Just cook everything.
— I’m a complete noob. I can’t even fathom how to cook those things. Help me!
— Sure sure. First boil the spaghetti completely and drain. Only after that, while it’s getting cold, you need to sauté (good luck knowing what that is if you don’t even know how to cook spaghetti) the garlic and carrots at this specific temperature (good luck figuring out how to do that on a stove). Despite having mentioned the peppers and herbs in the previous message, I’m not going to tell you what to do with those. Just chew them raw or something, I guess.
The demo shows that the model can answer, but the answers are frankly bad. Here’s what you could’ve done instead faster with better results: a web search for “spaghetti carrots peppers”. Don’t even need to add “recipe”.
Presumably you’ve been using the model as you develop it, why not show something real and useful instead of a generic, unrealistic and uninteresting scenario that above all makes it look incompetent? Show something that genuinely surprised you positively.
yieldcrv 17 hours ago [-]
this is really amazing! keep pushing guys, this will coexist in the memory footprint with vision models, and audio models, and other kinds of transformers so we still need memory to work with
you also might single handedly pop the hyperscaler investment and capital projects! that's the whole AI bubble essentially!
The article is about running it on a phone though, and shows an app with their branding running this in text mode on a phone. I'm asking where can I find this app to try what is being demonstrated in this article & video? Appstore only has an image gen app by them and other MLX apps I've tried don't seem to support this model
erelong 1 days ago [-]
I was trying Ornith 9B locally (it's up on Ollama) which claims:
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
Only tried it so much so far; it did a little better than Qwen 9B
liuliu 1 days ago [-]
Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).
gunalx 23 hours ago [-]
3.5 9B can do thinking. Its just disabled by default in its gguf chat template.
liuliu 23 hours ago [-]
It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.
b89kim 21 hours ago [-]
I had no failure mode in 3.6 9B thinking with llama.cpp. After release, there were updates for both model and llama.cpp.
janalsncm 1 days ago [-]
Is that a 1-bit LLM? I don’t understand the connection with this article.
erelong 24 hours ago [-]
Oh, I don't actually know the difference if you want to explain it
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol
verdverm 23 hours ago [-]
Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals
drob518 22 hours ago [-]
I tried it, too, and it got stuck in some loops where it couldn’t recover. Shame, it was promising for the same reason as Bonsai’s models.
verdverm 22 hours ago [-]
check out geyron-9b, I've only used it a bit, but seems better than orinth on vibe evals
huggingface.co/Tivaphraen/Geryon-9B-v1
drob518 22 hours ago [-]
Interesting, thanks. Looking at the model card on Huggingface, it’s combining the Qwythos and Qwable fine tunes from Empero.
verdverm 22 hours ago [-]
yea, it's an experiment in merging multiple fine-tuned models
theLiminator 23 hours ago [-]
This is useful research, but this particular model itself is likely absolutely useless.
oceansweep 23 hours ago [-]
Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.
Onavo 23 hours ago [-]
I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.
mchusma 22 hours ago [-]
I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.
drob518 22 hours ago [-]
Evidence?
runtime_lens 12 hours ago [-]
This feels like a more interesting direction than chasing ever larger models.
For a lot of use cases having a capable model that runs entirely on-device is a much bigger win than squeezing out a few extra benchmark points with a model that lives the cloud.
Imagine I had [a model that was good at math], [model that was good at code], [model that was good at writing], [model that was good at general knowledge]. If I then had [a model that was good at determining whether the user query would be best served by one of those models and sent it to it, leaving the rest of the models inactive], that is the platonic version of what MoE is. In practice, it works a bit differently. It instead basically restricts the number of pathways that can be utilized in solving problems during training, which allows for "expert neuron groupings" to form and "classifier layers" to form earlier on in the structure, but the effect is the same (better, even, since it allows some overlap between structures of experts). It also allows "routing to an expert" to happen token-by-token rather than at the prompt level.
In some workflows you might have 20 different models doing their specialized tasks. Pose detection, hand/eye/face detailers, classifiers, refiners, up scalers, taggers, etc can all use their own models and that’s not even the including the model(s) used for the actual image generation part.
I’m interested to see the optimization when this concept gets applied to other general ai tasks.
PHP/Wordpress code seems OK (better than the Gemma) but it gets stuck in reasoning loops.
Mind you, I am something of a cynic about the underlying 27B dense Qwen; I think the 35B MoE model is often better and it is just so, so much faster.
If these buddies are similarly bad on text, then they definitely don’t get anywhere close to big boys, no matter what the synthetic stats claim upon release.
If you can "hide" different models of 8GB VRAM requirements each that have those specialties and mix and match them for me without having to manage it manually, I'll be impressed. Until then I will keep using my Claude, because "remarkably good _for their size_" models I've tried so far just sucked at trying to use them the way I code at work with Claude.
Worse than Gemma at tool calling? Gemma's already bottom tier at that (at least when there's Qwen to compare to), that would just be unable to do tool calling at all.
* I will say that early on there were a LOT of issues with the chat template, across all engines. I dunno who decided using crappy Jinja templates was a good idea, but clearly it has its limitations. In the latest version of vLLM (0.25) they've ditched the Jinja templates for an in-engine parser and I've seen no issues.
Good evaluation from 2024 https://arxiv.org/pdf/2402.18158
I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.
The 12B QAT model is indeed sort of mindblowing.
The best local agentic coding experience I’ve had so far is Qwen3.6-27B with Pi.
Yes and no. I think where frontier models really blow small models away is in how thorough they are in order to infer your intentions and how best to accomplish them. So you can tell Claude "change this code to make it do X", whereas a Qwen3.6-27B or Gemma4-31B can do the same, but you have to be a lot more thorough, i.e., "change this code to make it do X, but first, let me explain the concept of X as I see it and some notes about things to avoid or pay attention to while you're doing it." So for best success with small models you really need a big toolbelt of skills and MCPs.
I have two main tasks I want to see if I can improve, coding and doc understanding/summary
Hopefully more of the lab releases are trained under QAT so we can all benefit.
But, what I really want is for Google to release bigger Gemma 4 models, particularly a bigger MoE, like a ~70B or ~120B. Gemma 4 is the best all-rounder among the models I can self-host even though I've got a 128GB Strix Halo. A 4-bit QAT version of a 70B MoE would probably be the sweet spot.
A bigger Qwen 3.6 with a 4-bit QAT version would also be welcome, as the prior bigger versions aren't notably better than 3.6 27B, but I guess Qwen is done doing larger open weights models. They did release AgentWorld recently, a post-train of the 3.6 MoE, so they're still doing some open things.
I think I want to see more third-party testing of this ternary Qwen to know if crushing it to 1.56 bits kills it; there are tons of benchmarks of Qwen 3.6 27B, so it's an ideal candidate to figure out what the extreme compression does to it.
How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?
I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem. Anyone care to explain this poor silly fellow some of those points?
[1]: https://github.com/microsoft/fastcontext
[1]: https://huggingface.co/models?sort=trending&search=fastconte...
The system as a whole is meant to support that use case, where each task (ticket in its jargon) can be tackled using a custom workflow that can each use a different agent/llm (so, it should support local LLMs if you have configured your coding agent to use them).
Sidenote: it's still not where I want, but getting there...
It's going amazingly. The orchestrator holds the big knowledge from grilling and also enables me to do more grilling to refine the specs. The trick was to check and re-align after each phase was developed. Also carefully defining each workflow step explicitly otherwise it makes mistakes like trying to self review the code. Also needed to define very explicit contracts. I chose the same surfaces as the matt skill set.
Edit: So life got in the way and I couldn't isolate it. But I have made this project public and committed the dev and code review agents here.
https://github.com/rick2047/meuseum-game-new/tree/main/.open...
The delegate builder is too ugly to share yet. But to be honest just use the normal builder with a frontier model and instruct it to delegate to developer any developer task. That works well.
Of course you have to install the skills [1]
[1] https://github.com/mattpocock/skills
Binary: 9 t/s prompt, 6 t/s generation. Ternary: 0.8 t/s prompt, 0.7 t/s generation. It looks like CPU inference for ternary isn't optimized yet.
https://www.theregister.com/on-prem/2000/08/02/jobs-snubs-at...
They could more or less redistribute Gemma as-is in the developer program; they are unlikely to be troubled by any of the licence terms.
They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?
Whether things are different now I don't know, but Prism would potentially be a good acquisition.
This is not the first Bonsai model from Prism using this technology, and they've also applied this technology to an image model.
The model aspect isn't that significant (even in this news), because it is Qwen, under the hood. The encoding efficiency is.
Prism's technology might be valuable, and their team could well be.
And they very evidently do care a lot about efficient on-device AI; they just don't care about developing frontier cloud models.
I think their strategy is broadly correct, actually; I think there's still a bit of scope for "sit and wait and do it right" here. But acquiring more edge AI tech and edge AI people would potentially be in their interest.
apple’s secrecy agenda has been defeated to an extent by the practicalities of ubiquitous technology?
I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.
Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....
You can also join Discord to communicate with us directly http://discord.gg/prismml
Though this says mainline llama.cpp has their patches for Metal and CPU backends, so maybe it's simply "use current llama.cpp" if you have a Mac or fast enough CPU/memor to use the CPU backend.
The fork runs fine for me. The model gets very notably stuck in a reasoning loop on one of my simple tests, though it might be that it has the same issues with setting reasoning effort high.
On my M1 Max I still think the MoE Qwen 3.6 and Gemma 4 models are the best options. And I am far from convinced that the 35B is actually worse; it gets stuck in reasoning loops much less often than 27B in my experience.
I assume that in addition to the significant memory savings, this should also lead to much simpler matrix multiplication operations? Could models like these run on CPU's efficiently, or does the geometry of the compute mean GPU's are still a better choice?
Matmul is very well parallelized. More, lower power cores will always pay off handsomely.
eg asked it what Signoz is. It reckoned it is a woocommerce/shopify competitor aimed at India market
[1] https://jackson.dev/post/dont-sleep-on-bitnet/
I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.
Even the lowest quality one has 12g/100g.
https://en.wikipedia.org/wiki/Pasta
> Main ingredients: Durum wheat flour, water/eggs
May be going out on a limb here but I think it might be the eggs?
/satire, obviously ;)
For the price, from a protein perspective, it would be cheaper to get it elsewhere.
https://shopfelicettipasta.com/products/monograno-felicetti-...
[0]: https://m.media-amazon.com/images/I/81UbkYz+pZL._SL1500_.jpg
> 25g protein for "spaghetti, carrots, peppers, garlic and herbs"?
Maybe it assumed the pasta was some kind of protein chickpea pasta? =P it definitely seems wrong.
Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.
1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.
is it a float? if so, how many bits is the float?
I've never heard of a bit ever having more than two possible values
Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.
It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.
If that's the case then why not just train at Q2? I guess the counterargument is that then you lose the nice properties of things like the FairyFuse kernels. I wish there were some good discussions of these trade off.
e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing
You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59).
At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.
It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.
The way they do it is packing like the other comment says.
Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.
The biggest issue I've found is absent mindedly opening YouTube or the like that spike ram requirements and freezing the system up. But that's a me problem
> the load on your machine is gonna make doing other stuff while it’s running painful
Is your question about something else?
https://github.com/Liquid4All/antidoom
Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.
When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.
I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.
How do you find the overall experience? And do you have any special sauce or recommendations for going this route?
The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.
At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.
I'm interested in the CPU inference application of these models with things like the FairyFuse kernels.
I've tried trillim previously but was disappointed that i got higher tok/s just with similar sized models through ollama using just Q4_K_M quants.
I see there is bitnet.cpp and litespark-inference. What else should i look at?
However, this is the 5th product post in 2 weeks that proclaims that AI use is shifting, and why [insert tradeoffs] are the perfect fit.
Paradigms shift don't happen in the release announcements.
I suspect this is an AI-ism, making all the release posts sound so paradigmshiftery.
Likely apples / oranges
OS: WSL2 on Windows 10
Now open weight LLMs/VLMs/LMMs are becoming even larger to the extent that consumer-grade hardware are no longer able to run these models. In contrast, quantization and pruning make the model better at the size-performance pareto and provide people with strictly more possibilities.
The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.
My brief experiments with the ternary version suggest that it broadly meets their claim to be a 27B model that fits in much less RAM, that is for sure. It is about as fast as the underlying Qwen 27B but it gets stuck in reasoning loops quite easily.
Do they have plans to bring even bigger models down to ~16GB VRAM so that more consumer hardware might be useful?
This is actually difficult because there are so many invented words in the poem which have extremely low frequencies in the training data. So a model that can do it properly is likely to be very good in other ways. Qwen-3.6-27B can do this until it gets overly quantized.
It always auto disconnects.
I was using vscode and it seemed to interperet the system prompt right and then started actually inspecting and doing stuff.
Unfortunately the vscode system prompt is 24000 tokens, and I was getting 100 at beginning, 69 by the end of it, but honestly I'm super impressed. Great work team 1
Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)
Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)
Software was already complex, now we are adding highly non deterministic elements into the mix
— Hey, model, see this fake-ass stock photo of a variety of spices, vegetables, and spaghetti? What meal can I make with this?
— Just cook everything.
— I’m a complete noob. I can’t even fathom how to cook those things. Help me!
— Sure sure. First boil the spaghetti completely and drain. Only after that, while it’s getting cold, you need to sauté (good luck knowing what that is if you don’t even know how to cook spaghetti) the garlic and carrots at this specific temperature (good luck figuring out how to do that on a stove). Despite having mentioned the peppers and herbs in the previous message, I’m not going to tell you what to do with those. Just chew them raw or something, I guess.
The demo shows that the model can answer, but the answers are frankly bad. Here’s what you could’ve done instead faster with better results: a web search for “spaghetti carrots peppers”. Don’t even need to add “recipe”.
Presumably you’ve been using the model as you develop it, why not show something real and useful instead of a generic, unrealistic and uninteresting scenario that above all makes it look incompetent? Show something that genuinely surprised you positively.
you also might single handedly pop the hyperscaler investment and capital projects! that's the whole AI bubble essentially!
I can just see their image tool on the app store
Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b
The article is about running it on a phone though, and shows an app with their branding running this in text mode on a phone. I'm asking where can I find this app to try what is being demonstrated in this article & video? Appstore only has an image gen app by them and other MLX apps I've tried don't seem to support this model
> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.
https://deep-reinforce.com/ornith_1_0.html
Only tried it so much so far; it did a little better than Qwen 9B
The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?
edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol
huggingface.co/Tivaphraen/Geryon-9B-v1