Bonsai 27B: A 27B-Class model that runs on a phone

What I most want to see it compared to is Gemma 4 12B in the 4-bit QAT version. It's barely bigger than this at just under 7GB, so it also runs on just about any modern device and is remarkably smart for its size. It's an excellent tool user, crazy good vision for its size. I'm still trying to wrap my head around how much is lost with each step down in resolution, but the QAT versions from Google seem to prove the answer is "very little" at four bits.

I need help understanding this. I understood that the magic here is the quantization that allows it to use from 50G to 4G and their process retain most of the intelligence within Pareto limits of gain. And then they proceed to compare with other quantized models as in the level of intelligence per size. It gets to my attention though that the performance in tool calling is mostly affected which is a problem for other small models.

How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?

I am not shtng on them or anything. I'd rather find it amazing, BUT given my limited knowledge, I feel the results miss fair comparison plots and the ones might be misleading. Buy I also reckon it might be me the problem. Anyone care to explain this poor silly fellow some of those points?

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.

[1] https://jackson.dev/post/dont-sleep-on-bitnet/

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models

I've tried a couple in LM Studio - the GGUF one and the MLX one - but neither worked there. Anyone else get them to work? Might be that LM Studio needs to upgrade their llama.cpp or MLX engines first.

Quite weird that heavy quantization method on a dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.

At this point all the different quantization and 'compression' (look at MPO applied to LLMs...) techniques start feeling a bit like snake oil. It's just gut feeling - or scores on benchmarks models are optimized for - what ends up deciding whether a technique is good enough or not.

TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1

The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.

I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.

I still don’t see the point of this. In my testing, it’s worse than Qwen 3.5 4B and even 0.8B.

What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?

The problem, of course, is if you run the UD_Q2 variant (Unsloth) which does only post-training, the number is pretty close to 1-bit model here and the 5% drop in tool-call is significant than it suggests in real-life use cases.

Tried it on Android and got "!!!!!!!!!!!!!" for answers.

For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.

After using a highly capable 2-bit quant as my daily driver for months now, I get pretty excited about releases like this. After a few days for the kinks to be worked out, I’ll be excited to try it.

Nice!

Do they have plans to bring even bigger models down to ~16GB VRAM so that more consumer hardware might be useful?

I've been watching and waiting for this, interested to see how smart it is, as it fits with my interest of getting the smartest possible model running in 10GB of VRAM (RTX3060 that has to drive 2 monitors and run an llm)

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.

Preliminary analysis via lm-evaluation-harness + vllm

    model         | disk | wikitext | gsm8k (match/error)
    baseline      | 55G  | 8.00     | 0.50/0.09
    nvfp4-gptq    | 27G  | 8.25     | 0.47/0.9
    nvfp4a16-gptq | 27G  | 8.11     | 0.53/0.9
    bonsai-4bit   | 19G  | 16.75    | 0/0 (eval bug?)

Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. I'm also not sure what is up with the gsm8k, their benchmarks show something different, but they are using another eval tool. I'll have to add it to my setup. Also why I'm building a setup instead of taking model devs word for benchmarks. (https://github.com/modelscope/evalscope)

Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)

Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)

This is going in a good direction.

Entire blog post seems to be AI-generated :/

That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?

I was trying Ornith 9B locally (it's up on Ollama) which claims:

> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.

https://deep-reinforce.com/ornith_1_0.html

Only tried it so much so far; it did a little better than Qwen 9B

27B is way more than you need for a phone. Doesn't matter how much you try to compress it, it's the wrong application of the wrong tool. There are already useful tiny models that fit on phones and do basic things really well. Dumb down a big model too much and it becomes worse than a small fine-tuned model.

This must be some sort of unpublished app?

I can just see their image tool on the app store

This is useful research, but this particular model itself is likely absolutely useless.

For those curious about their demo, I’m pretty sure it’s using Locally AI (iOS only) that lmstudio acquired/aquihired a couple months ago.

The models themselves are showing up on Hugging Face here: https://huggingface.co/prism-ml/models

Depending on which model you're running, you might need to use the custom forks.

Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1

That's awesome. What's the largest model that could fit onto a single 16gb gpu at 1.125 effects bits per weight?

I was trying Ornith 9B locally (it's up on Ollama) which claims:

> Ornith-1.0-9B, which can be easily deployed on edge devices, matches or exceeds the performance of much larger models such as Gemma 4-31B and Qwen 3.6 35B.

https://deep-reinforce.com/ornith_1_0.html

Only tried it so much so far; it did a little better than Qwen 9B

This must be some sort of unpublished app?

I can just see their image tool on the app store

There's two variants of this (or, as the joke goes, for very big values of bit):

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.

You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.

Yep, that’s the question. I asked just that when Bonsai’s first models got released. Super interesting if we can push the parameter count over 100B with 1.125 bit quantization and still keep pretty good performance versus 16-bit 100B models. That’s a definite sweet spot.

Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

Is that a 1-bit LLM? I don’t understand the connection with this article.

Orinth was not impressive in my vibes testing, I just completed my first grid analysis with real evals on qwen 27b. I can now scale that grid analysis and intend to include the qwen 9b ftunes I've seen going around. They were actually a main motivation because so many claim this or that one is better, but very little in the way of evals

It's a LLM model, not a phone app.

Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b

Didn't work for me in Unsloth, but it will probably be fixed in a day or two when the next batch of updates comes out.

Doing some naive math, the F16 filesize is ~53.8gb, the 1-bit version is ~3.8gb, about 7% of the original size. The F16 size is roughly 2x param count, so that gives a rough ballpark of ~110B.

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.

Depending on which model you're running, you might need to use the custom forks.

Details are here -> https://github.com/PrismML-Eng/Bonsai-demo/blob/main/README....

I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac

Note that 3.5 9B cannot do thinking (while 3.6 27B can, pretty effectively, quite verbosely).

There's two variants of this (or, as the joke goes, for very big values of bit):

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

Is that a 1-bit LLM? I don’t understand the connection with this article.

You also need to pay close attention to BFCLv3 multi-turn result, that helps you to get a sense how frequently these quants will be in a doom loop.

It's a LLM model, not a phone app.

Available on HuggingFace: https://huggingface.co/collections/prism-ml/bonsai-27b

3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

this is a really dumb question, but how is -1 represented?

is it a float? if so, how many bits is the float?

I've never heard of a bit ever having more than two possible values

I tried it, too, and it got stuck in some loops where it couldn’t recover. Shame, it was promising for the same reason as Bonsai’s models.

Oh, I don't actually know the difference if you want to explain it

The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?

edit: I asked AI for the difference and understand a little better, thanks for the heads up to learn the difference between models... I think the thing was, although ornith was created for a specific agentic purpose, it was still outperforming a previous generalist model I had running locally (so in my mind I thought it was still a better local model) - I'd like to try bonsai out if I can figure out how to run it lol

I'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom:

https://github.com/Liquid4All/antidoom

Indeed.

The article is about running it on a phone though, and shows an app with their branding running this in text mode on a phone. I'm asking where can I find this app to try what is being demonstrated in this article & video? Appstore only has an image gen app by them and other MLX apps I've tried don't seem to support this model

I spent quite sometime trying to install their tools and nothing really worked. I used these repos you shared but the dependencies all fail on mac

If you can share details on where it is failing, we'd love to help fix.

You can also join Discord to communicate with us directly http://discord.gg/prismml

I'm curious what kind of results one could get from combining the clever quantization PrismML is doing here with something like LiquidAI's antidoom:

https://github.com/Liquid4All/antidoom

Oh, I don't actually know the difference if you want to explain it

The title says it's 27B grade running on a phone and what I was comparing it to in my mind was a model that runs at 35B grade that could presumably run on a phone "better"?

Indeed.

I tried it, too, and it got stuck in some loops where it couldn’t recover. Shame, it was promising for the same reason as Bonsai’s models.

check out geyron-9b, I've only used it a bit, but seems better than orinth on vibe evals

huggingface.co/Tivaphraen/Geryon-9B-v1

3.5 9B can do thinking. Its just disabled by default in its gguf chat template.

It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.

this is a really dumb question, but how is -1 represented?

is it a float? if so, how many bits is the float?

I've never heard of a bit ever having more than two possible values

It appears they are using Q2_0 in llama.cpp, which is 2 bits per weight + 1 float16 scale per group of 64 weights. This is inefficient in two ways: one bit pattern is wasted on each weight, since ternary weights only use {-1,0,1} and Q2_0 allows {-1,0,1,2}; and their group size is 128 weights, so the scale will be stored twice in two groups of 64 instead of stored only once in one group of 128.

Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.

It's possible to pack 5 trits into a byte, but the unpacking is not very efficient. Another recent idea is to add the constraint that exactly one weight in each group of four be zero, which gives exactly 32 possible states, so it fits in 5 bits.

> never heard of a bit ever having more than two possible values

It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.

packing multiple trits together

e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.

If you can share details on where it is failing, we'd love to help fix.

You can also join Discord to communicate with us directly http://discord.gg/prismml

I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!

check out geyron-9b, I've only used it a bit, but seems better than orinth on vibe evals

huggingface.co/Tivaphraen/Geryon-9B-v1

It is disabled because it doesn't work :) Try it and see the doom loop it gets itself in.

Their fork corrects the second inefficiency by using a group size of 128, but still uses 2-bit weights AFAICT.

> never heard of a bit ever having more than two possible values

It's not represented by a "bit", binary digit with value of 0 or 1; but with a "trit", ternary digit with value of {−1, 0, +1}.

I did not know you guys would be watching. For sure. Let me do that tomorrow when I turn it on again :) I am happy to see the message. Thanks!

It’s still a bit with only two possible values. But they add a scaling factor to a group of them (128 for example) which when you factor in, results in a fractional number of bits per parameter.

I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer.

The way they do it is packing like the other comment says.

Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.

packing multiple trits together

e.g. 5 trits (243 states) into a byte gives 1.6 bits per trit: https://compilade.net/blog/ternary-packing

It's impressive how close to optimal this is.

You can beat the efficiency of 5 trits in 8 bits (1.6) with as few as 17 trits in 27 bits (~1.588), but once you account for rounding up to a whole number of bytes for practical reasons, then beating the efficiency requires going to at least 111 trits in 176 bits (~1.586), or perhaps more practically for fast unpacking, 161 trits in 256 bits (~1.59).

At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.

How does this model compare to a recent 4G model? How do we know it retained intelligence from the parent rather then being fine tuned for the benchmarks?

Quite weird that heavy quantization method on a dense model gives better results than slightly quantized MoE models like 35B-A3B from Google.

Awesome! I've been waiting for them to start scaling ternary models for over a year[1]. Excited to try it out, typical Qwen 27B is too heavy for me to run on my local hardware at reasonable speeds.

[1] https://jackson.dev/post/dont-sleep-on-bitnet/

This is going in a good direction.

Preliminary analysis via lm-evaluation-harness + vllm

    model         | disk | wikitext | gsm8k (match/error)
    baseline      | 55G  | 8.00     | 0.50/0.09
    nvfp4-gptq    | 27G  | 8.25     | 0.47/0.9
    nvfp4a16-gptq | 27G  | 8.11     | 0.53/0.9
    bonsai-4bit   | 19G  | 16.75    | 0/0 (eval bug?)

Code if you'd like to reproduce or try other test sets: https://github.com/verdverm/quantr (lightly tuned to a single oem spark, probably possible in 32-48G)

Good paper to understand the effects of quant regimes across model families and tasks: https://arxiv.org/abs/2402.18158 (Evaluating Quantized Large Language Models - 2024 ICML)

I believe the scaling comes in later, to turn the 1 and -1 into large numbers that may or may not activate the next layer.

The way they do it is packing like the other comment says.

Each byte represents 5 trinary values instead of 8 binary, and there is a little bit of waste.

It's impressive how close to optimal this is.

At that level, even if you have, say, 27B trits, the more efficient encodings would save something like 38-45MB (theoretical limit ~48MB), likely at the cost of some slowdown.

from what I understand prismml isn’t doing a quant like normal models where you take a model trained at fp16 and then chop off some bits to reduce vram, but rather they’re training the model natively with 1 bit weights. It’s explained more in the article. They’re also doing some other tricks like a fp16 weight per block of 128 1bit weights to get some more data out of 1 bit weights

Same here. I’m excited to have a model that might be usable on a 16 GB laptop.

4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn.

Good evaluation from 2024 https://arxiv.org/pdf/2402.18158

I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.

If you read to the bottom of the page, it says they're funded by a few people, and one of them is Samsung. I'm betting Samsung wants to be able to ship a capable AI system on a future model of their phone so they can compete with Apple.

Tried it on Android and got "!!!!!!!!!!!!!" for answers.

The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.

That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.

When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.

start saving your money.

Entire blog post seems to be AI-generated :/

Do you think people who work on AI for a living are not going to use it?

I still don’t see the point of this. In my testing, it’s worse than Qwen 3.5 4B and even 0.8B.

It is still worth experimenting. Although we do need independent evaluation of these models instead of labs posting such biased and skewed results.

Apparently Apple is "in talks" with the PrismML: https://www.cnbc.com/2026/07/14/apple-prismml-ai-compression...

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

Nice!

Do they have plans to bring even bigger models down to ~16GB VRAM so that more consumer hardware might be useful?

bigger quant'd harder is not always better than a model of more modest size and quant

What model? And what hardware do you run it on?

I find these style of models are great, but fail hard, and fail randomly. I'd be hesitant to use it for a daily driver, but I'm using dual 3060s, so it's not like I'm quantizing a frontier model here.

How do you find the overall experience? And do you have any special sauce or recommendations for going this route?

What have you been using?

This is useful research, but this particular model itself is likely absolutely useless.

Why make this comment without having tried it first? It very clearly is not useless and performs a lot better than one might expect. I am currently waiting to do more benchmarks of it in comparison to the full weight model, but it seems promising/better than Mistral Nemo at a lower file size.

The KV-cache memory usage also seems remarkably frugal, even at the full context length. That could make this model particularly useful in multi-agent coding workflows.

I wish KV-cache memory usage and related optimizations were discussed more clearly in new model announcements and demos.

quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.

I don’t know if the llama cpp implementation is wonky (and only supports the binary version) but it’s a lot slower than 35B-A3B @ Q4_KM + MTP with CPU offloading.

Most probably not optimized yet for this model...

Same here. I’m excited to have a model that might be usable on a 16 GB laptop.

quanting kv cache hurts attention / recall, and long-form tasks by proxy. Model families and sizes have different tolerances to quant ting different parts of the model, same for intended tasks.

That's what happens when you quant too hard. I'm working on quant strats and evals for the same underlying qwen 27b models.

When I saw 27b on a phone, I thought not fitting, big phone, or aggressive quant. NVFP4 still takes 27G before KV cache.

It is still worth experimenting. Although we do need independent evaluation of these models instead of labs posting such biased and skewed results.

bigger quant'd harder is not always better than a model of more modest size and quant

Good evaluation from 2024 https://arxiv.org/pdf/2402.18158

I'm currently working towards an updated version (not an og author), curious if others are aware of similar surveys, as I have yet to do a real lit search.

The key point here, I think, is not the 4-bit but the QAT — the model is trained with the objective of losing the least at 4-bit quantiZation (I am assuming it is literally about assigning numbers that quantize better).

The 12B QAT model is indeed sort of mindblowing.

What have you been using?

Do you think people who work on AI for a living are not going to use it?

The qwen models really seem to have this as a failure mode, its so annoying having a proper trace ending up in !!!!!! Garbage.

What model? And what hardware do you run it on?

How do you find the overall experience? And do you have any special sauce or recommendations for going this route?

start saving your money.

Notably, PrismML CEO Babak Hassibi told CNBC this, so it’s either (1) bullshit, or (2) he just ended any chance of a relationship by leaking news of the talks.

Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.

DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4

The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.

Of course not, personally almost all of my code these days is generated.

The LLM style of writing is just very distracting to read. “It unlocks X”, “Y changes the equation”, and why is there always something shifting? Makes my eyes glaze over in an otherwise interesting post.

I think what OP means is that the "minimum viable product" for a daily use LLM is probably somewhere around e.g. GPT 4o's level of intelligence (YMMV). Below a certain threshold, you are better off using specialized machine learning models rather than general purpose LLMs. It's very difficult to get that level of intelligence fully local on a mobile device without streaming to the cloud.

Wait in a regular sentence, what is the probability of "!!!" being followed by "!"?

Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.

I’m using DeepSeek V4 Flash on 128gb mbp - it’s a bit different using a 200b+ param model. It’s MoE so performance is acceptable. It will still malform a tool call every now and then, but the capabilities are so far ahead anything else that the majority of the time it works really well and solves really complex problems.

Or watch and wait as models get denser

The 2000 ATI leak showed up here a few weeks ago.

https://www.theregister.com/on-prem/2000/08/02/jobs-snubs-at...

Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).

either way, maybe a portent for the times.

apple’s secrecy agenda has been defeated to an extent by the practicalities of ubiquitous technology?

Most probably not optimized yet for this model...

The 12B QAT model is indeed sort of mindblowing.

I haven't dug into QAT deeply, better recovery is my understanding as well, and also that it is out of reach for most people because you have to train a model to back prop errors based on estimated error under quant.

Hopefully more of the lab releases are trained under QAT so we can all benefit.

Agreed, and the prevailing wisdom now seems to be that unless you can release a truly frontier model, you might as well release yours as open source to undercut your competition.

Or watch and wait as models get denser

DeepSeek V4 Flash with DwarfStar: https://github.com/antirez/ds4

The 2 bit quants are really good. I have a lot of memory so I can squeeze it all in at ~80gb.

I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.

either way, maybe a portent for the times.

apple’s secrecy agenda has been defeated to an extent by the practicalities of ubiquitous technology?

The 2000 ATI leak showed up here a few weeks ago.

https://www.theregister.com/on-prem/2000/08/02/jobs-snubs-at...

Apple would punish him severely unless they cleared it in advance, it might be to their advantage for some reason (negotiating with Google for Gemma rights? idk).

Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.

Wait in a regular sentence, what is the probability of "!!!" being followed by "!"?

Sounds like the model is not following a proper probabilistic choice here, so maybe more a programming error than a model training error.

After the third !, the probability of a fourth probably skyrockets =)

Hopefully more of the lab releases are trained under QAT so we can all benefit.

Of course not, personally almost all of my code these days is generated.

The text is mostly content-free. Headline + charts are enough for most HN stories.

I do think this would be interesting if they made these easy to finetune, as I do think this level of intelligence is likely sufficient for many applications and could be extremely cheap to run.

After the third !, the probability of a fourth probably skyrockets =)

Apple is too desperate to be making demands. You confuse the Apple of yesterday and of today. Things change fast and things have changed.

You’re right, Apple only has 68 billion dollars in cash, up 40% since last year. Definitely on their last legs.

> Apple is too desperate to be making demands

They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

The text is mostly content-free. Headline + charts are enough for most HN stories.

> Apple is too desperate to be making demands

They don't give a F about AI or any new AI model that was announced this morning. Wasn't there news a while ago about them buying Perplexity?

You’re right, Apple only has 68 billion dollars in cash, up 40% since last year. Definitely on their last legs.

Free cash? Crazy. I think I can live with the interest of that in the bank.

I remember the infographics of 5-10 years ago. They had 200 billion. What happened? Sinking ship?

Free cash? Crazy. I think I can live with the interest of that in the bank.

I remember the infographics of 5-10 years ago. They had 200 billion. What happened? Sinking ship?

They never had $200 billion in cash in any quarter. Not at least since 2014. Besides, how much cash to hold is an executive decision. If the cash balance went down, it doesn't mean their business is not doing well. Cash can be used for Capex, like what the hyperscalers are doing right now. It can be used for buying back equity.

Today, we're announcing Bonsai 27B, based on Qwen3.6 27B, the new multimodal flagship of the Bonsai family and the first model of its capability class to run on a phone.

Our earlier releases proved that models with 1-bit and ternary weights could produce commercially useful language models. Bonsai 27B extends that frontier to a new capability tier: multi-step reasoning, structured tool calls, vision tasks, and computer-use agentic loops that stay coherent across many steps. Until today, deploying that tier locally has been impractical for a concrete reason: a 27B model occupies roughly 54GB in 16-bit precision, and even a good 4-bit build, at 18GB, is too large for a phone and for most laptops.

Bonsai 27B changes that. It comes in two variants:

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight. At 5.9 GB, it is the quality-oriented variant: it runs on an everyday laptop with the full reasoning, tool-calling, and agentic capability.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight. At 3.9 GB, it is the footprint-oriented variant, which fits within the memory budget of an iPhone 17 Pro, bringing a 27B-class model onto a phone for the first time.

As with every Bonsai release, the low-bit representation runs end to end across the language network, embeddings, attention, MLPs, and the LM head, with no higher-precision escape hatches. Both variants are multimodal, with the vision tower shipping in a compact 4-bit form so on-device workflows can see screenshots, documents, and camera input, not just text. Bonsai 27B carries a full 262K-token context, and supports speculative-decoding, compounding the speed with lossless draft-and-verify acceleration. Everything is available today under the Apache 2.0 License.

Retaining the intelligence

Across a 15-benchmark suite spanning knowledge, reasoning, math, coding, instruction following, tool calling, and vision (evaluated in thinking mode, where the model's full reasoning is exercised) Ternary Bonsai 27B retains 95% of the full-precision baseline, and 1-bit Bonsai 27B retains 90%.

| Category (benchmarks) | Qwen 3.6
27B | Ternary Bonsai
27B | 1-bit Bonsai

27B | | --- | --- | --- | --- | | Math (GSM8K, MATH-500, AIME25, AIME26) | 95.3 | 93.4 | 91.7 | | Coding (HumanEval+, MBPP+, LiveCodeBench) | 88.7 | 86.0 | 81.9 | | Agentic and Tool-calling (BFCL v3, TauBench) | 80.0 | 74.0 | 66.0 | | Instruction following (IFEval, IFBench) | 78.4 | 71.8 | 65.8 | | Knowledge / STEM (MMLU-Redux, MuSR) | 83.1 | 77.0 | 73.4 | | Vision (MMMU Pro, OCRBench) | 72.6 | 65.2 | 59.6 | | Overall (15 benchmarks) | 85.0 | 80.5 | 76.1 |

‍_Fig I: Benchmark scores of Bonsai 27B (thinking mode) against the full-precision baseline. Full per-benchmark results are in the whitepaper._‍

Read the table by capability and the story is sharper than the averages: math and coding are nearly untouched, tool calling stays within a few points of full precision - exactly the capabilities that agentic workloads depend on. For comparison, the most aggressive conventional low-bit build of the same base model scores significantly lower than 1-bit Bonsai 27B while occupying 2.5x more memory.

This is the same Pareto shift we demonstrated with our earlier language and image models, now at 27B scale: 27B-class capability at a footprint smaller than a full-precision 2B model. By intelligence density — the measure we introduced with 1-bit Bonsai 8B — 1-bit Bonsai 27B delivers 0.53 per GB: more than 10x the full-precision baseline, and roughly 2.7x the best low-bit alternative available.

Fig II: Intelligence density (per GB) of Bonsai 27B compared to other models in the same parameter class.

Why this is an important paradigm shift

The most valuable AI workloads are shifting from single responses to sustained work: assistants that operate real tools, workflows that run unattended before returning a result, and research that synthesizes dozens of documents. This shift changes the shape of the workload — an agent doesn't make one model call, it makes hundreds, each one carrying context, producing structured output, and feeding the next.

Cloud APIs will remain the right choice for many products. But for agentic workloads, cloud-only execution imposes structural constraints: every step is a remote request, per-token cost accumulates with every iteration, and every plan, tool call, and intermediate result crosses the network including the user's private files, screen, and data.

Carousel I: End-to-end agentic workflow with Hermes, powered by our Ternary Bonsai 27B model on NVIDIA GeForce RTX 5090.

Local execution changes the equation. When a model capable of sustained agentic work fits on the device, the agent can live inside the product: the marginal cost of a hundred-step loop is zero, and the user's data never leaves the machine. Entire categories open up — persistent on-device agents, assistants that work offline, assistants that reason over private local data by construction. What has been missing is a model small enough to deploy this way and capable enough to trust with the work. Bonsai 27B is that model.

It also unlocks a new system architecture: hybrid deployments that route non-frontier and privacy-sensitive tasks to a capable local model and reserve frontier cloud models for the hardest steps — collapsing the cost-per-task of agentic systems.

Bonsai 27B reaches up to 163 tok/s in 1-bit and 134 tok/s in Ternary on an NVIDIA GeForce RTX 5090. On an M5 Max, it reaches up to 87 tok/s in 1-bit and 58 tok/s in Ternary.

Fitting a phone is a stricter gate than storage numbers suggest. A phone never exposes its full memory to an app - a 12 GB iPhone offers about 6 GB for the model to use on-device, and the model shares that budget with its KV cache and activations. No conventional build of a 27B model comes close to clearing it. At about 4 GB, 1-bit Bonsai 27B is the first to pass through with room to work.

That constraint is why the family ships two deliberate operating points, specifically keeping that in mind: ternary for laptop-class quality, 1-bit for phone-class footprint.

Demo II: Multimodal agentic use-cases powered by 1-Bit Bonsai 27B on an iPhone 17 Pro Max (Demo Mode: Cached & Prefilled Image Context)

The frontier keeps moving

Every Bonsai release has moved the intelligence-per-gigabyte frontier left, and Bonsai 27B moves it past a practical threshold: the full capability set of a modern model with thinking, multimodal understanding, vision, reliable tool use, now fits on the devices people already own.

We believe intelligence density will be one of the defining axes of the next stage of AI progress. Raw capability determines what a model can do; density determines where it can do it. Every leftward shift of the frontier expands the set of devices, products, and environments where advanced AI can operate and changes the economics of every deployment surface it touches, from phones to single-GPU serving. The methodology behind Bonsai is architecture-agnostic, and the frontier will keep moving: larger models and new architectures are already in progress.

Early computers filled rooms; today they live in our pockets. Intelligence is making the same journey, and Bonsai 27B is its largest step yet.
**
Platform Coverage**

Bonsai 27B runs natively on Apple devices (Mac, iPhone, iPad) via MLX and on NVIDIA GPUs via CUDA, through custom low-bit kernels built for its hybrid-attention architecture. Model weights are available today under the Apache 2.0 License. With this release, we’re offering a free, limited-time developer preview API so developers can easily try our model.

Full technical details of our compression, evaluation, and benchmarking processes are available in our whitepaper.

Join Us

PrismML emerged from a team of Caltech researchers and was founded with support from Khosla Ventures, Cerberus, and Google, with continuing support from Samsung. We've spent years tackling one of the field's hardest problems: compressing neural networks without sacrificing their reasoning ability.

If you want to help build the next generation of state-of-the-art AI, we'd love to hear from you. Check out our careers page.

Hacker Times