Same here! Qwen3.6-35B-A3B is the only local model I've found that runs reasonably on my iGPU. Looks like me and and my noisily-wheezing laptop will be sitting out this upgrade.

I'm hoping too that they'll put out some MoE variants.

Qwen3.5:122b:a10b can run about twice as fast as this 27b dense model.

Considering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.

I wish each quant was benchmarked on the same tests as the original network so we could compare their performance

That's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.

> Beats Opus 4.7 Max

I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage.

Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.

> I'd gladly take A5B or A8B or even A10B as a sort of middle ground.

Whats up with focusing on the active param count? Do yall fiddle with the weights or something?

Is there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?

I could kiss you right now

> I've settled on GLM-5.3 (formerly Deepseek v4 pro 0813) for architecting

Dude, GLM-5.3 released _today_.

The phrasing "I've settled on" is incorrect for this context.

Go for a slightly more quantised version, and experiment with different MTP settings. I find that MLX versions are marginally faster on my 64GB M1 Max, but I usually use Unsloth's GGUFs via llama.cpp as there's a much greater range of quants available and I prefer llama.cpp. MTP sometimes also helps a little, but I suspect it's less helpful on my system than others.

Unsloth: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

This might work for you, but I didn't get on very well with MTPLX when I tried it a while back; YMMV: https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized...

Check out MTPLX and limit your context size.

Unfortunately, that chip just doesn't really have the memory bandwidth to run this (or nearly any) model at acceptable speeds. I have the exact same chip (M4 Max 128GB) and I've been trying to optimize a completely purpose-built implementation with Fable and this is just not possible. Even if you could reach the full 576GB/s, it's just physically impossible to exceed these numbers with the model's architecture:

2 bpw - ~85.7t/s

3 bpw - ~58.0t/s

4 bpw - ~43.9t/s

6 bpw - ~29.5t/s

8 bpw - ~22.2t/s

16 bpw - ~11.2t/s

without cheating. You'd have to exclude layers, skip operations, etc. basically do stuff the model wasn't trained for. And speed collapses so fast with context that even 2 bpw would be looking at ~37.6t/s after just 128K tokens.

MTP only improves the situation by up to 2x in the ideal case, while drastically reducing the performance floor. While optimizing a 9B model on this hardware, I've found that the GPU just doesn't have enough FLOPS to handle speculating more than one or two tokens ahead on a single stream, regardless of quant level, simply because of the arithmetic cost of the forward pass. The 27B model would be even more expensive than that, potentially such that it's already bottlenecked by the GPU itself rather than memory.

I wouldn't get my hopes up for the 35B-A3B either. Not only is it reportedly much less intelligent, but I hit the exact same 85t/s wall in practice (again with highly specialized inference).

Without speculation I can reach around 120t/s on Qwen3.5-9B and with n-gram speculation (not even MTP; this derivative didn't come with one) around about 150t/s on average. This is on the very very edge of what I'd consider acceptable for me to even consider using such a compact model. YMMV due to the silicon lottery but the situation isn't good.

With spec decode should easily get to >100tps

on my dual 3090s qwen 3.5 27b was running at around 110tps using the config from https://github.com/noonghunna/club-3090

make that 200tps on a single 5090, 4x faster than opus https://x.com/radixark/status/2088285681131110446

devs about to get handed a two 5090 box each and told to max that out

can you tell me ideas of usecases of 9 or 10B language models ? I cant find any usecases other than training a lora on them to give good bash commands for example

strong visual reasoning apparently, which is nice. still lacking native audio however. hoping for more companies to embrace the spirit of something like `gemma-4-12b-qat` for actual multi-modality (text, image, video, audio).

There is a down to earth guy on YT that performs a series of tests against LLMs running on non-god-tier commodity hardware. He will likely be testing this soon enough.

https://www.youtube.com/@lukesdevlab

I don't know if that is what you are looking for or not and as always your experiences may be different.

> They do not beat opus on real-world usage

We have an internal eval that measures performance on tasks for a handful of embedded systems repos for our mmWave radios (mostly Rust, some C for microcontroller stuff). Qwen3.6-27B scores only 4% lower for pass@1, n=250 compared to Opus-4.8.

For the labeled dataset, the average PR size they're being measured against is around 1.5k SLOC.

This is very much "real-world usage" for us. The sort of change sets that come in daily/weekly and are solving non-trivial issues in the respective codebases.

As is usually the case, the most broad claims from both the labs and from the consequent pushback are talking past each other.

You can run these on CPUs at a somewhat reasonable speed.

Depends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus

Run the unsloth if you are using llama.cpp (GGUF)

Run the one you linked if you are running vllm (safetensors)

Unsloth one is gguf for llama.cpp (and some other on-device engines).

So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.

> ...but no. They do not beat opus on real-world usage.

I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.

hence the "former deepseek v4 pro". I tried it out this morning and have had no complaints. I already liked glm 5.2

A wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".

Only useful benchmarks are those you (in particular) don't have access to.

> but then we just need meaningful benchmarks that clearly show that!

That's the rub. AI benchmarks are IMO, by and large totally unreliable. We think of them as similar to traditional benchmarks of deterministic processes where the number of variables is low. But they're anything but that. Non-deterministic processes with an astounding number of variables and fuzzy acceptance criteria.

It leads to results like these, where if you take it at face value, the only conclusion you can draw is "wow Anthropic must be stupid if Opus takes 1T parameters to do what Qwen can do in 27B."

The sentence still doesn't make sense, because "settled on" implies a long testing phase with a verdict eventually emerging out of that.

What you're currently doing is "testing out"

This is why they've been making bank on the secondary market. They can retire now.

I don’t think so for two reasons:

1. They still have extreme revenue growth (and despite what HN says, good gross margins supposedly).

2. People will pay a premium for the best model. So far, we see catchup but no leapfrogging.

The real target for Anthropic, OpenAI and tons of others are large enterprises and workflows. I think they will do all right if they execute well.

I mean, that was obvious from the word "go".

Even Zuckerberg isn't as deluded to think that there's money in constantly releasing closed models and hypserscaling their deployment, which is why we got muse-glimmer 30b last week. I have been playing with it on a 32gb MBP w/ M2 Pro. If prosumer-grade hardware from almost three years ago can now offer a novice a way to host a possibly serviceable software development agent, then there's little incentive to pay out for LLMs like Anthropic's or OpenAI's over the long run.

IMO AI companies are selling the infrastructure and easier access to the models, beyond competing to get the smarter model. We can run Qwen locally, but at what cost? A good enough GPU for running 27B is more than 4,000 USD where I live (Brazil). Even if I were to spend 100s of hours in credits I would never reach a point where the 4000 USD investment pays off. Unless I'm doing it for the sport, building my own LLM rig is not financially viable.

This is without saying that company clients of Anthropic or OpenAI will not have the upfront money to get a RTX 3090 for each and every employee. Much better to sign a deal with the LLM providers and get cheaper token price with a sales person.

Another way of rephrasing that though is that the public (including large pension funds, retirees, etc.) just dodged a bullet and aren't left holding the bag.

Nevertheless, I doubt anyone's going to emerge unscathed when the valuations start falling though. Vast sections of the American stock market will be wiped out yet and with it the global economy.

Trading blows in some benchmarks is a bit exaggerated. If you try the model, `xhigh` is basically feels like the `max` mode (i.e. massive thinker and extremely presistent), and the amount of world knowledge and intent understanding is nowhere close to an Opus class model even from 6mo ago.

It's still very useful, and it'll probably displace a good bit of API spend; but it's not really "trading blows with SOTA from just a half year ago". A bit overblown on the Anthropic/OpenAI has missed the window I reckon.

Also, on the open weight frontier side, Kimi K3 is pretty expensive, and Deepseek V4 Pro/Flash is getting a little less juicy with price increases.

I can get 128k context on a 5070ti with 16 GB of VRAM (using the Unsloth 2-bit quant[0]). This is via a .bat file on Windows 11. I'm getting about 50-60 tokens/second and the quality is much higher than Qwen 3.6 27B. I'm using llama.cpp[1]:

    llama-server.exe ^
        -m "Qwen3.8-27B-UD-Q2_K_XL.gguf" ^
        --presence-penalty 0.0 ^
        --repeat-penalty 1.0 ^
        --fit-ctx 128000 ^
        -ctk q4 0 ^
        -ctv q4 0 ^
        --reasoning-budget -1 ^
        --chat-template-kwargs "{\"preserve thinking\": true}" ^
        --host 0.0.0.0 ^
        --port 8033

[0] https://huggingface.co/unsloth/Qwen3.8-27B-GGUF (UD-Q2_K_XL)

[1] https://github.com/ggml-org/llama.cpp/releases

Instructions if you want to do the same:

1. download two files llama-b10434-bin-win-cuda-13.3-x64.zip and cudart-llama-bin-win-cuda-13.3-x64.zip from that llama.cpp Github releases page, and extract both into the same folder.

2. Download the Qwen3.8-27B-UD-Q2_K_XL.gguf file from huggingface and put it into the same folder beside the `llama-server.exe`.

3. Create a file named "RUN_QWEN_3.8.bat" next to `llama-server.exe` and put the text above into that bat file. Double-click the bat file, then open http://localhost:8033 in your browser to see a chat window.

You can use it with any agents by pointing them at http://localhost:8033/v1 which is a working OpenAI compatible endpoint (it doesn't use a token, if you give one it's ignored).

Congratulations, you're now running Qwen 3.8 27B.

Note: I built the computer in question for playing games, yes it needed to be Windows 11 for anticheat reasons to play games with family, I didn't want to dual boot so here I am. I figure I should share instructions for folks who may also have a Windows PC around for such purposes. Specs for this are AMD 9800X3D, 32GB of system RAM, RTX 5070Ti 16GB

Do you mean 3.6 27b? Because qwen 3.7 didn't have an open weight version

Those are all for MoE models. And I prefer measuring my tokens in t/s instead of s/t

IME this is a strong/reliable model smell, you typically see smarter and less benchmaxxed models' thinking traces spending more time exploring the solution space, and benchmaxxed models more time trying to refine/decide on the response contents. It has always been a big problem with qwen.

In general it's kind of a benchmaxxing/distillation (I don't mean that pejoratively here, read on) artifact where test time compute's purpose is to refine/zero-in on a particular input:output pair. Basically they're trying to "remember"/re-derive the answer rather than arrive at it deductively - it gets comical/absurd when you see it expending a huge amount of tokens trying to figure out the answer to some trivial question or response to some input, as if it were a trick question or the choice of wording was of the utmost importance. But it's not a bad thing when the output matters or the task is hard: basically the model has been trained to respond/act like a much smarter model and it's probably better for it to overgeneralize that behavior.

Also, IME Gemma and most other local models that support thinking tend to have the same problem, and it's partly only a "problem" because they give you access to the thinking tokens themselves (which you don't in many cases see when working with frontier american models) and you can actually see how they're getting spent. For the local or bulk (runnin on owned or rented hardware) use case where you are paying in time/watts rather than purely by the token, IMO it's a good idea to just look at the task success rate / time per task and not worry about the raw thinking token count.

“Thinking” is just a guiding methodology to help iterations (between the initial prompt, results, and a mixture of harness back and forth to the LLM) converge on something sane in a massive parameter space.

I like to think of it much like (as a common example most people can relate to) the Newton-Rhapson method for finding roots of a (mathematic) function. Your initial prompt runs, then the ‘harness’ kicks in using whatever methodologies are behind them to iterate on that prompt (back and forth with the model, occasionally with the user to get better guidance) and refine the outputs to hopefully converge back to some sensible output or actions the user was initially looking for.

So you’re hoping for an LLM that sort of ‘zero shots’ or needs minimal iterations from a prompt to give usable results. I find from my anecdata it varies across models and what I’m trying to get it to converge on. I tend to prefer models to not zero shot attempt because they tend to not do great, I want them to get feedback often to let me push them down the route of convergence in spaces I already understand well, meanwhile I like them to explore and give me new paths in spaces I’m not too familiar with.

That’s really what all that “second guessing” is, it’s making sure you’re following a sane path in a massive parameter space of an ambiguously defined problem. Imagine if in Newton’s method you checked the slope and it didn’t decrease from the last iteration and you just say “screw it let’s keep trying that direction.” LLMs and their harnesses tend not to have that base assumption like iteration on decreasing slopes to guide them closer to convergence, it’s a lot messier.

You can add a thinking budget thats not much effort in llamacpp. You can align the cut off message with your agent instructions.

What you describe is a engineering harness problem.

If you, and i mean the royal you, actually read tge thinking traces you can see and figure out where its stuck

This means an effective harness would observe when the model is overthinking and step in with reasonable redirection, like increasing logging.

Llamacpp can set reasoning budget and message per reauest, so it can be dynamic.

Your complaint is "skill issue" based and will be resolved by people who do something ither than vibe code react demos.

This is why the "reduce thinking" approach of BottleCapAI is both quite interesting and very useful in practice:

https://huggingface.co/bottlecapai/ThinkingCap-Qwen3.6-27B

Worth distinguishing knowledge/task benchmarks from IF / agentic. It doesn't seem out of the question that you can have a small model thats generally good at instruction following and long-horizon agentic, as usually in those cases any requisite knowledge is in the context.

Most of the benchmark improvements afaict are in agentic and instruction following benchmarks.

gemma4 can't hold a candle to 3.6

Depends on your tooling and quant? I grabbed the unsloth Q3 and it works out if the box in opencode. I had issues with OpenWebUI with a random 3.6 A3B.

With default config via Ollama and 65k context I get 50tps on a 3090.

For those who don't wanna open twitter: https://xcancel.com/ErdalToprak/status/2088299678085308761?s...

Thanks this bumped my agent success rate from 67% to 92.5% (!!!)

As was the case with GLM 5.3, it seems that there is still much juice to be squeezed from post-training

Does anyone know how to get this working with Claude Code via llama-server? I'm getting a jinja template error about the system prompt not being the first message.

Have you tried to put the nvidia card into "compute-only mode" since you're not using it for display?

I have a B70, what llama options are you using and what performance are you seeing?

Running it on 2x3060 now. Works pretty well but VRAM is tight. 4bit quants. 1x128k context, 8bit KV, MTP on.

Lol, "burn money on apple hardware instead!"

Use 3.6 27b as a daily driver for months with charmbracelet crush. Gemma 26b-A3b is not even remotely comparable in terms of coding for me. YMMV depending on how you work, what harness you use, etc I suppose.

Oh, can it work with the /v1/completions/ auto-complete endpoint?

Reduce or turn off thinking:

https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates

Was recently thinking about doing something similar, do you basically just have the qwens describe what they see for the flashes?

Was considering adding a LoRa/vision head to Flash, but seems like it could take a while to get it right.

If DSv4 Flash was multimodal, I’d probably be done model shopping for a while

I absolutely love this comment. I wished there was a website where people would post their working command lines as well as what hardware they are using to run that stuff on + tokens / sec prefill + gen.

What kind of things you only get with a larger model?

Any chance I could run it on a GeForce RTX 5060 Ti 16 GB, (64 gb ram)

You don't need $10k to run DeepSeek, I run it on a $1000 system.

To add for me (and claude assisted but tuned myself), here is for 3060 12gb + 5950x 128 gb ram without image support: https://pastebin.com/HsAUBGSY

Here is the chat template I used (and renamed to qwen.jinja): https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates/r...

For image support, drop to NGL 24 and add:

  --mmproj-url https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/resolve/main/mmproj-F16.gguf \

  --no-mmproj-offload \

  --image-min-tokens 1024 \

And drop the --no-mmproj

I wonder how this would stack up against 4x RTX 3060, assuming you have the physical room for them.

Thanks for posting! Have you had any success with running without kv cache quantization? Is there a noticeable difference in quality without any? I would assume that would eat into context but 170k is pretty generous!

I think you've been drinking the "LLMs only improve by adding parameter counts" that SOTA labs are selling VCs to build data centers so they can keep eating through cash to their own benefits.

To the countrary, the reason Chinese models are excelling in the smaller area is because there's tons of fat in closed source models because of the crazy cash being thrown around.

There absolutely is space to improve intelligence and capabilities without lathering on more and more parameters.

> --cache-type-k q8_0 --cache-type-v q8_0

In my tests, even Q8 quantization for the KV cache comes with notable drops in performance for longer tasks. It does provide more context length in limited RAM budgets, but the longer context tasks are where KV quantization starts to show problems. It’s basically unnoticeable for simple and short tasks.

> --spec-draft-n-max 5

5 is a lot of tokens to draft. Are you really seeing acceptance rates to support that? When I tested it, 2-3 was the peak. Anything more started reducing performance except on highly predictable short outputs.

I'd love to understand this more. Are you saying the Qwen team spends their very impressive human and compute resources on publishing these amazing models and then botches the chat template with mundane bugs?

Like maybe I just misunderstand what's the hard part but wouldn't you assume that people who can put together an impressive model can also write a proper jinja chat template for it?

Hm, I have a 4090 as well, and:

$ build/bin/llama-server -m Qwen3.8-27B-IQ4_NL.gguf --mmproj mmproj-BF16.gguf -c 170000 --parallel 1 -ngl -1 --cache-type-k q8_0 --cache-type-v q8_0 -b 1024 -ub 512 --flash-attn on --no-context-shift --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 5 --spec-default --cache-type-k-draft q4_0 --cache-type-v-draft q4_0 --threads 24 --jinja --reasoning on -fit off

0.02.993.689 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 911.53 MiB on device 0: cudaMalloc failed: out of memory

Update: Oh, it works after I stop Xorg. But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?

I understand the B70 is a bargain vs AMD and especially nVidia offerings, but to me it feels like I would be buying something that would feel too limited in less than a year. 48G would be much more confortable.

And I know the 96G nVidia cards are selling for over 10k$.

The future can’t arrive fast enough!

Interesting. Why don't the unsloth guides (https://unsloth.ai/docs/models/qwen3.8) mention this? Do they already include the fixes in their GGUFs?

Pretty sure you can use Gemma models on Google's "Vertex AI".

Why are you comparing a 2.4 trillion Max model to a 31 billion model?

even a 3090 will give you the VRAM headroom. i run Q8 on an 3090/A6500 combo. well, Q8 of 3.6-27B. I'm building the Q8 GGUF for 3.8 now, assuming mine will finish before someone else's.

Others in this thread said it runs on RTX 4090.

Where do you see that? From what I can see on Open Router, Qwen 3.6 27B (the closest dense equivalent to Gemma 31) is $0.28/m. Am I missing something?

https://openrouter.ai/qwen/qwen3.6-27b

Yeah, I would appreciate if someone could make sense of the pricing differences between these models. How can a provider run DSv4F at lower cost than a 27B dense or 35B A3B model?

Does it come down to utilization and/or specific model tricks and efficiencies (attention, kv cache, etc.)?

DeepInfra prices:

Qwen 3.6 27B: $0.32 in / $3.20 out

Gemma 3 27B: $0.08 in / $0.16 out

DeepSeek V4 Flash 0731: $0.08 in / $0.18 out

Qwen 3.6 35B A3B: $0.10 in / $0.95 out

https://openrouter.ai/qwen/qwen3.6-27b

https://openrouter.ai/google/gemma-3-27b-it

https://openrouter.ai/qwen/qwen3.6-35b-a3b

https://openrouter.ai/deepseek/deepseek-v4-flash-0731

Good news, MTP support is already included in this release. Not sure why they haven't made this clearer.

Why would anyone rely on the world knowledge built into a model when the harness can just let it search for current information?

Intent understanding is a big point for sure, but world knowledge I'm not sure I see a use case for it.

> RTX 3090 for each and every employee

That’s quite inefficient. Get 2x RTX 6000 Pro and run DeepSeek v4 Flash for the whole company. This may or may not be cost effective compared to cheap third party providers, but it lets you own your own data, is not actually that expensive, and reports suggest that you can get 2k tok/s or so from a setup like this (at respectable batch size, but that’s exactly what you want in this situation).

I’m not sure I believe that the unquantized performance is as good as people are claiming, but even light quantization would make it fit easily.

Why do you assume 2-3 American companies failing will have a more negative impact on the economy than positive impact of the widespread availability of self-hostable, open-sourced, and efficient models?

There will surely be tons of new companies that make a business off of hosting these models and even tuning them for specific purposes. Not to mention the massive pricing difference will benefit people who actually utilize them

Lacking world knowledge is fine for me, I rarely want to rely on the model’s training anyway when there are plenty of great search options to integrate with.

Yes. It won't be as fast as fitting the whole model into VRAM but llama.cpp defaults are pretty smart about GPU/CPU splits these days. Just YOLO it with `llama serve -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL` and it'll definitely at least run.

Could you elaborate please? Genuinely interested

pretty sure this exists already...

Thank you, this is exactly what I needed.

You could run the 2-bit quantization at about 10GB for the weights. You'd probably also need to quantize context to get a decent-sized context. The Unsloth model card shows the 1-bit quantization doing seemingly shockingly well on a task (but there is notable loss in model quality for each bit you give up below about 6 bits). An MoE model can partially reside in system RAM without catastrophic performance loss, but a dense model like this one cannot. You have to run it entirely out of VRAM, or it'll slow to a crawl.

Similar to the way they asked Sol to solve Erdos problems, that's what I want my model to do for programming.

I don't want to try to take my best educated guess at what the best design is BEFORE implementation - especially if you're designing a feature for a codebase you're not an expert in, you don't know like the back of your hand (i.e. one that is mostly or entirely LLM generated).

What sounds good on paper - often times becomes unideal in practice when you get to the reality of implementation.

It may not be worth re-architecting your entire system to get to a "pure" design that would be the best - all things considered.

Instead, I'd like the model to independently design many plausible and coherent good solutions, then implement each of them, then intelligently pick the few winners (after its fixed any bugs that could be causing promising solutions to look artificially bad) - unless there's an obvious one - and then give me the data I need to make an informed decision on which one to go with, all before I even look at the design or implementation.

You're not getting this from a one shot prompt from a 30B model today. You can't even really get it from Sol or Fable - IME. But you can get somewhat close.

Same, with a multimodal DSv4 Flash I would just stop paying attention to things. Very smart, and at 260 tok/s it's too fast to care about anything else. If you ever graft something like that I would love to hear about it.

Yes, I have a very dumb flow. The harness has a describe_image tool that takes an image and a prompt and so DSv4 Flash uses it to get an idea of what it's looking at.

Given that it apparently defaults to 'xhigh', this is probably the answer.

Granted, it's still much lower tokens/s than you'll get out of many MoE models.

Edit: Even set to medium or low there's still a lot of second guessing, less consistency, lower 'acceptable response' rate, and slower/more token churn vs gemma4:26b-a3b. I think gemma4 is just a better 'general purpose' model.

As a MBP and DGX spark owner, I would love such a site... (Feel like it would be a low effort feature of hugging face).

Searching through Reddit and forums for best commands is annoying.

Go on… what is this $1000 system?

IME using 5.6 Luna and DS V4 Flash, I notice that although they are excellent at programming, even Opus-like in the way they try to debug, the thing they are worst at is inferring user intent and making good decisions with little information. They are absolutely terrible at that, will misinterpret small wording ambiguities. I suspect that's an ability you can't add with RL training, that it requires the depth of understanding from vast pre-training.

whats the tok/s you get on that. I have heard a few claims of around 30-50 with mtp, but for how cheap the setup is I am surprised I don't hear more about 3060 stacks so I assume there has to be some catch.

In my experience with Qwen3.6 35B-A3B, disabling thinking made the model generate inaccurate replies. Ask it for the recipe of egg salad and it gives you the recipe of an omelette.

Did I miss something, is it possible to have that model be reliable without thinking?

And yet it's also a laptop you can basically take anywhere unlike a giant video card with 1000 watt power supply requirements.

What stackoverflow should have become.

Sorry, I wrote autocompletion by force of habit. I simply meant it can complete code you have already created a structure for, which personally is very nice

Asking it factual information[1]. You just can't compress the entire human knowledge into a 30GB file.

[1] Without searching the internet. And even if you allow it, you'll get much worse results because search means browsing and parsing the top results, and search results are horrible, whereas internal knowledge from training encompasses the entire internet plus all books including very niche stuff.

really? crush.... it's trash harness compared to pi.

I didn't realise there are people out there unironically using crush

Problem with that is I think that it quickly devolves into cargo culting, nonsense and noise.

Arguably, what I am doing is also very very close to that, with the only difference being that I am somewhat less of an idiot than the average internet dweller you'd get on such a site. Or rather a different flavor of idiot.

Ideally, the people building the tools build them in a way that just does the right thing - which I am confident that llama.cpp does or will do in the future.

So you encode that knowledge not in language and online comments but in code and with a filter for actual expertise.

And, frankly, there's really not all that much to it. It's like maybe 3 parameters to play around with.

The valid solution space is pretty small, but people will want to make it "theirs" regardless, so you get non-solutions just so that everyone could also be a part of it. The usual social dynamics foo.

How many tokens/second you getting? I have the same CPU but a 5070 and only 64 GB of ram. I just got llama.cpp built and am now hitting a whopping 5-6 tokens/s.

Okay, so, I did some actual coding with qwen3.8 in the last 2 hours, and if I (let the agent) take the average of all those "mean len" loglines, and let them get weighted by how much happened leading up to that logline, it's an average of 3.33

The unweighted average is 4.1

So with qwen3.8 as well, I think I would not go below 3 in this setup with this workload. Meaning 4, and, to push it a bit further, 5.

Also, cool that it carries over from qwen3.6 like that.

The chat templates are usually the first thing that every major release bork on, and all new model architectures end up having a ~2 week initial window of small fixes before they’re not DoA

I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.

Since the 3060s would have to communicate constantly using pcie, I think I'd go with intel

You may be able to run a quantized version. One strategy I've found effective is to set Claude or Codex loose on the problem and they'll do a decent job setting up the best possible model revision.

>But nvidia-smi only showed Xorg using 200M out of the 24G, so why would a 911M alloc fail?

That was just the last buffer allocation request that failed, it didn't tell you by how much it failed by. It could have failed it by a few kilobytes, it could have failed it by 910MB. One would guess it probably failed it by a couple hundred megabytes in the end judging by your results.

32GB is perfect for models around 30B parameters. Since qwen has really hit the spot with their 27B dense models, I think it's a good bet. Also, 32GB is enough for other tasks such as image/video generation and loading multiple smaller specialized models

Yes, I was fixing issues piecemeal until I found the froggeric template, I've had to fix I think one issues with that one but it's better.

Yes, and this is not the first time they messed up. They had tokenizer bugs where the trained weights do not match the template back to Qwen3 series.

Thanks for this! I got the same card, but not the same CPU nor RAM, but this is a great starting point to start tweaking

@danielhanchen may be able to answer this

The person above is running the OS from a separate GPU, so the command can use all of the VRAM of the 4090.

If you're trying to use the 4090 for your desktop at the same time, you will need smaller context size. You could remove the '-c 170000' and try turning fit on to see what fits.

They usually do include fixes yes.

Yes for the first question. Google of all companies didn't even get it right with Gemma for a while until recently. For some reason it doesn't seem like people can actually get these templates right.

You have understood correctly.

One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.

But they don't.

Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.

I also don't understand how this repeatedly happens.

According to this shitty vibecoded thing "I" built https://hypfer.github.io/will-it-fit-llama-cpp/ (and I guess according to math too), FP16 K/V would give me something like 90k context at the same model quant, which doesn't really fit my usage.

But maybe someone else has experience to share there

Yes yes, oh god yes. They also spread FUD in the form of terrible recommended sampler settings.

If you're using llamacpp, turn on top-n-sigma with sigma of 1, turn off top-p/top-k. You'll thank me later.

You can buy two b60s for $1300 right now (650 each) if you want a total of 48gb. Intel recently raised the price on all of their gpu's except the b60 series, so they are currently the best deal per gb I think.

There has to be a better way of sharing config , a profile of sorts you can upload and share

Yes to both.

The thing is that I can either use the q8 context, or have not enough context window, so I just live with whatever degradation there is. The same can be said about the IQ4_NL. I would not go any lower though.

As for the draft count, indeed that depends on what you do with it, but for coding, reverse engineering and that kind of stuff it does pay off in my testing, though 5 is really pushing it, but the 4090 has so much compute.

Last logline I saw scroll by right now had 47% acceptance rate for 4th and 28% for 5th, but not sure how representative that is. I think when tuning 3.6, I saw more like 33%? But not 100% sure.

They're comparing Qwen 3.8 Max to Gemma 31B, fundamental mistake.

I don't see an MTP entry on Unsloth though. Maybe it's not available in a lower quant I need for my poor GPU.

DeepSeek V4 Flash is natively FP4 MoE with very compact KV cache. Say 8 GB/s. Qwen 27B is about 60 GB/s at full FP16 precision.

Firstly, I don't think it will be just 2-3 companies that fail. Due the circular investing between companies in the AI ecosystem, the fallout is going to be much bigger.

Secondly, even with that being true, I think open models are a great net benefit for all mankind in the long run.

I think both of those things are true at the same time.

What would the time to first token be on a setup like this?

Depending on your compliance needs this may be the only way.

Yeah. I use it to do extensive work on full source bootstrapping, deterministic operating systems, compiler debugging, kernel debugging... all with one tiny go binary without endless NPM deps like pi (which /I/ regard as trash)

Works better than opencode (pi based) or anything else I have tried for my needs, and by far the prettiest and easiest to reason about what is going on.

But I will bite. What does pi do today better than crush for your use cases?

4x CMP-170HX he purchased with his time machine last month

The official docs recommend different sampler options for thinking off. Have you tried adjusting those?

I think someone ought to encode this into a harness. It is really insightful into how we should be spending time if it is going to be spent reviewing AI code.

Turning off thinking mode you need a different tokenizer and a whole host of settings. It's more like switching your vehicle from gas to diesel, than turning your headlights from lo to hi.

I haven't tried lowering thinking, however, I actually asked a solid question earlier regarding a real world scenario I encountered and all that excessive thinking made it give me an amazing answer. The thinking actually all made sense, and honestly I found it thought of similar stuff to what I thought when I drew my own conclusion.

I don't usually rely on AI for much (I'm actually kind of anti-AI, although I follow stuff like this enthusiastically because of the rapid advancements, "average joe" access, and openness), however for what I asked? It was spot on.

The subject was a bit personal, so I won't share. I was just curious what AI would say about the situation and it definitely surprised me, especially since Qwen, while usually great on development/coding stuff, has shown weaknesses in other areas.

Definitely a solid release, and this one runs on my 4090 with minimal loss of quality!

still could! there's the network site https://ai.stackexchange.com/ just need people to ask questions and to downrank accepted answers older than a year (that might not be possible though).

I used GPT-5.6 Sol high to optimize it, and it claimed it was getting 50. I'm seeing ~40 on my goto smoketest: "Make me a vector add in CUDA".

Funny side note. It successfully one shot the program, but it wasn't able to run it because there literally wasn't enough VRAM left to allocate CUDA memory. Watching it try to debug that was fascinating. I'm pretty sure it would have killed the llama-server (and thus itself) if it hadn't been running in a separate container.

Yeah, I might just replicate what you're doing. Main issue right now is just finding spare vram to actually run another model in parallel... And yeah, if I train up a vision adapter somehow, I'll try to put it up/post about it, seems like we're getting the killer apps for local LLMs right now, where it's just feasible enough if you're enthusiastic enough to be a bit economically irrational, and just useful enough to sort of rationalize.

This is a harness problem not a model problem, try prime agent it can do that and it will do it well even :P but you need to prompt it in according to its tools and processes.

I've found that once you factor in multiple GPUs things can get complex quite quickly because the default packing routine in the LLM runners tends to be very coarse resulting in substantial amounts of VRAM wasted. More so if you start running drafters and multiple models at the same time.

Let me return the favor:

Deepseek V4 Flash 0731 Q8

This is on a 4x3090 box with 256G RAM and a 5955WX TR; the CUDA drivers are patched to allow the 3090's to talk to each other. MB is a WRX80E-SAGE (which has 7 slots at x16), BAR > 4G on and PCIe BAR resize set to AUTO in the bios. OS is Ubuntu 25.04.

GGML_CUDA_P2P=1 build/bin/llama-server -m ../DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf --alias DeepSeek-V4-Flash --threads 32 --threads-batch 64 --cache-type-k f16 --cache-type-v f16 --metrics --cache-ram 0 --load-mode none --fit on --fit-ctx 1000000 --host 0.0.0.0 -fa on --top-p 0.95 --temp 0.7 --repeat-penalty 1.2 --jinja --tensor-split 1,.8,.1,.15 --cont-batching -b 4096 -ub 4096 -ncmoe 36 --parallel 1 -ot 'blk\.3[67]\.ffn_(gate|up|down)_exps=CPU'

I'm still working on an optimal packing for the draft variation on this theme but that gives all kinds of issues. The 1,000,000 context is very nice to have.

You're going to be waiting for a while.

Even Fable is bad at this, I would constantly have to fix it going down architectural dead ends or just making obvious mistakes.

Which sucks for people that want LLMs to do everything like a genie, but does mean senior engineers have a few more years before they become redundant.

So add a pre-ingestion agent to your Pi Coding Agent subagent repertoire, problem solved.

> You just can't compress the entire human knowledge into a 30GB file.

...I'm just asking questions here... how sure are we of this?

If you'd asked me six years ago whether we could compress all of human knowledge into a 1 TB file, I would have said no, and yet here we are.

If you can do 1 TB, why not 30 GB? It's, like, within one order of magnitude.

> You just can't compress the entire human knowledge into a 30GB file

Fortunately, that isn’t necessary! What LLMs need is a level of fluency with key concepts so that they can (1) make effective use of retrieval tools and (2) understand the material in the context window. 30GB-sized models can absolutely store enough knowledge to do this.

Here is an example from the field of law. Most lawyers who have litigated contract disputes in California know about Civil Code § 1717, which makes any contract providing for attorney fees to a prevailing party mutual, so that even if the contract was written to be one-sided, it won’t be enforced that way. It’s a simple enough concept, but there are many more particulars to it, such as what happens when the fee provision is only written to apply to part of the contract. (Answer: it depends on other facts.)

When a lawyer recognizes that they’re in a situation where § 1717 is relevant, the first thing they will do is pull the statute and read it, because nobody has it memorized. And they don’t need to.

I thought so, but thanks for the clarification. I am a little bit disappointed that local autocompletion models have been left by the wayside in favor of models post-trained for agentic coding. Both Codestral and Qwen-2.5-coder are more than a year old at this point, but local auto-complete seems to me to be such a great usecase.

Looking forward to having your problems :)

What's wrong with it? I like Crush. It's hard to find good harnesses that don't pull in mounds of Javascript like Pi and OpenCode.

They have terrible cross connectivity, it's not wise to share a model between them

How is terrible settings a case of FUD?

Yes, llama.cpp supports presets (an ini format). Though it's not always clear which llama-server flags it does not support.

I did SFT / RL post-training on Qwen3 models a bit. This is an issue that dates back long ago.

My favorite theory is that they had many model variants internally, each using a slightly different chat template, so when it comes to the release they are not even sure what to use any more.

Nvidia and AMD both have their own unified memory laptop SOCs, now. Apple Silicon's GPU is relatively weak, it's one of the less-efficient ways to use 100w for compute.

Even the fastest Apple Silicon chips like the M5 Max and the M3 Ultra still put up worse GPU compute performance than last-gen laptop RTX 4080 chips. And they don't scale, the largest M3 Ultra cluster you can configure is still ~2,000x smaller than a DGX SuperPOD. There's a reason Apple discontinued their rackmount hardware, there's very little demand for Apple Silicon in the datacenter.

Same, I have one workload where on 3.6 drafting 6 tokens is the fastest setting.

For those us us who don't know, what do those parameters do and why are they better?

just to clarify. yes YOU built it. just because you used some tool doesn't mean the idea, prompting, reprompting, babysitting was not your creative input and effort.

put differently, if you put a random person infront of whatever model you used (say, a 50yo receptionist at a pharmacy in india), they would not have been able to create that, because they would have lacked the motivation, idea, background knowledge, taste, etc to create such a thing.

Is the chat template used at all when they benchmark the model?

Diverging from the sampler used in RL training is not good for long multi-turn results-- it's a great way to knock models into reasoning loops that wouldn't otherwise.

If active weight bandwidth is everything, then wouldn't Qwen 35b a3b (FP8) be roughly half the cost of dsv4f (FP4)?

dsv4f appears bigger and more memory/computationally expensive to run any way I look at it.

English Wikipedia is around 25GB compressed

Amusingly enough, someone from a frontier lab could probably answer this empirically. Years ago Microsoft demonstrated training LLMs on synthetic text-- books rewritten by LLMs to be more concise and more accurate. https://arxiv.org/abs/2306.11644 It is well known that Anthropic extensively uses synthetic text in training. You could probably get good 50tile, 90tile, 99 etc numbers just from the size of the training materials on Anthropic servers.

Please, could you point me to any documentation about this pre-ingestion?

I’ve looked here: https://github.com/earendil-works/pi/blob/main/packages/codi...

> (1) make effective use of retrieval tools and (

A downside is that you can't just download a lot of that knowledge, vs with the weights the copyright infringement has been outsourced to the lab. Nor can you just search for the info because the internet as a whole is increasingly aggressive at blocking anything that looks like an AI agent.

I'd love to see more retrieval powered local AI-- I think it's an area that open source development could excel. ... but there are advantages of having the knowledge in the weights!

Perhaps what needs happen is for someone to make an "ultrapedia", an AI restatement of a huge library of reference works-- created expressly for the purpose of being a locally stored corpus for AI agents.

I honestly think we've got about a year tops.

(mumbles) Shannon entropy... Kolmogorov complexity... something, something...

On a more serious note, it depends on your cutoff for "entire human knowledge". It's easy to prove for a generous interpretations of "entire human knowledge" that it can't be done, but hard for something like "all useful human knowledge".

Woah woah woah buddy... suggesting that people use anything other than ClaudeCode or Codex is simply not allowed around these parts.

they likely use their internal infra to run benchmarks; aligning external releases with internal environments is always painful and somewhat underincentivized

Peer reviews NeurIPS caliber paper to prove that? Because I can show you one titled "Long context generation is a sampling problem"...

As I mentioned above you can't take the data center into the cafe somewhere. We're talking about running local models here.

It’s unlikely they are benchmarking that downstream. They probably have private benchmark scripts with purpose-specific run telemetry and logging, etc.

I’m sure there is some basic testing but it might be agentic (LLM likes its own output) and maybe just some human smoke tests.

> and more memory/computationally expensive to run any way I look at it.

Just speculating, but look at attention. 35B-A3B context is heavier -- about 20GB per 1M tokens vs ~7GB. So if you have ≥10M tokens of context per inference node, then DS4 Flash uses less memory than Qwen 35B.

Qwen attention is probably more computationaly expensive as well.

Also I guess not much interest/competition in serving Qwen edge model.

I generally agree and expect this to be the case from a legal perspective.

Legal questions of authorship are going to have to be established in terms of doctrines like SSO [0] and AFC [1]. Currently the incredibly sparse caselaw around this has yet to involve such non-literal notions of copyright.

[0] https://en.wikipedia.org/wiki/Structure,_sequence_and_organi...

[1] https://en.wikipedia.org/wiki/Abstraction–filtration–compari...

The latest Qwen models (including 3.8 27B) do still support FIM-style in-editor code auto-completion if that's what you're looking for. I wouldn't want to use a large dense model like 27B for such a task (since FIM-style auto-completion really works best with low-latency responses), but it works.

Yeah, it should be basically free. No idea why it is not. I guess KV cache taking up RAM and possibly bad business sense or amortized engineering costs, I honestly do not know.

I wonder if we could take a page out of the Solar power book and do MPPT but for draft count. Constantly sweep through possible values to find the best result for the current conditions.

Though it might turn out that that doesn't offer any benefits. Has it been tried yet?

___

EDIT:

LLM pointed me towards this thing I'm not going to read

https://arxiv.org/abs/2512.11280

"AdaSD: Adaptive Speculative Decoding for Efficient Language Model Inference"

Some higher level nerd please go have a look at that. I have important meetings and golf classes to attend.

You sound like your trying to reassure yourself of something.

I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!

Temperature, top-up, top-k, min-p all control which token the model predicts next and how likely it is to select one token over the other.

You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.

That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.

So if I hire an artist and am a motivated individual, have an idea for a painting, have background knowledge about paintings and have taste in paintings and can offer a critique of the painting as the artist paints it, then somehow I created the painting?

Absurd logic. The AI built the website.

Some people will now argue it was the chisel—not Michelangelo—who created David.

I'd consider a year a while, but then, time is relative.

Wikipedia (text) is about 25gigs compressed. I think that's a reasonable starting point.

IDK, 30GB is a lot of data when we're talking about text!

Moby Dick, uncompressed, is ~1MB. Compressed, it's around 500KB.

I feel fairly certain that one could fit all of the textual knowledge required to cultivate a world-class <insert name of preferred professional knowledge worker> in <60,000 Moby Dicks. (Arguably in <5,000 Moby Dicks with intense effort/pruning).

I wonder how much more effective LLMs would be at general knowledge if you just download wikipedia and set up an MCP for it.

Your negative impression is surprising. Could you point to any threads to support your message?

No, it's the difference between management and direct work.

None would claim they chiseled anything, if it was 3D printed. They may claim they designed something.

no, because there is another human involved.

llms are not human.

Deepseek's DSpark does dynamically adjust speculated token count per user/completion.

https://arxiv.org/abs/2607.05147

But they do so to maximise total throughput, I don't think there's reason to do that for batch=1.

I just read through a couple of your posts that weren't dead or buried, and it seems like you're pretty anti-AI. You should really start to have an open mind towards it. It's going to be the future (if it isn't already), and as you continue to get older, you're going to really wish you spent your time right now learning and embracing the technology instead of being so against it. A lot of the skills and things that you're holding on to right now might not be relevant by then, but you'll be at a disadvantage from not keeping up with the industry and need to play catch-up.

The idea guys will really think themselves gods, now.

Nothing absurd about that. What do you think an "Executive producer" is? A "Director" ? Does Peter Jackson get credit for creating the Lord of the Rings Trilogy films? Christopher Nolan for his films? But did he make them ? No, it was the collective effort of thousands of individuals all working under their direction.

Just like if somebody creates software today, and the end result is generated by the collective effort of thousands of agents, the "Director" still gets credit.

I broadly knew that about temperature, but lack the background in machine learning/statistics to differentiate top-n-sigma from top-k/top-p.

There have been plenty of workshops where artists hire assistant painters while maintaining authorship over the works themselves, from Rembrandt to Warhol to Hirst.

I think in that case it's fair to say you created the painting with the artist, even if the artist should get majority credit. I don't like the analogy though, to me it feels more like you're a project manager directing a team of genius but single minded interns.

> I sure hope my boss doesn't think he built my work!

Most managers do though?

You know how many pieces of art Damien Hurst creates himself Vs his studio assistants creating them under his direction?

For example, of his 1500 spot paintings, he only actually made 5 of them.

It's not uncommon at all for artists to work this way.

Don't you just need to occasionally try n+1 for your current setting? Then record acceptance rates. The nature of the thing is that if you try n, you already have a good sampling of how well m works for all m<n, right?

"Carve me a naked guy. Make no mistakes."

> I sure hope my boss doesn't think he built my work! He'd probably get fired pretty quickly during on call!

Your boss is a human.

This is a computer program running on your PC.

I hope you can see the difference.

I've heard that some people even have a plan on how they'd like their life to look like in 5 years that they're aiming towards.

I think a specialized model could squeeze all you need to know about a certain profession in 30GB. But not all professions at once, which is what these models try to do.

Or maybe not, maybe there's a world model needed for human level at any profession that is very hard to quantify and requires more than 30GB by itself.

This is all a pretty active area of research, both adaptive drafting, different draft approaches (DFlash etc), I expect to see really big improvements land in the popular runtimes this year.

I've worked with a good few types of managers of the years, none would take ownership of my work. The end result? Sure, for shareholders/this managers that is fine.

But that's not the same as me being needed to be on a call with integration teams and having to run the call. It's just a layer of abstraction for management. I own the creation side of the product.

Who created it is still me.

So do I, but we live in the future: https://chatgpt.com/share/6a7fc3d2-39f4-83e8-a7c6-825ddfb5e7...

Well, this universe might be God's own AI slop.

I cannot fulfill this request. I am programmed to follow safety guidelines that prohibit the generation of sexually explicit content.

He still didn't create the paintings, he offered instructions / feedback to the artists that created them. Whether or not he is credited with authoring them is an entirely separate issue.

That’s an interesting way to look at it, and still generous to the llm.

By which I mean it puts the llm in the role of actor and such… people practicing an art. It seems harder to make a case that the llm is actually making creative decisions rather than aping a synthesis of past human decisions and expressions of taste, strictly, at every turn.

Though I guess it’s all “shades of grey”. I never felt like I was contributing a ton when I was essentially wrapping a large, capable library that some smarter person or people made, either.

Tangentially… remember back when we used to see headlines like “12 year old made a web browser!” and we all thought, “No. No they didn’t.”

It all feels a little like that, again.

Top-K: example setting 20. Select only from the 20 most likely tokens.

Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.

Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.

The purpose of all of these is to exclude very unlikely next tokens.

Sure - and a factual statement would be, these artists hired assistant artists to help them create their paintings. Just like it would be factual to say that the person directed a LLM to build an app for them.

I'm not sure that typing messages to a chat bot requires much catching up, but since you think this is what I should do rob, I better listen!

The director receives accreditation for directing the film, not creating it.

I've never encountered a project manager who has said: I built this thing, after a project has wrapped up. I have encountered ones who've said I managed this project.

Yea to be clear I think >70% of the information is not profession specific.

I just think about all the content I’ve consumed in my life to become a professional software developer and I would be very surprised if it couldn’t be adequately represented by <30GB of uncompressed text. Most of the work was in “training”, not really in data.

The “foundational overlap” of K-12 education is identical for most professions with 2-8 years of “finishing” on top.

My mental model is that the budget is spread across maybe 20% genetics (most of our instinctive/genetic information is surely pretty useless), 50% k-12 education, 30% for professionally-specific knowledge.

Did Steve Jobs create the iPhone?

Sure. So does this diminish the role of Rembrandt? Or is this some semantic black hole you’re drawing us into?

> I'm not sure that typing messages to a chat bot requires much catching up

Lol, yeah I'd wager AI tooling skills are less relevant than knowing your way around a shell. It doesn't help that all of the AI tooling has event more churn than js libraries and package management systems!

I think there's probably some value in understanding how LLMs work, but beyond that it's the same boring skills that matter the most... Critical thinking, design chops, attention to detail, perseverance, etc.

If you don't get reliable results, but others do, then maybe you should indeed. But your choice.

You might find this article relevant: https://news.ycombinator.com/item?id=49151933

I've heard some variant of "we built this thing", which I think is valid. I'm not sure how I feel about anthropomorphizing a coding agent at this point though, which is what you're implying.

Doesn't the director generally receive more credit than the producer?

How many films do you remember the producer above the director?

The human genome is 800MB, (and 500MB with compression according to GPT) so that part is easy.

I think the problem is that the conversation drifted from "all useful human knowledge" to "enough knowledge to do my job".

Both a human and a current large model will bring up unprompted some tax regulation that applies in your district that could matter to your task. A small model won't know the tax code of every city in the world, as it's probably impossible to fit, and it's the kind of thing that unless you already know about, it's very hard to search for, let alone come up with unless specifically prompted about something tax related.

Unless we start having different small models trained for a certain profession for a certain city, I think we'll need those trillion parameters.

Would it exist in it’s present form if he didn’t exist?

No, he did not. When it went wrong for v4 with the radio. I didn't see him take the blame.

I've run the inference to get the answers I linked to. If someone else does the same thing, that involves extra energy. If I read your conversation instead of generating my own, then that's one less tree that has to be chopped down.

Until we go advanced geothermal or we crack fusion, energy is dirty. Read my inference or link me to yours so we don't boil the planet.

What does it matter who receives more credit? The producer produces the film and the director directs it. If I pick my phone and record a video, I'm now the producer and director of the film and the sole creator of it.

If there were multiple people involved in the creation of a film I helped to create, I cannot factually say I created it. Just like if someone builds something using code generated by AI, they can't factually say they created it.

this topic always goes into this level of ridiculousness. Your question can't even be answered. We should focus more on encouraging discussion, not asking unanswerable questions. It wont bring anything other than points of both sides with no objective correct answer.