So, any solution to this “problem” must include ALL open-weight models. As far as I understand this is exactly what they intend to do. Axios article linked in the post mentions that. As in this quote:
“The source described leading AI labs or their allies approaching the administration every 3-5 months with an idea to ban open-source models.”
It doesn’t say “Chinese” open-source models. Because they already know that it’s not feasible. Any regulation must cover all the models.
Now there are solutions for that latter problem. But they are all ugly and restrictive. Making a DRM-like license protection system mandatory can be a solution. If a company wants to run an open model in their own servers, they can only use approved and certified pure “American” models. This of course creates a monopoly for the big labs who are authorized to train and distribute such “open” models. A company can fine-tune the model for its own needs but of course can’t distribute the derivative model.
I’m sure there are other solutions but all of them would be equally ugly. Also these regulations can’t be enforced to other countries easily so only Americans will be restricted.
That's going to hit first amendment grounds pretty quick, the same way that software in general did.
The modern version of the decss flag will be a character that says "I think good weights are {...weights go here...}"
They could, however, ban any payment to a chinese entity, or any entity owned by a chinese entity for inference/ai services/etc
Might make sense eventually. Same logic as companies working on Linux. "An AI model is a business necessity. But making an AI model is so expensive we should not make our own. So let's just use the open one, and contribute the stuff that we need."
Now, here is an idea that I have not heard before... and I think there is some merit to this. This is also the sort of thing that a state (looking at you CA, CO, IL, NY) could do, instead of just the federal government.
That assertion is sublime, and if not true, it should be and can be.
Thanks for the thought (and the optimism hit).
Kubernetes took off because everyone could run it on pretty much anything. Bigger hardware meant bigger clusters, but developers could spin up a cluster on their laptops and deploy their apps into it. More importantly, companies could repurpose their decommissioned servers as k8s clusters, a massive unlock seeing how huge companies had heaps of these in their racks.
This isn't possible with open weights models. Not in the same way.
First, you're out of the game if you don't have a data center class GPU (or it's sort of prosumer equivalent). Model servers support CPU inference, but you might as well watch paint dry as you wait for results...and you'll still have to run super quantized low-parameter models that aren't as good.
Realistically, companies will need to purchase millions of dollars of nVIDIA gear (through suppliers) to serve agentic-capable models at scale, an activity that is being made more expensive and complicated by the day as the hyperscaleds slurp up the demand.
It also really is a huge problem that all of the open weights models are coming out of one country that also happens to be a superpower. I don't think this can be handwaved away, and it's concerning to see so many folks here minimize this.
American companies running on Chinese intelligence. As a country that prides itself on being the knowledge capital of the world, the optics manifested by this are horrible, not to mention the absolutely massive supply chain risks (the counterfeit Cisco devices comes to mind, except worse because the rangers are in the weights and might not be possible to distill out).
This is a threat even if you focus solely on the individual developer. Recall how AWS became...AWS. They "fanatically" focused on the developer experience. They designated this as key to their growth strategy, and rightfully so. How people talk about using Qwen or Kimi and the like on here feels like that (ignoring how these models are ALSO from huge for-profits).
The only solution here is for the big labs to make some of their most capable models open-weights. There is a lot of secret sauce around routing, inference, hosting and other stuff that makes them work as well as they do, but the community can figure that out. Of course, this is basically a death sentence to those companies, but that's what they get for playing with monkey paws, I guess.
So what open weight models do is at least provide a baseline of inference cost to add some sanity to the price markers. And of course predictability too--if you really want Kimi K2 instead of K3 you can still use it.
So the competitive pressure and predictability offered by open models is helpful for users
oh now i see, the Chinese government is funding the training and release of their best models to pressure OpenAI, Anthropic, and others to do the same for competition's sake. I don't buy it, this seems more like a way to get SOTA models RL'd to comply with Chinese government approved information distribution. If I have to trust a black box of answers to questions i would trust one from a US for-profit publicly traded company subject to market forces over one approved, and heavily subsidized, by the Chinese government.
Honestly imo this is just proof apple will win in the end. Eventually a phone will be able to run a model good enough to do most things and it then is game over.
To be fair, OpenAI has released a couple of (then very good) OSS models. I run the 20B version at home and it is excellent for reviewing text and common tasks like drafting bash scripts. There is a larger 120B that you can't realistically run on consumer hardware at reasonable tok/s too. I wish OpenAI updated these models more frequently though.
What is your stack (harness, model) and how much do you pay per month?
How would you compare your experience to a typical subsidized plan like Claude Code + Pro plan?
I’m asking because i keep hearing that open weight models are cheap and efficient - is that really the case in practice?
What's the incentive for the Chinese labs to continue releasing weights 5 years from now? It's not a stable equilibrium and cannot last.
- The lab spending large sums on research and training does not get the inference revenue to fund those efforts.
- Unlike OSS where a single volunteer can keep a project going, training costs run into the $billions.
- OSS is often a two way street where features and integrations are built that the original author benefits from. Open weight models are largely a one way street because the marginal benefit is so much less than training costs.
In the short term, it means Chinese labs can attract talent and, I suspect, funding from their gov. Similar to every other industry the CCP subsidized to take over.
It baffles me that anyone can seriously believe that China is going to put its Mythos successor on Hugging Face and let us all strip the guardrails etc.
Things just didn't turn out the way OpenAI and Anthropic thought; nor did they turn out the way China thought.
Though the legalese might not even be necessary. The list alone will make sure that no American company runs these on their servers, including the hosting providers.
There is no great firewall so banning their import via the Internet is impossible
It is not possible to 100% ban open weight models getting released in the same way you cannot stop leaks.
Just ask Meta with the original Llama leak.
I'm sorry to do that hn comment thing where we all just race to contradict or talk in opposition of whatever was said before. I'm aware. But really guys, was this article really not just a miss?
Once the weights are out, it makes no sense to ban them in the US while the rest of the world takes advantage of it. But that doesn't mean developers of frontier models shouldn't take steps to prevent their weights from being stolen.
Don't count on that. "National security" == the cheat code for the US court system that instantly bypasses any First Amendment issues.
US Gov could make US companies comply, like have Huggingface take down models out of compliance.
But most likely, a foreign-to-US Huggingface replacement would be made and everyone would go there instead. Lose-lose for US.
What about the EU? Would they follow Uncle Sam's order to ban all open-weight models? Lately the EU hasn't been that cozy with american companies: there are EU companies and institutions moving to EU clouds, the EU just fine Google a cool billion, several are switching away from Windows to Linux, etc.
Or is it just the US that'd ban open-weights models, while, say, the EU and Japan would still allow them?
And that list of tasks grows smaller every day
> Everyone is talking about banning Chinese models but nobody talks how it is feasible to ban them. I think it’s impossible simply because technically there is no such thing as a “Chinese model”. There is no way to tell apart an “American” model from a “Chinese” one by looking at their weights. Weights are just numbers and you can’t assign country of origin to numbers. One can find very easy workarounds to any naive attempt to ban them by origin.
Historically just asking it about tianment square or getting some random answers turn into chinese (as latest interation of online deepseek likes to do recently) is enough
> Now there are solutions for that latter problem. But they are all ugly and restrictive. Making a DRM-like license protection system mandatory can be a solution.
I am very worried that's where consumer hardware will go to. All so AI companies can license local use of their stuff, and once that happens, less of an incentive to even have model be open.
Possibly even have DRM that counts number of computation done per model in pay per use model
FlashAttention was a hell of a drug.
There's just obvious and enormous incentive for the OpenAI's of the world, along with all of the other players, to confuse, misrepresent or just straight up lie a whole bunch about everything given how new and unknown the tech is.
The price is what the market is willing to bear for the available compute capacity and competitive landscape. You can only discover that price after trying different price points and seeing what happens.
Everyone is trying different pricing schemes and discounts as they test the market. The demand is fluctuating at the same time.
It’s probably very confusing if you’re primarily familiar with stable and mature markets. Price fluctuations are a common feature of new and evolving markets.
Artificial inflation to recoup R&D.
I think this is a very important aspect, especially after the huge GPT-4o backlash when GTP-5 came out. Each model has certain quirks, and areas where the previous model might be better for some use cases than the latest and greatest, and the labs so far seem to have no desire to offer some kind of "LTS" release.
I can only assume this account is pure shilling for the closed-source AI labs.
Open-weight models are becoming the foundation for the next AI ecosystem. The US should compete in it, not wall itself off.
I have seen a version of this story before.
In 2013 I co-founded Mesosphere, an open source cloud-native software company. We built on Apache Mesos, which my co-founder Ben Hindman had helped create at UC Berkeley. We later built DC/OS (Data Center Operating System) around Mesos, released it as open source, and commercialized it through an enterprise distribution with support and proprietary features.
After several years of massive growth, Kubernetes disrupted us. It was newer, fully open source, and it quickly galvanized the cloud-native community. Many of the world’s best distributed-systems and infrastructure engineers bet their careers on it, and even some of our most loyal community members changed horses.
Once that happened, innovation moved to Kubernetes. Whatever the platform was missing, someone started building: networking, storage, observability, deployment tools, policy engines. A large number of startups were created, and legacy vendors joined in as well. Almost every component required to run Kubernetes in production became available as open source. Cloud providers and companies including Mesosphere/D2iQ, Rancher, Red Hat and Nutanix then built businesses around integration, enterprise features, support and operations.
Kubernetes did not win simply because its repository was public. It became a neutral substrate that engineers, cloud providers and enterprise vendors could all extend to fit their customers’ needs. Common interfaces and vendor-neutral governance gave everyone confidence that they could build on it.
The lesson I took away wasn’t that open source always wins. It was that once an open platform that people can customize becomes the industry’s center of gravity, no single vendor can match the combined rate of innovation around it.
I think AI is approaching the same point.
First, a terminology note. Most models we casually call “open source” are more accurately described as open-weight. You can download and modify the trained parameters, but the training data and complete training process usually aren’t available. That falls short of the Open Source Initiative’s definition of open source AI. The distinction matters. It doesn’t, however, prevent an ecosystem from forming around the artifact people can run and modify.
The Kubernetes analogy isn’t perfect. Kubernetes contributors could inspect and change the actual source, and improvements could flow back into a shared upstream project. Model fine-tunes usually don’t work that way. Frontier weights may be downloadable but still require expensive hardware, and there is no AI equivalent of the CNCF providing neutral governance and common interfaces. Those are real differences. The common mechanism is that a sufficiently capable, portable substrate can attract complementary innovation far beyond what its original creator could build alone.
The first reason to use open-weight models was self-hosting. Companies wanted control over their data. They wanted to run models in their own cloud or data center. And as usage and inference costs grow, they increasingly want control over cost as well.
That demand produced a healthy open source serving stack: vLLM, SGLang, llama.cpp, Ollama, MLX and others.
But self-hosting is only the beginning. Open weights turn the model itself into something developers can adapt and redistribute. Hugging Face now hosts more than two million public models. Around popular families such as Qwen and Gemma, developers produce:
Until recently, it was easy to dismiss this activity at the frontier. Open models were useful, but the base models weren’t good enough for the hardest coding and agentic tasks. That gap is narrowing quickly. Z.ai has released GLM-5.2 with public weights under an MIT license. Its own evaluation reports 62.1% on SWE-bench Pro versus 58.6% for GPT-5.5, although results vary across benchmarks and agent harnesses.
Moonshot says Kimi K3 approaches the closed frontier on long-horizon coding and has promised to publish its weights on July 27. Artificial Analysis supports the performance claim, scoring it alongside Opus 4.8 and GPT-5.5 in its independent evaluation.
Once the base model is good enough, the ecosystem can compound. I expect new projects around agent runtimes, coding harnesses, sandboxes, evaluations, observability and specialized fine-tunes. Together they can become a production-grade stack: an open-weight model running on open source software, customizable for a team’s workload, hardware and economics.
Will that stack beat every closed model on every benchmark? Probably not. But I would not bet on any single vendor out-innovating the combined open ecosystem over time. Frontier technology is a talent war, and open ecosystems give talented people everywhere a reason to build on the same foundation.
This brings us to the current debate in Washington. After the release of Kimi K3 and other capable Chinese models, the Trump administration is reportedly considering restrictions on Chinese open-weight models. The exact form of a potential ban remains unclear.
A broad ban on American researchers and companies using Chinese open-weight models would do something else entirely. It would cut the US off from an ecosystem that is already attracting many of the world’s best AI researchers and engineers, including a large number of Chinese researchers. The rest of the world would keep building. American developers would be the ones locked out.
We’ve seen this dynamic with Qwen already. Hugging Face reports that Chinese models accounted for 41% of model downloads over the past year. If the best open-weight foundation models increasingly come from China, innovation will accumulate around them in the same way it accumulated around Kubernetes.
The US should compete in that ecosystem, not retreat from it.
American labs need to release frontier-grade open-weight models under licenses that startups can actually build on. There has been progress. NVIDIA’s Nemotron models are commercially usable under NVIDIA’s own permissive license. Thinking Machines released Inkling under Apache 2.0, as did OpenAI with gpt-oss and Google with Gemma 4. But OpenAI’s and Google’s strongest models remain closed, as do those from most American frontier labs.
The government should use procurement to create demand for portable, interoperable systems rather than permanent dependence on one API vendor. The Department of Defense has done this before. Platform One provides open source tools and enterprise products that different military programs can build on. The same playbook can accelerate innovation around open-weight models.
American companies need to build the rest of the stack. Startups can customize and extend the models, embed them into products, and provide the serving, tooling, support and operational layers. Our leading silicon companies will keep improving the hardware. Hyperscalers and neoclouds can serve the models and their ecosystems.
Safety is the strongest argument for restrictions, but a blanket ban is too blunt and would sacrifice access to the entire ecosystem. A better approach is independent testing and standards for frontier models. The analogy isn’t exact: Kubernetes conformance tests compatibility, not safety. But the governance model is useful. Demis Hassabis has proposed a US-led independent standards body along those lines.
America should not respond to open Chinese models by building a wall around its own developers. We should run the models ourselves, tear them apart, benchmark them, improve on them and build better American alternatives. Then we should make the American stack the easiest one in the world to adopt.
The United States has spent decades attracting the world’s best technical talent and giving it room to build. Turning that advantage into a walled garden while the rest of the world standardizes on a more open stack would be a spectacular own goal. We would be giving up our role as the AI leader by choice.
The article certainly has a point, the current release cadence is untenable for startups that want frontier-grade models from American labs. Unless America wants to cede that market entirely, OpenAI et. al. need to kick it into gear, fast.
gpt-oss-120b runs at 30+ tps on Strix Halo and +75 tps on a MacBook Pro M5 Max 128GB.
> I wish OpenAI updated these models more frequently though.
I think the spiritual successor is the Nemotron 3 series, although they also are getting a bit long in the tooth: https://research.nvidia.com/labs/nemotron/Nemotron-3/
The Gemma 4 models are a bit more up-to-date: https://huggingface.co/collections/google/gemma-4
Or Qwen3.6: https://huggingface.co/collections/Qwen/qwen36
Quantised models running overnight go most of the way for non-coding tasks.
Deepseek is at least on par with Sonnet (ghcopilot at work)- I don’t use opus, too spendy and I don’t need that level of ability.
The cost is for me was $5/6 months of use. Not a big user I guess! It’s good though, fast enough and incredibly inexpensive.
Been testing Qwen 3.6 28B on a 5090, and it’s also quite good for “free”.
I mostly do small self serving embedded projects based on esp32, so not very complex.
Scale matters for these things. If we divide the available chips among 5 competing companies we end up with models that are trained on 1/5 of the resources that they otherwise could've been.
Let the companies take turns training on shared hardware, force them to publish results in the open, and then reward them based on how well the resulting model performs at democratically chosen benchmarks. Meritocracy not monopoly.
Make it about how well you wield the silicon, not how much silicon you wield, and make it a positive-sum game. If the people's data is going in, then the people should benefit from what comes out whether or not they have a subscription.
If capitalism as we know it can't complete, so much the worse for capitalism as we know it.
I'm running this stuff at home on my desktop and using it through an app on my phone. 60-140TPS depending on model / use case.
It's more than fast enough to even maintain voice conversation.
Claude and ChatGPT are good deals right now, with the subsidies. They produce things faster and better. I guess not cheaper, in that inferrence on my macbook is basically free, although the macbook itself definitely wasn't. My focus on running local is around three principles:
1. I don't want to support surveilance capitalism by giving these companies my data anymore, when I can avoid it. And LLM companies want to vacuum up every detail of your life.
2. I don't find these companies to be remotely trustworthy, and I find them hostile to a healthy society, so I want to avoid giving them money going forward
3. I think they're going to start charging a lot more
I think that's the case for people who compare it to proprietary models paid via API—which I think is irrelevant given the majority of people daily driving AI coding are doing on a subscription plan.
The better analysis then is not about AI coding, since there's no subscription plan for Kimi K3.
Instead, compare the cost of running some agentic _task_ that isn't coding which can only be done via API. Think of all the startups wrapping around ChatGPT and Claude to provide some additional set of tools, context, data and hoping to turn it into a profitable service.
To those companies, which are many, open models are the difference between the math working out today vs. praygeing they can scale fast enough to find profitability.
1. You must be willing to be resourceful. 2. Be willing to learn, do the hard things. 3. Accept the tradeoffs.
When I expect to need a lot of tokens and the task isn't too difficult I use sota to plan and create a thorough set of instructions and let deepseek chip away at it. With thorough instructions the quality tends to be satisfactory, and you pay something silly like $15 for 600m tokens.
GLM 5.2 seems like a decent price/perf and Kimi 3 has some real nice performance for an open weights model, but gpt 5.6 is unexpectedly affordable (especially if you don't automatically use Sol at max) so I don't think either is worth it atm. The exception is when you're working on something that US models get cold feet about, which seems like a constantly growing list. For me Fable is already too much of a headache in this regard, but chatgpt is still okay-ish. Hopefully it'll last. If not, there's Kimi.
tldr SOTA for most things because gpt 5.6 is token efficient. If I expect to burn a lot of tokens I use deepseek 4.
We walked in, and it was fine. Because it was all kubernetes and laid out like every other app for the most part.
The kube hate is just sad at this point. You need to know like 15 concepts that are all applied in the same way. It mostly just works.
For the Hyperscalers...and Oracle...cant wait for the day...
Can't comment on how it compares to plans (I really don't like the limitations and general shenanigans I see around plans, so I've never tried them).
It is notably slower than Fable / Opus / Gemini, but also vastly cheaper than their API pricing.
That said, even the foundational models fail at the hard parts of my code so I use it opportunistically.
I have reduced down to just using zed, will three locally hosted models.
Qwen 3.6 27b on 1x3090 llama.cpp with 128k context ~50tps
Qwen 3.6 35B-A3B on 1x titan v + 2x1080ti llama.cpp with full context ~30tps
GPT-OSS 120b on pure cpu (slow)
I just use zeds parallel agents, task switching, stopping and fixing the code when a model gets stuck.
This still lets me stay engaged, and to modify code to be maintainable etc…
It gets me 80% there and I use to keep a subscription but often times just using googles AI mode is just as good.
That said I have 30 years of experience and insist on knowing how my code works, so this gets me 80% of the short term benefits while not depending on a 3rd party to keep my code moving forward.
Your mileage will vary and 2*5060ti 16gb cards would get around 100/tps with Qwen 3.6 35B-A3B on cards that are widely available.
To be honest the more modern cloud models are using draft tokens etc… that while they are superior for common coding tasks are degrading with more domain specific tasks.
That is just the cost of the draft model being ~10-20% of the foundation models size, and even the biggest Blackwell GPU is limited to ~250/tps so MoE or draft models are required for scaling performance at the foundational level IMHO.
The hard part is my use case are the OOD or small examples in corpus level, the above hurts there.
A Lamborghini may be nice, but I personally need a minivan more.
Even not being able to significantly update a model that is burned on a chip the performance gains are immense. You also don't need the latest chip fabs to make them drastically reducing the cost.
I had a shower thought on how to counteract this, specifically related to the AI dumping. If China is losing substantial money on every token, why wouldn’t an adversary try to maliciously increase consumption? This strategy is not really viable against physical goods dumping because demand is finite and there are large environmental costs. Software demand is infinite and the environmental costs are quite low compared to the financial cost to make it, even with ultra cheap Chinese tokens
https://www.amd.com/en/developer/resources/technical-article...
At 7 months of claude code subscription per node, the cluster pays for itself in 28 months. On a 5 year (60 month) depreciation schedule, you can buy two of those clusters for basically break even, so you get two concurrent request streams (each of which can batch, etc).
The next generation hardware has already been announced, and should ship roughly two Moore’s law doublings later. It’s likely its steady state price is <= $1400 USD (2024), and it is faster.
So, once the bubble pops (because the financial machinations eventually will come to an abrupt halt), and the labs stop buying hardware for data centers, local inference will be extremely practical and cheaper than a subscription.
My main question is, when that happens, will UNIX Surplus be selling inference servers for pennies on the dollar (like after the dotcom crash), or are the power requirements too exotic for home use?
The flip side is that it takes four H200s to run, and that will only let you cache context for maybe three users.
Fingers crossed our Blackwells show up and Kimi 3 really releases weights, because at some point devs are spending a significant portion of their salary on tokens and it's somehow cheaper to buy these ridiculous DGX servers and rack and run them.
I don't see how these are related.
The accuracy and capabilities of your model are directly related to its size. You need a lot of memory for that.
It will be decades before we get enough useful memory in a phone form factor at a price point people can afford it before something like a frontier model now is useful on the phone.
Now, you can run some models on your phone today.
Either way, Apple is using Google today. That could change, but Google isn't exactly getting out of the TPU business and they've been doing it a long time.
Also, some of you live in a very weird Apple bubble. Apple is not so relevant outside the US.
At 10k context I get about 40 tps generation and 500 tps prefill. At 100k context I get about 25 tps generation and 400 tps prefill.
It works, but I often use gpt or claude to make a detailed enumerated plan of what I want to do first, then have qwen follow it.
I'm not sure if it is economical or not, but I have solar on the roof so the power use is not really an issue and I already have the hardware.
The biggest benefit for me is that it's all done locally, and I know the harness is not uploading anything or sending telemetry to someone else.
Just some thought: Wouldn't it make sense to build some kind of volunteer computing project to train the next-generation LLM by volunteers, similar to the BOINC [1] projects or Folding@home [2]?
N.B.: BOINC was particularly famous for SETI@home (completed), Einstein@Home, Rosetta@home and PrimeGrid.
I still remember the time when Einstein@Home was in its heyday, and many people who loved putting together fast PCs contributed sometimes even for the reason of showing off in the statistics [3].
---
[1] https://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_f...
Chinese release open models to drive the state of the art, OpenAI and crooked Sam do it to keep you down.
More like a generation of models with different specific use cases
And basically any AI company has to sell to US companies or consumers. That'd probs be sufficient to force them to use US models.
As of early June 2026, Opus 4.8 in fast mode cost $50/M output tokens and Opus 4.6 & 4.7 cost $150/M output tokens in fast mode
How can supply and demand explain the price drop? Was it cheaper to serve Opus 4.8? Is the demand for the newer Opus lower than for the older Opus? These are just fixed prices that seem picked out of thin air
Same situation in protectionism in AI. The rest of the world will simply lap us.
Another shill account I gues
https://en.wikipedia.org/wiki/Crypto_Wars https://en.wikipedia.org/wiki/Bernstein_v._United_States
This is where the next wave of lossless ultra-compression comes in - but I believe in Claude Shannon so that’s unlikely.
I’ve been running my own personal k8s cluster on digital ocean for the last 5+ years now and it’s dead simple.
Takes me 30 minutes to create a new namespace a deploy an app, love the bonus of having complete flexibility on my stack too — PVC + SQLite ftw
The value built on that stability was probably worth acquiring, and it’s infra will decay.
First anthropic guardrails blocked totally normal stuff on fable and knocked you down to opus. At this point, they kick you off fable, then opus, then sonnet. Claude then automatically builds up memories of techniques to bypass the guardrails in my long running sessions (the coordinator agent notices the subordinates got shot in the head and their sessions were pulled from context, so it parses out the lost context from ~/.claude json files, then reformulates parts of the task and uses partial results until the guardrail doesn’t trip.
If I were paying for the API, this dance would cost $50-100 a pop, but I’m not, so whatever (for now).
It's extremely expensive to build that and you'll be at least two major model generations behind before you even get your first wafers back. By the time you got your production run ready to go and packaged for market nobody's going to care.
Once we end up going something like 24 months between major advances and capabilities for these models then I can start to see asics for a model being possible.
For instance, I worked at FICO. When I mention this, people wonder what they do. The average person only knows FICO as a “score.”
FICO was founded in Silicon Valley.
The average person doesn’t think about how credit scores work, fundamentally.
It’s just software, at its core. FICO incentivizes certain behaviors.
Ever been banned from an online forum?
Now imagine if a corporation could shut you off from a technology that’s literally indispensable.
Same idea.
Yeeeees but the models are in some sense doubling in performance every 4 months, so I expect this to happen in serious quantities approximately when the economic bubble bursts and investors are no longer willing to pay for training.
(Based on widespread news reporting of the existing impact on US electricity markets, I expect this around the end of this year; but with regards to news reporting I am aware of the Gell-Mann amnesia effect, so if this is as much BS as the water issue turned out to be…)
LLMs are fundamentally tools intended to be useful. But LLM vendors don’t understand their systems well enough to actually price the product people are trying to buy (for example, the actual product of a coding model is the code that it produces, not the tokens, which are just an internal mechanical process involved in the creation of the code). Token based pricing is that lack of understanding leaking out of the organization that ought to be responsible for it, and being dropped on the user.
Imagine if we made cars like this! You’d go to the car dealer and ask for a car. They’d bring you a pile of parts, charge you for them, and try to put them together in front of you. You’d go back and forth for a bit, rephrase where you want the steering wheel, etc. Some of the parts wouldn’t fit but you’d be invited to pay for replacements as well. In the end you’d either have a car or not, that’s your problem.
Are you sure it isn't just another form of Chinese industrial policy? China does not have frontier labs, but through distillation and their own work they can get pretty close. It's not enough to be competitive with Anthropic and OpenAI, but there is still money to be made by selling compute (software as a service), and in any case it's better than being left behind in the AI race.
If you think of docker as "kinda like vms except not really" and k8s as "kinda like deploying and composing docker containers but not really", this may be for you:
https://ojensen.net/infra/understanding-k8s-1
It's actually really neat, i wish i had bothered to learn it years ago.
Its 2026 and its the de facto method of deploying software basically everywhere.
You gotta really work for it to not know what its for by now.
There is something special about complex systems that just-work(tm)
What 15 concepts? You're making me worry that I missed something. It was straight forward: pods, nodes, hw type, lifecycle, deployment. They run almost the same docker as the old ec2s used.
What did I miss? Is there something important I need to read?
Are there any articles you’d recommend for this?
I have Qwen running on an HP Z8. Very nice platform.
I have mine in a sandbox, due to privacy fears.
Your solution sounds more elegant.
The United States if it dares to (I think they will try) but isn’t going to be able to stuff AI models back into the bottle open source is the future and when it comes to AI models yes you’ll be able to customize it to your specifications locally but the genie is out of the bottle. The bull out of the barn and is running down the road.
If United States insist on trying to lock the doors, censor, sanction, the rest of the world will just design and engineer around the United States. Trying to put up a wall, will damaged the United States more particularly with the current performance of Taco. None of the other countries are going to follow the United States not with the current administration they will hedge their bets.
Apple, is using Google now but that will change because the world is probably going down the open path, it’s looking like there was no real rush and no reason to spend so much money on something that’s going to be a commodity in the end, the only hold up is hardware and if it wasn’t for this current memory fiasco, many more people would have access to the hardware that they need to run models locally.
I do think that corporate price gouging is a huge problem that does need to be addressed. But especially with smaller business types — creatives, et al— I still haven’t gotten any grownup answers about what would compel people to get professionally good at something and innovate in the complete absence of copyright: the vastly better business model would be waiting for someone else to do something new and interesting, stealing their work, and then undercutting them in the market because you don’t have R&D/et al costs to recoup. You can’t say that wouldn’t happen because it’s exactly what the AI companies did to billions of people, scoffing at any protest. And ironically, they’re now whining about the Chinese doing it to them.
"You made this? I made this."
Of course a conspicuous architecture would still give it away.
This isn’t as unusual as some people are trying to make it sound. This has been happening since the dawn of finance.
I thought this would be less foreign to everyone since we just went through this whole conversation for a decade with Uber and Lyft. Their demise was predicted from the start from everyone who thought that it was going to collapse as soon as they couldn’t subsidize your rides with promos. There was much wailing and gnashing of teeth as their prices changed to feel out the market. Then they found profitability and the critics went silent.
The foundation of this is thirty years old:
Mark Andreesen founded a company to do what Amazon did: Loudcloud. This was cloud computing, BEFORE AWS was public. Loudcloud was founded in the nineties; AWS opened its APIs in 2006.
Loudcloud failed and became Opsware. Opsware was server automation.
Its competitor was Bladelogic.
Luke Kanies, from BladeLogic, founded Puppet. Puppet was open source, and steamrollered over nearly every installation of BladeLogic and Opsware in existence, because you can’t beat free.
Folks from BladeLogic migrated to jobs at DCOS.
DCOS was steamrollered by Kubernetes, the same way Puppet steamrollered BladeLogic.
The author of the piece predicts that open weight models will steamroller everything next.
I fear he’s right.
I worked for Opsware and BladeLogic.
I watched it happen in real time.
Financially, Andreesen’s wealth stems from Opsware. He is known for Mosaic, but Opsware put him on the map, financially. HP bought them.
If one wants to follow in Andreesen’s footsteps, study how he did it at Opsware.
Conveniently, there is a book.
“The Hard Thing about Hard Things.”
That is some really impressive bubble you are living within. Basically everywhere - nowhere near that, no.
[edit]: Maybe containers, but software in general is so much more broad than containers.
I would say:
- deployments - pods -services - ingress - namespaces - cert-manager - external DNS - external secrets - configmaps - hpa - docker - volumes/pvc
That's the basics
I'll agree that (nearly) everyone should know when Kubernetes is useful, but let's not pretend its the default method for everything. Even then, choosing to deploy on K8s falls on the sysadmins/DevOps I wouldn't expect the devs know or do much more than provide the Dockerfile.
Already running models locally without apple hardware, and have an encrypted vpn tunnel from my mobile devices back to my desktop over the internet. The time is now.
Quantization + KV cache paging + speculative decoding (MPT or draft) is a fairly good mixture here.
Some examples as I don't have access to run tests on a 5060ti right now:
https://njannasch.dev/blog/gemma-4-mtp-vs-qwen-speculative-decoding-5060ti/#vs-qwen-36-mtp
https://www.reddit.com/r/LocalLLM/comments/1umw7vj/dual_5060_ti_16_gb_llm_inference_performance/
27.21.533.298 I slot print_timing: id 0 | task 7843 | n_decoded = 1780, tg = 62.15 t/s
27.24.537.173 I slot print_timing: id 0 | task 7843 | n_decoded = 1965, tg = 62.10 t/s
27.27.541.142 I slot print_timing: id 0 | task 7843 | n_decoded = 2152, tg = 62.11 t/s
I think you underestimate the computational ressources that the mentioned (and similar-kinded) scientific projects needed. Also consider how much computational ressources people invested into cryptocurrency mining.
No, I think the reasons are different:
- Many companies that train AI model use training data which must not be distributed for copyright reasons (and using it is a legal gray zone)x.
- Also consider that the amount of training data is insane. Scientific projects (and cryptocurrency mining, too) have the property that typically the amount of data (storage requirements) is small (or at least the computation can be partitioned so that each sub-task needs little data), but the required computing ressources are insane.
- AI companies consider a huge part of their training data as their "secret sauce" (they often even paid lots of money to generate it, for example by paying world-renowned experts for writing an answer for some important question).
Thus: Yes, the required computation ressources are huge, but this is a problem for which I consider it to be plausible that it can be solved. The real problems are in my opinion different.
I really just iterated over the harness over and over for about two weeks with opencode until I was sort of satisfied (still lots to do there :).
For the llama.cpp I asked claude fable to optimize it for my hardware and iterated a few times. In the end I landed on the following: https://pastebin.com/2PpJFUC0
From my very ignorant standpoint, K8s seems to be about running a "cluster", but I don't know why I would want to do that.
While selling that LLM API usage, they then capture all the prompts, outputs, and intermediate thinking the LLM does, and then sell those logs to the companies making open-weight models. The open-weight model developers then train on those logs to 'distill' a model.
But yes, the reason brand name drugs are drugs are more expensive than generics is due to intellectual property, both the patent and the trademark.
Because that just absolutely murders any competition that manages to not get that level of free money. You're not pouring money in to make it happen at all at that point, you are pouring money in so nobody else can get the part of the pie.
Which is great for investors, bad for everyone else
As I see it, an investment in AI hardware is an investment in my own future.
IE, I drive my car a couple of days a week, and it’s perfectly normal to spend $500 a month on an asset like that. When you factor in the SPACE it takes up, that’s the REAL cost of owning a car: the real estate you have to buy for your car to occupy.
Once that’s factored in, the “true” cost of having a car can easily be $2000 a month, even for a crummy car. The space that the car occupies is expensive.
Yet people balk at spending even $2000 on a GPU.
Makes no sense to me. I choose to invest in the future.
It allows you to stop caring about the individual machines, and just treat them as combined compute, which starts mattering if you leave a single machine setup and need to start thinking about scaling in and out and gluing the individual parts together. Then you have known abstractions to do it.
Of course you can do everything kubernetes does using a bespoke solution, and the concepts aren't new, but having a widely supported technology has a lot of advantages and creating something with even half the feature has a high chance of just being worse.
We used to call that competition.
Imagine making this argument with a straight face in any other industry:
"The claim is that Japanese car companies are buying Ford vehicles, and then leasing them to American consumers at cut-rate prices. In return for the cheap cars, the customers are letting the Japanese observe their driving behavior, studying how they use their F-150 and then the Japanese car companies are applying that data to design new vehicles that will directly replace Ford!"
https://www.project-syndicate.org/commentary/prizes--not-pat...
My point was to whether it was artificial not whether it was bad.
We can have artificial interventions in markets, and that can be a net benefit and a good thing.
I’m not sure why everyone is responding as if “artificial” and “bad” are the same thing
If supply was not artificially restricted (through patents), then competitors would be able to manufacture the drug, supply would expand, the price would collapse, and the original inventor would not be able to recover their R&D costs.
Professionally, my experience is that certain software components need to run together on an individual machine (e.g. database server, app server, web server), and then those machines need to be networked in a certain way (e.g. web server talks to app server, which talks to database server), so I really need to care about the architecture of individual machines. You can then scale this out horizontally (e.g. add another web server) or vertically (e.g. upgrade your database server).
I'm old, so maybe I'm out of date, but having a cluster of "compute" that I can run arbitrary workloads on sounds neat, but is a capability that I've never needed.
I would take a kube cluster over a bunch of VMs I have to hand wire: wire releasing to, managing processes, restarting crashed processes, log aggregation, load balancing, networking, secret injection, cert management, DNS management, monitoring, etc... Any day of the week.
You just don't know what you're talking about, sorry. Kube is really easy now.
That's not what they are observing, it is the behavior of the engines.
In the background kubernetes routes all these docker images with each other without you having to think about this. You may have 1000 db.domain.tld nodes caching a master db host - if you configure that in the docker image, that's not different to bare metal replication.
Same with load balancing in webservers. You can do that! Or you can just have kubernetes handle it. I'm not sure but would expect it to swap images in a way network is optimized - i don't use this kind of software. I just know it's done like this because the bottleneck of modern software is not the local network so its not an issue to have these pieces on different bare metal hosts. And its easier to stay operative if some hosts fail - but you can all solve this by hand.
I work with bottlenecks between network, ssd, ram and vram but if you deploy npm riddled software for >1 Million users well then you may want Kubernetes. Or if you are google and have 10k engineers that need to agree on a standard!
If you can do this yourself with load balancing, replication etc - do it yourself and don't think there's something wrong with that. It's more elegant, efficient - but you have to agree with others how you do it. And that can also be a bottleneck ;)
Still i think you doing it by hand is better so don't worry you are not old you may just have higher standards.
fine, s/observing the way the Ford cars work
Step 2: European company distills or just adjusts the model slightly, and publishes its model on HF
Step 3: American company uses model from step 2. Has to testify under oath where they got it from. "We got it from these French guys"
Or you think all kind of fraud can be committed through some "french guy"?
Also, I am not confident, receiving illegal materials from French guy gates you from personal liability.
For the rest of us, that giant increase in revenue is an increase in our healthcare costs.
As for the American company, it's pretty difficult to check the provedance of open weights. It's even difficult to check the provedance of open source code, because chains of attribution aren't always clear. I posted elsewhere that Anthropic's MCP Python SDK is a fork of an open source project with the attribution removed. We saw the same with Cursor's Composer model, which didn't attribute its Chinese base. It's very hard to claim an American company should be liable for using a purportedly European model with attribution removed.
legislation will block importing Chinese models to the US
> As for the American company, it's pretty difficult to check the provedance of open weights.
government or some companies can build benchmarks/system which will give y/n answer