I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB.

Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

Strange that in the prior art they didn't list DwarfStar, as it is able to run the same model (probably quantized differently though) in less memory. Maybe the author isn't aware of it?

nice! i think the higher HBM on Mi300x is really useful for this kind of thing

we did some work on this for 2xMi300x (kindly referenced in the readme) https://blog.doubleword.ai/deepseek-v4-flash-mi300x. https://hotaisle.xyz/quick-start hotaisle is great for getting Mi300x to experiment with

Another headline of “model runs on x”, which usually means “let’s list how much you give up to run on x”.

Dumbed down quantization?

No. Full intended inference weights preserved, so far so good.

Slow performance?

No again. Looks like you could get over 150 tokens/second.

Give up context window size?

Yes. Original model is trained for and served at 1M, this is 256k. A very practical tradeoff though. Codex is in this range, and quality does start to drop off toward the full size.

This is still quite a bit away from the performance that deepseek gets on their H800. In their DSpark paper they report a throughput of 15k tokens/s/gpu. The MI300 should be able to compete with the H800 so there are probably still quite a few optimizations that can be made.

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params?

BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Unfortunately, the MI300X is an OAM module. The MI350P is the one you want: It's a PCIe card, but it has less memory: 144GB.

Luckily, DeepSeek V4 Flash will run in 144GB too because it's 256 MoE exports are native MXFP4 quantized.

Strange that in the prior art they didn't list DwarfStar, as it is able to run the same model (probably quantized differently though) in less memory. Maybe the author isn't aware of it?

I don't think you can buy a single "MI300X" unit, right? Only the box with x8 of these at a cost of ~250K EUR.

It’s available on demand from a few cloud providers. Seems like the cheapest is AMD Developer Cloud (https://www.amd.com/en/developer/resources/cloud-access/amd-...) powered by Digital Ocean at $1.99/hour.

Edit: Now I think about it, this might be the cheapest way to run the DeepSeek V4 Flash 0731 on a dedicated inference server at original weights. I haven’t run mixed load benchmarks but I guess it’s possible to generate $3-$4 worth of tokens per hour and still maintain a usable per-user throughput.

The MI350p exists and should run a decent quant (say, the ~96GB antirez mix) well, but you can get two rtx pro 6000s for one of these, or 8x (actually more) r9700 + probably the gear to run them, etc.

Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.

I thought it was a consumer grade GPU until I saw the 192GB of HBM and 256GB or RAM.

Give it an AI-bubble pop and these will be flooding the market.

Is their hardware programming interface reasonable for implementing inference of frontier models: no quantization, several tera params?

BTW, how many many params open weight frontier models have? A few teras, 100s of teras?

Kimi-K3: 2.8T

Qwen3.8-Max: 2.4T

DeepSeek V4 Pro: 1.6T

DeepSeek V4 Flash: 284B

(all are total parameter counts, not active parameters)

Otherwise, you can probably buy one of these second hand from somewhere (SXM A100s are available that way) and run it in an adapter board.

I thought it was a consumer grade GPU until I saw the 192GB of HBM and 256GB or RAM.

If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X.

I found an offer to rent two at $1.65 per hour https://spark.enverge.ai/#pricing

The MI300X will vastly outperform it for only a slightly higher price.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

You need to optimize the KVCache part(save to disk to save compute) to achieve this goal.

I thought MI350P wasn't available yet, curious where to source it right now.

To be fair the development of GPUs have stalled over the years. If they kept up with the progress instead of focusing on enterprise market, likely 256GB consumer GPU would be a norm today.

Kimi-K3: 2.8T

Qwen3.8-Max: 2.4T

DeepSeek V4 Pro: 1.6T

DeepSeek V4 Flash: 284B

(all are total parameter counts, not active parameters)

If you have 2x DGX Spark it will run quite nicely. They cost only $8000 or so and use less power so you may be able to rent them cheaper than the MI300X.

I found an offer to rent two at $1.65 per hour https://spark.enverge.ai/#pricing

The MI300X will vastly outperform it for only a slightly higher price.

You need to optimize the KVCache part(save to disk to save compute) to achieve this goal.

To be fair the development of GPUs have stalled over the years. If they kept up with the progress instead of focusing on enterprise market, likely 256GB consumer GPU would be a norm today.

I thought MI350P wasn't available yet, curious where to source it right now.

Give it an AI-bubble pop and these will be flooding the market.

no they won't , the bubble is a financial thing. the demand is real and not going away.

They will be instantly bought out by companies, not individuals. The consumer bubble won’t pop for quite a while yet. Production also won’t ramp up while lack of real competition keeps the demand high.

When is it popping? Is the AI bubble in the room with us now?

Thing is, GPUs will always be on demand, look at their history, initially for gaming, then for hash cracking, then 3D rendering, then for crypto mining, and now AI training and fine tuning. When AI bubble bursts, there will be another bubble taking over.

The only solution is more companies making high end units, only competition will make it better for consumers.

At 830tok/s * 1 hour that's almost 3M tokens which is just $0.54 worth of tokens at Deepseeks current output price.

830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.

You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.

You get privacy for 4 times the cost

How is that economically viable? They are selling at a loss?

This is exactly what I came to say. The price of Flash is so cheap that trying to run it locally or with your own hardware is pointless. I was using it about a month ago to program some stuff and ran it for 4 days non-stop and it cost me about $2.

no they won't , the bubble is a financial thing. the demand is real and not going away.

The only solution is more companies making high end units, only competition will make it better for consumers.

You get privacy for 4 times the cost

When is it popping? Is the AI bubble in the room with us now?

Tomorrow? Next year? In 5 years? Nobody can say. But we do know that AI is overvalued, so it WILL pop.

Next month perpetually

The AI bubble will pop when China gets access to EUV, so the earliest it could happen is 2030

830t/s is burst aggregate. ~500 is sustained and it's for 8 concurrent users. Meaning for $1.99/hour if you serve 8 users it's 8*$0.54, not just $0.54.

You shouldn't rent one out if you're just serving it for yourself, but from a financial standpoint if you sell to users you can take a 100% margin.

How is that economically viable? They are selling at a loss?

Next month perpetually

The AI bubble will pop when China gets access to EUV, so the earliest it could happen is 2030

Bro 500 Aggregate. so that's 500 * 60 * 60 = 1.8M output which is .5$ at best... Not including pre-fill and stuff.

This is not the real margins, even if you are selling to 8 users it's 90 tps per median stream. So assuming that .6-.7$

This is not even remotely worth it.

You need to 3x this tps(~1500 tps) to be worth it, and that's what most providers are doing, at 20-30 users at 50-60 tps with better optimized batch processing and kernels you can make some profit.

Agentic workloads are somewhere around 1%/0.5%/98.5% input/output/cached tokens. Cached tokens are pretty much free for inference providers (if they implement sparse and compressed attention properly) and throughput for input tokens is much higher.

Lets assume that you've got 2 million input tokens, 1 million output tokens and 98.5 million cached tokens to process. That would cost 2 * $0.14 + 1 * $0.28 + 98.5 * $0.0028 = $0.8358 with DeepSeek API pricing.

For comparison, it would take 2M / 8000 + 1M / 800 = 1500 seconds to process this amount of tokens with the linked framework, which is about $0.83 when we assume $2/hr for one MI300X.

However, other inference providers have 10 times higher prices for cached tokens, which results in a comfortable margin.

And we should not discount that DeepSeek also gets paid in data, which is probably more valuable to them.

And I believe that this framework still has some room for optimization for generation with high batch sizes.

They claim their advantage is knowing how to serve their models efficiently, which is quite possible since they design for it.

Use nvidia hardware instead and use a larger cluster serving many more users concurrently. Easily 10x–20x higher token rate per GPU with public solutions like dynamo and sglang.

If you don't do any attention steering, custom decoding or meddle with the weights maybe. Services are worthless unless all you do is write positive prompts.

As others have mentioned, there's the privacy factor as well.

With the cost of electricity, hardware depreciation and tok/s it rarely makes sense to run locally.

> trying to run it locally or with your own hardware is pointless.

Serving local models has advantages other than price. If you work in restricted industries, or have a strong need to protect your IP, or if you just value privacy more than cost, you now have options.