You are doing god's work.

In fact you're also doing the work Nvidia should have done when they put together their (imho) ridiculously imprecise spec sheet.

As long as you're going to add FP8 dense, you could do the same for the parts mentioned in the FP4 section. Divide by two from dense => sparse, and another two for FP4 => FP8.

That gives you 250 tops of fp8 for Spark.

So how many generation T/s we can expect for a dense model?

I assume we can go up to 120B using fp8?

How does the process management comparison work for GPU vs full systems?

So they can sell you the next model which upgrades the RAM capacity.

Memory is the bottleneck. It limits the size of the models you can run and what you pay for.

  Spark: 128 GB LPDDR5x, unified system memory
  5090 :  32 GB GDDR7,

Model sizes (parameter size)

  Spark: 200B 
  5090 :  12B (raw)

It's not good value when you put it like that. It doesn't have a lot of compute and bandwidth. What it has is the ability to run DGX software for CUDA devs I guess. Not a great inference machine either.

5090: 32GB RAM (newegg & amazon lowest price seems to be +300 more)

4090: 24GB RAM

Thor & Spark: 128GB RAM (probably at least 96GB usable by the GPU if they behave similar to the AMD Strix Halo APU)

Note you cannot actually get a 5090 for $1999 that's just the RRP. I believe they actually cost $4k

where does an RTX Pro 6000 Blackwell fall in this? I feel like that’s the next step up in performance (and about the same price as two Sparks)

If you want to run small models fast get the 5090. If you want to run large models slow get the Spark. If you want to run small models slow get a used MI50. If you want to run large models fast get a lot more money.

RTX 5090 is about as good as it gets for home use. Its inference speeds are extremely fast.

The limiting factor is going to be the VRAM on the 5090, but nvidia intentionally makes trying to break the 32GB barrier extremely painful - they want companies to buy their $20,000 GPUs to run inference for larger models.

RTX 5090 for running smaller models.

Then the RTX Pro 6000 for running a little bit larger models (96gb VRAM, but only ~15-20% more perf than 5090).

Some suggest Apple Silicon only for running larger models on a budget because of the unified memory, but the performance won't compare.

What are you trying to do?

it uses LPDDR5x instead of the datacenter variant’s HBM3e.

the GH/GB line is also based around Arm. the CPU here doesn't matter.

Do larger LPDDR5 chips exist yet? Isn't 32GB the max for a 32-bit package?

True... It would be very interesting to make a comparison of various open models based on token generation speed on these platforms. Presumably starting st some size the larger accessible RAM wins out over raw speed but low VRAM? Although I suppose things like MoE and FP would also matter.

It's great at one thing: memory. And that's interesting because memory is a commodity, but they still make bank on just being able to access it.

I just googled it and the first result was one in stock for £2200. That's including tax. I assume $1999 is excluding tax. Without tax and converted to dollars it's $2470.

From other less reliable sources like eBay they are more like £1800.

Also Thor is actually getting sent out to robotics companies already. Did anyone outside Nvidia get a DGX Spark yet?

This has been getting delayed for months. Display out isn't working or something.

Not much. There was a presentation yesterday but it's mostly what we already knew: https://www.servethehome.com/nvidia-outlines-gb10-soc-archit...

One of the first things I looked at too...

No. It's quite common for technical slang to deviate from general vocabulary.

Cf. "compute" is a verb for normal people, but for techies it is also "hardware resources used to compute things".

It should be "run inference on" in my opinion, and would be best shortened IMO to just "prototype, fine-tune, and run".

I'argue that "inference" has taken on a somewhat distinct new meaning in an LLM-context (loosely: running actual tokens through the model) and deviating from the base term to the verb form would make the sentence less clear to me.

I don’t think either of those are right…

What other use cases would use 128GB VRAM but not require higher throughput to run at acceptable speeds?

This thing has a ConnectX-7, which gives it 2 x 200 Gbps networking. The 10 gig port is far from the fastest network interface on the Spark.

You’re almost always going to bottleneck on your home internet or upstream ISP, rather than this local interface. That being said, you aren’t going to be waiting too long either way, depending on download speed. Deepseek R1 is 671GB. Multiply by 8 to get into bits: 5368Gb At full 10gbps (which, again, you probably won’t get): 5368Gb / 10gbps = 537 seconds to download 537s / 60 = 8.95 minutes. Call it 10m with overhead.

No. These are practically useless for AI.

Their prompt processing speeds are absolutely abysmal: if you're trying to tinker from time to time, a GPU like a 5090 or renting GPUs is a much better option.

If you're just trying to prep for impending mainstream AI applications, few will be targeting this form factor: it's both too strong compared to mainstream hardware, and way too weak compared to dedicated AI-focused accelerators.

-

I'll admit I'm taking a less nuanced take than some would prefer, but I'm also trying to be direct: this is not ever going to be a better option than a 5090.

There are cheaper ASUS and MSI versions "coming soon" with the same chip and less storage / memory.

That was their Digits box.

That's very true and what's segmenting the market, but I don't understand why you're saying the 5090 supports only 12B model when it can go up to 50-60B (= a bit less than 64B to leave room for inference) as it supports FP4 as well.

I thought the 6000 was slightly lower throughput than 5090, but obviously has a shitload more RAM.

The 395 although it can be supplied with 128GB can’t use all that for VRAM (unless something has changed in the last couple of weeks).

My understanding is the DGX Spark is optimized for training / fine tuning and the Jetson Thor is optimized for running inference.

Architecturally, the DGX Spark has a far better cache setup to feed the GPU, and offers NVLINK support.

Once the updated Mac Studio with M4/M5 Ultra comes out, pretty much going to make the DGX irrelevant right?

It's the same as Strix Halo and M4 Max that people are going gaga about, so either everyone is wrong or it's fine.

Ordered one in spring. Delivery time was pushed from July to September. Apparently they had a bug in the HDMI output.

It does say `Experience up to 1 petaFLOP of AI performance at FP4 precision with the NVIDIA Grace Blackwell architecture.` in the features section.

But yeah, this should have been further up.

It can write Crysis.

If raspberry pi can then this probably also can.

https://www.jeffgeerling.com/blog/2024/amd-radeon-pro-w7700-...

It’s gonna run Crysis for my wallet alright

from https://www.servethehome.com/this-is-the-asus-ascent-gx10-a-...

  ASUS and NVIDIA told us that their GB10 platforms are expected to use up to 170W.

[edit] the PSU is 240W so that'd place an upper limit on power draw, unless they upgrade it.

Perhaps "infer from"? I was also taken aback by how they just decided to make "inference" a verb, though. A decent writer would have rewritten the sentence to make it work, similar to how a software implementation sometimes just doesn't work out. But apparently that's too much to ask from Nvidia marketing.

Funnily enough things like this show that a human probably was involved in the writing. I doubt an LLM would have produce that. I've often thought about how future generations are going to signal that they are human and maybe the way will be human language changing much more rapidly than it has done, maybe even mid sentence.

Came in here to say the same thing. Have bought 3 Nvidia dev boards and never again as you quickly get left behind. You're then stuck compiling everything from scratch.

My experience with Jetson Nano was that it had to have its Ubuntu debloatred first (with 3rd party script) before we could get their NN something library to run the image recognition, designated to run on this device.

These seem to be highly experimental boards, even though are super powerful for their form factor.

RDNA3 CUs do not have FP8 support and its INT8 runs at the same speed as FP16 so Strix Halo's max theoretical is basically 60 TFLOPS no matter how you slice it (well it has double INT4, but I'm unclear on how generally useful that is):

    512 ops/clock/CU * 40 CU * 2.9e9 clock / 1e12 = 59.392 FP16 TFLOPS

Note, even with all my latest manual compilation whistles and the latest TheRock ROCm builds the best I've gotten mamf-finder up to about 35 TFLOPS, which is still not amazing efficiency (most Nvidia cards are at 70-80%), although a huge improvement over the single-digit TFLOPS you might get ootb.

If you're not training, your inference speed will largely be limited by available memory bandwidth, so the Spark token generation will be about the same as the 395.

On general utility, I will say that the 16 Zen5 cores are impressive. It beats my 24C EPYC 9274F in single and multithreaded workloads by about 25%.

NVidia Spark is $4000. Or, will be, supposedly whenever it comes out.

Also notably, Strix Halo and DGX Spark are both ~275GBps memory bandwidth. Not always but in many machine learning cases it feels like that's going to be the limiting factor.

Bit of a non-sequitur, but it seems like it's been playable via box86 for quite a while now: https://youtu.be/NcBJG3z8kF0

If you scroll down a little and see the chip icon, where it says "NVIDIA GB10 Superchip " it also says "Experience up to 1 petaFLOP of AI performance at FP4 precision with the NVIDIA Grace Blackwell architecture."

Further down, in the exploded view it says "Blackwell GPU 1PetaFLOP FP4 AI Compute"

Then further down in the spec chart they get less specific again with "Tensor Performance^1 1 PFLOP" and "^1" says "1 Theoretical FP4 TOPS using the sparsity feature."

Also, if you click "Reserve Now" the second line below that redundant "Reserve Now" button says "1 PFLOPS of FP4 AI performance"

I mean I'll give you that they could be more clear and that it's not cool to just hype up on FP4 performance, but they aren't exactly hiding the context like they did during GTC. I wouldn't call this "disingenuous"

> Ryzen AI Max 395+, ~120 tops (fp8?), 128GB RAM, $1999

Just got my Framework PC last week. It's easy to setup to run LLMs locally - you have to use Fedora 42, though, because it has the latest drivers. It was super easy to get qwen3-coder-30b (8 bit quant) running in LMStudio at 36 tok/sec.

Maybe the real value of the DGX spark is to work on Switch 2 emulation. ARM + Nvidia GPU. Start with Switch 2 emulation on this machine and then optimize for others. (Yeah, I know, kind of expensive toy).

  Mac Studio max spec, ~120 tflops (fp16?), 384GB RAM, 3x bandwidth, $9499

512GB.

DGX has 256GB/s bandwidth so it wouldn't offer the most tokens/s.

Fine tuning text to image/video models perhaps?

For the newest models unless you quantize the crap out of them, even with a 5090 you’re going to be swapping blocks, which slows things down anyways. At least you’d be able to train on them at full precision with a decent batch size.

That said, I can’t imagine there’s enough of a market there to make it worth it.

I have 25 Gbps symmetric ethernet from my ISP (not XGPON). they are talking about rolling out 100 Gbps.

on the reserve now page for USA, there's:

  ASUS Ascent GX10 - 1TB $2,999

  MSI EdgeXpert MS-C931 - 4TB $3,999

the 1TB/4TB seems to be the size of the included NVMe SSD.

the reserve now page also lists

  NVIDIA DGX Spark Bundle
  2 NVIDIA DGX Spark Units - 4TB with Connecting Cable $8,049

The DGX Spark specs lists an NVIDIA ConnectX-7 Smart NIC which is rated at 200Gbe to connect to another DGX Spark, for about double the amount of memory for models.

It's the same thing. They renamed it from Digits to Spark.

Digits is no more, it’s DGX. Source: signup link in the press release (https://nvidianews.nvidia.com/news/nvidia-puts-grace-blackwe...) now goes to DGX Spark preorder (https://www.nvidia.com/en-us/products/workstations/dgx-spark...)

There's a lot of segmentation going on in the Blackwell generation from what I'm told.

From YouTube it seems up to 105gb Models disksize work, yes.

In Linux, you can set it as high as you want, although you should probably have a swap drive and still be prepared for you system to die if you set it to 128GiB. Here's how you'd set it to 120GiB:

    # This is deprecated, but can still be referenced
    options amdgpu gttsize=122800

    # This specifies GTT by # of 4KB pages:
    #   31457280 * 4KB / 1024 / 1024 = 120 GiB
    options ttm pages_limit=31457280

Memory Bandwidth:

Nvidia DGX: 273 GB/s

M4 Max: (up to) 546 GB/s

M3 Ultra: 819 GB/s

RTX 5090: ~1.8 TB/s

RTX PRO 6000 Blackwell: ~1.8 TB/s

The other ones are not framed as an “AI Supercomputer on your desk”, but instead are framed as powerful computers that can also handle AI workloads.

But can you hook that up to a normal PC?

I think I interviewed you the other day and you didn’t get the job…

  Their prompt processing speeds are absolutely abysmal

They are not. This is Blackwell with Tensor cores. Bandwidth is the problem here.

It's more throughput, but way less value and there's still no NVLink on the 6000. Something like ~4x the price, ~20% more performance, 3x the VRAM.

There's two models that go by 6000, the RTX Pro 6000 (Blackwell) is the one that's currently relevant.

Its for comparison using raw, non optimized models. Both can do much better when you optimize for inference.

Information is in the ratio of these numbers. They stay the same.

Ultras are pretty expensive.

I run 4 Mac Studio ultras at work (they’re pricy when maxed out), for local-first AI dev services. But there’s a few things that make me want to switch to the Spark. Networking is the biggest one, the Macs have Thunderbolt and Ethernet, but if I run distributed inference with EXO over Thunderbolt; the drop in tokens/second is massive. These Sparks get RDMA and can stack nicely. The other big one is access to CUDA, MLX has come a long way but being able to have CUDA and GPU access in containers would simplify the stack so nicely. If I had a USB-C/Thunderbolt backplane it might compare, but scaling with the Spark is likely a lot more straightforward.

I call the stack with Mac Studios “MacAIver” because it feels like a duct tape solution, but the Spark equivalent would likely be more elegant.

If that would be true why aren't Mac sales banned in China instead of Nvidia GPUs?

M4 max has more than double the bandwidth.

Strix Halo has the same and I agree it’s overrated.

That's eerily similar to what happened to Qualcomm's failed Snapdragon X Elite dev kit. That one eventually shipped in small quantities with a Type-C to HDMI dongle in the box to make up for the built-in HDMI port going missing. Then Qualcomm cancelled the whole project and refunded everyone, including people who had already received their hardware.

Even if that "sparsity feature" is that two or of every four adjacent values in your areay be zeros, and that performance halves if not doing this?

I think lots of children are going to be very disappointed running their blas benchmarks on Christmas morning and seeing barely tens of teraflops.

(For reference see how the still optimistic numbers are for the H200 when you use realistic datatypes.

https://nvdam.widen.net/s/nb5zzzsjdf/hpc-datasheet-sc23-h200... )

I'm pretty new to this, so if I wanted to benchmark my current hardware and compare to your results what would be the best way to do that?

I'm looking at going for a Framework Desktop and would like to know what kind of performance gain I'd get over the current hardware I have, which so far I have a "feel" for the performance of from running Ollama and OpenWebUI, but no hard numbers.

Hi could you share if you get a decent coding performance (quality wise) with this setup? IE. Is it good enough to replace say Claude Code?

Very encouraging result, I'm waiting super anxiously for mine! How much memory did you allocate for the iGPU?

I think you can get something a lot cheaper if that’s all you want, e.g. something in the Jetson Orin line. That’s more similar to the switch, also, since it’s a Tegra CPU.

tokens/s/$ then.

You were complaining about speed. Yes, a PC can have the same ports, and then you get much faster speeds than Thunderbolt can provide.

Why would you ever want a DGX Spark to talk to a “normal PC” at 40+ Gbps speeds anyways? The normal PC has nothing that interesting to share with it.

But, yes, the DGX Spark does have four USB4 ports which support 40Gbps each, the same as Thunderbolt 4. I still don’t see any use case for connecting one of those to a normal PC.

Yes. Just buy the Mellanox card. We had a bunch of ConnectX 5 hooked up through SFP. Needs cooling but fast.

why criticize something in the first place when you clearly have not even looked at the product?

What?

Perhaps they are referring to default GPU allocation that is 75% of the unified memory, but it is trivial to increase it.

They're abysmal compared to anything dedicated at any reasonable batch size because of both bandwidth and compute, not sure why you're wording this like it disagrees with what I said.

I've run inference workloads on a GH200 which is an entire H100 attached to an ARM processor and the moment offloading is involved speeds tank to Mac Mini-like speeds, which is similarly mostly a toy when it comes to AI.

Ok then just to clarify: you can fit 4x larger models on the Spark vs 5090, not 17x.

Those are high end GPUs that aren’t comparable

because they realized it sucked.

the RTX Pro 6000 (Blackwell) does not have NVlink? if so, what the fuck Mr.leather jacket.

I mean the spark is $3,999 and current M3 Max 28-Core CPU 60-Core GPU is the same price. I would expect the refreshed studio will stay around the same price.

You'd have to stack 16 of these to get 2TB of VRAM, equivalent to 4 Mac Studios 512GBs chained together.

16 compared to 4. Surely even much faster networking in the Spark would degrade with that many devices?

Biggest problem with Macs is that they don't have dedicated tensor cores in the GPU which makes prompt processing very slow compared to Nvidia and AMD.

I would expect/hope that DGX would be able to make better use of its bandwidth than the M4 Max. Will need to wait and see benchmarks.

Because Tim bribed Trump with a golden calf, or more seriously it's easier to ban a component and its manufacturer vs broader systems.

Of course it doesn't; artificial segmentation because they really want you to buy their even more expensive datacenter GPUs for AI training.

n.b. there's been a little speculation that Apple adding TensorOps to Metal 4 suggests M5/M6 may get tensor cores.

https://x.com/liuliu/status/1932158994698932505

https://developer.apple.com/metal/Metal-Shading-Language-Spe...

Not unprecedented though, Playstation 2 had export restrictions.

But it was a different time. Most policies had some connection to the subject at hand.

Policies today are all about brand Trump and brand MAGA.

It should. It has tensor cores which should drastically improve prompt processing. It should also be highly optimized for most AI apps.

funny to hear the tards that haven’t listened to any of the policy screeches pretend like they know what’s going on.

The GPU memory allocation refers to how capacity is alloted, not bandwidth. Sounds like the same 256-bit/quad-channel 8000MHz lpddr5 you can get today with Strix Halo.

Again, prompt processing isn't the major problem here. It's bandwidth. 256GB/s bandwidth (maybe ~210 in real world) limits the tokens per second well before prompt processing.

Not entirely sure how your ARM statement matters here. This is unified memory.

In Germany the 96gb version is 5000 EUR and the 256gb version is 7000 EUR (no 128gb available as far as I can see).

Matrix vector multiplication for feed forward layers is most of the bandwidth as I understand things, there's not really a way to do it "better", its just a bunch of memory-bound dot products.

(Posting this comment in hopes of being corrected and learning something).

384GB is 75% of 512GB. The M3 Ultra bandwidth is over 800GB/s, though potentially less in practice.

Using an M3 Ultra I think the performance is pretty remarkable for inference and concerns about prompt processing being slow in particular are greatly exaggerated.

Maybe the advantage of the DGX Spark will be for training or fine tuning.

Are you comparing prices with or without taxes? US usually prices without and EU with.

At that point it's far superior to fly to the US, buy it, and fly back. Hell, have a nice week in a hotel and bring two.

Training is performed in parallel with batching and is more flops heavy. I don't have an intuition on how memory bandwidth intensive updating the parameters is. It shouldn't be much worse than doing a single forward pass though.

The problem is different parts of the SoC (CPU, GPU, NPU) may not actually be able to consume all of the bandwidth available to the system as a whole. This is why you'd need to benchmark - different chips may be able to feed the cores better than others.

I very consistently see people say prompt processing is slow for larger context sizes ("notoriously slow"), something that is much less of an issue with eg CUDA setups.