it allows you to run smaller models much better

imo 3090s make the most sense if you can buy at least 2x ideally 4x but of course we're talking about a completely different budget at that point

And if your jobs do fit onto a 24GB card, then you are not the target user for the "AI mini PC" niche that these guys are trying to carve out

Plus a reasonably inexpensive super low-latency interconnect.

It's great to get lots of tokens, but being able to handle and extent context is why it'll continue to be a great machine compared to any of the small graphics cards.

However for local single-user setups, it's often better to have access to more capable/bigger MoE models at reasonable speeds and lower concurrences, which is enabled by these platforms.

Only thing that separates them is the build quality and the extra 20W of boost the framework desktop and this variant support.

They have a note on the thermals but no measurement of noise. Doesn't matter if it's stricly a whoosh or a whine, only if they bother people in the same room. And the small ones like Bosgame get a consistent complaint about the noise in in-depth youtube videos.

The framework desktop even has a usable PCIe 4x slot available if you put the board in a different case. They sell the 128GB board on its own for $3150.

With a desktop your system memory is slow and your fast graphics memory is limited in size.

To me it seems like the best bang for your buck in the BYO desktop PC space is to get a board with dual PCIe slots then find some old generation 24GB GPUs like RTX 3090.

But you’re not getting access to more than 48GB of fast memory without something similar to this or a Mac Studio.

128 bit: 96 GB?

256 bit: 192 GB

512 bit: 384 GB?

1024 bit: 768 GB?

https://community.frame.work/t/was-there-no-possible-way-to-...

With the current RAM and SSD prices... I rather a bit later.

I do (and have historically done) quite a work with both local LLMs and local diffusion models. I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s

While the M3 Max MBP can serve up MoE reasonably fast (~60 token/sec)the RTX 4090 is an entirely different experience (~170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX.

The raison d'etre for the DGX spark is not practical home inference, but rather offering the same fundamental architecture as data center cards for a affordable CUDA prototyping. If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. The DGX spark solves this by having the same fundamental setup as what those cards have.

That makes these non-NVIDIA DGX-like devices confusing to me. The entire benefit of the DGX series is the NVIDIA architecture itself.

Anyone interested in home LLMs should decide whether a Mac or a dedicated GPU is the more sensible path based on their budget and other computer use. Each has their own benefits.

For anyone considering these devices, the only reason I would recommend against them is if you plan on getting multiple to link together - the DGX Spark has a much, much faster interconnect bandwidth ceiling than the AMD devices do.

Otherwise, they're great!

You'll need a custom-built distro image, but that goes for like 90% of ARM hardware on Linux.

Open, cheap & good enough will win the race.

Now I think it's totally fine to have a less capable offering, and the Strix Halo is still a mighty capable machine for inference on mid-size MoEs. At 2k it was a tinkerer's dream. But the performance difference should be reflected in the price. This is roughly a doubling of the price compared to less than a year ago without adding any notable features, it's appalling.

> A shame, really, as the Ryzen 7640U, 7840U, 7840HS, and 7940HS all support 256GB of RAM.

To be fair, those platforms support dual dimms per channel, which Strix Halo would not, at least not at it's high speeds.

But reciprocally Gorgon Halo 400 just launched and it supports... 192GB. And is the exact same APU.

Memory chips did finally have their first big doubling per chip semi recently (available last February), with 48 & 64GB dimms becoming available. There is some reasonable lag here, that Strix Halo & Gorgon Halonuse lpddr5x, which perhaps had some lag, that 32GB (x4) was the best available. But now with Gorgon Halo being 192GB capable but not 256GB, it sure feels looks & seems like this is just bad spirited fuckery from AMD. https://forum.level1techs.com/t/where-are-the-ddr5-unbuffere...

Waiting for the market to be less insane is somewhat akin to waiting for the s&p500 to drop a decent amount so you can buy in.

PS6 "undertaker of physical media" will supposedly be priced >$1k: https://youtu.be/-F1JS-4Abjo

[0] https://en.wikipedia.org/wiki/Apple_M4

Microsoft = yes, they care enormously, as Surface has taken away many sales. Albeit they sold some ChromeBooks

Isn't adding pins kind of expensive?

This is one of the main reasons (the other is the number of PCIe lanes) why high end desktop and server CPUs have like double the number of pins and so much bigger sockets as compared to consumer desktop CPUs.

Which is a huge problem? Even using 2x memory controllers in typical consumer motherboard can make system very unstable.

The differences are basically, sparks require ARM and sparks allow interconnects; so if you do have dreams of electric sheep to chain them together, you're not gonna get the AMD halo units.

But if you just want to putz around with a dev machine and do other things, not sure you'd want a spark.

lol this is so wrong it's funny - equities go up in price, commodity goods go down in price. the two markets are literally diametrically opposed.

And as for DRAM channels, typical cheap motherboard has 2 channels and 4 slots, it should not be super difficult to add 2 more channels.

Yes they spent those costs to switch from DDR4 to over-priced DDR5 and I suggested the cost could be spent on adding more DDR4 channels instead.