Right now the speed isn't good for GLM 5.2, Deepseek V4 Flash speed is okay for me (actually reading the output) and quite usable. See kyuz0's great recent video here: https://www.youtube.com/watch?v=PkKXm_mKCCM
With a bit more speed and model improvements, local AI becomes a reasonable practical thing! The biggest problem is all the tech companies making consumer hardware completely unaffordable, and I don't think this is accidental. Look at Micron's profits and share price lately...
I got my Strix machines for ~2k eur each, best computers this 90s kid has ever owned, but those days are gone :(
I do hope that apple opens up RDMA for their TB4 machines... ds4 using TB5 macs works great - but there are a lot of capable tb4 (M2/1) machines out there and afaik there's no hardware limitation preventing RDMA from working (at lower bandwidth, but with the latency gains!) on the older stuff.
The device was not perfect by any means, but the ability to run fairly large models is some kind of magic.
I'm working on a three node strix halo agentic OS factory designed to be maintained by local agents: https://github.com/projectbluefin/testing-lab
This memory bandwidth combo is amazing for homelabbers. kyuz0's work on these containers has made the investment in this kit so valuable I hope Framework is sending you hardware!
https://projectbluefin.io/server/ is what I'm hoping to ship, designed to just ship setups like this ootb and things like this would be so much harder without kyuz0!
(Note: The 64GB ones are going for $1700-ish empty, the prices on the 128's are outrageous we can just keep making the labs more deterministic over time!)
Would love to see DeepSeek V4 flash/pro and MiniMax M3 benchmarks but already these are pretty impressive, first strix Halo setup I've seen with some serious performance.
EDIT: Apologies - I think I misunderstood these benchmarks - it seems this is actually very slow when compared to a M4 or M5 chip with a good amount of memory. Looking at the creators video here: https://youtu.be/Cfl3TS7ME5s?t=734 -- it seems the performance of strix halo is much much slower than I get on my M4 MBP - which gets ~400 prefill and ~20 tok/s generation
- 2x Framework Desktop AI Mainboards with 128GB of RAM for $3150 each
- 2x 100G Ethernet controllers for ~$500 each
So the Framework board has a single PCI-e 4.0 x4 slot, which amounts to 8GB/s or 64Gbps theoretical so you're not getting 100G. Also, the 100G cards all seem to be PCI-e x16 slots for obvious reasons so you need a riser or an adapter or something to even get them to work.
I don't know how hot a 100GbE copper NIC runs but, from experience, 10GbE NICs have been basically giant heatsinks, basically. So fiber might be advisable and I expect short fiber cables here probably aren't cost-prohibitive given everything else.
As an aside, if you are using Ethernet for clustering and you're clustering 2 devices, in an ideal world you'd be using simplex Ethernet but that's not an option here.
I wonder if the author considered USB 4.0 for clustering? I ask because I know people who have clustered Mac Studios over TB5 and that bandwidth is up to 120Gbps. The version of USB4 on the Ryzen AI 395 seems to be 40Gbps, which isn't that far off 8GB/s over PCI-e 4.0 x4.
But the limiting factor with Strix Halo (and DGX Spark for that matter) is memory bandwidth, both under 300GB/s. The obvious comparison is to the Mac Studio. Unfortunately the largest spec they currently sell is 96GB. It had been as high as 512GB. And 96GB is $6700+ but you're also getting way better performance AFAICT eg [1]. The M3 Ultra has ~900GB/s memory bandwidth.
You can alternatively buy a Macbook Pro with M5 Max and 128GB of RAM (now $8000, was $5500-6000 a few days ago) but that tops out at ~600GB/s, which is still double these mini AI boxes.
Oh and if you don't want to go the way of these Framework motherboards, you can buy a whole 128GB Strix Halo PC for $3k or less.
I think the main point here though is we're only a few years away from running 300B+ (or even 1T+) param models at useful speeds on enthusiast hardware.
[1]: https://www.reddit.com/r/LocalLLaMA/comments/1u5mfaq/you_can...
In 1-3 years the hardware crunch will be over, local distilled models will provide Opus 4.8 like intelligence, and the hardware will exist to provide usable performance.
Thats the problem with these AMD laptop class cores, they have very little IO. They have been saying they will release in a desktop form factor, but then it probably wont have such good memory bandwidth...
The Nvidia boxes have 200Gb ethernet thats much more useful for clustering.
Another note: In my experience, RoCE works much better on CX4+ generation. CX3 is best with Infiniband. I think some firmwares on the CX3 generation, has a messed up config for RoCE. But running Infiniband is not a complex task, is way easier than people think, like 10x easier and faster to setup than Ethernet.
It's not even worth it at that point.
You can get a used enterprise grade SXM baseboard with 4-8 V100/A100 GPUs off eBay at a similar price. That will even get you actual HMB ram and NVlink. Along with 10x the AI performance, assuming you don't care about your electricity bill of course.
Or maybe you're right, I originally remembered 2k as well. I wanted to wait for the AI Max 395+ upgrade of my laptop, and now it makes no sense to upgrade.
You realize "tech companies" isn't a monolith? Micron charging inflated prices doesn't magically benefit OpenAI. The "high prices keep out competitors" theory doesn't make much sense either. It's like saying Dennys benefits from higher egg prices because it makes cooking eggs at home more expensive.
Only if you pay the Framework premium.
https://www.bosgamepc.com/products/bosgame-m5-ai-mini-deskto...
I don't have access to the USD price, but it's 2500€ (tax included), up from 1600€ in November when I ordered mine.
It’s classic capex vs opex. I’d keep paying my openai subscription instead of dropping $3k to run a subpar model. If the thing costs $1k I would consider it.
[Tangent: all my life I've been downvoted into a smoking hole in the ground, particularly on reddit r/hardware, for questioning the wisdom of laptops for high performance computing, including gaming. Everyone insists they need the mobility, and then just leave it plugged in the whole time, absolutely refusing to admit it's about aesthetic preference.]
AI + 100GBE (under load) + tiny box = unreliable and eead very quickly.
Honestly I think this is just a bad time to be buying hardware - everything is marked up an insane amount that doesn't really make sense.
And could you not use something like an N5 + iSCSI for storage?
Why do you do all this? To avoid collisions and the loss of effective bandwidth from back-offs.
It only really works with 2 computers because if you add a 3rd, now you need 12 NICs instead of 4 for unidirectional point-to-point connections.
With the laptop you probably won't get silent operation at the peak 100-140w, i.e. you've now massively overpaid for lower performance.
This guide details how to configure a two-node AMD Strix Halo cluster linked via Intel E810 (RoCE v2) for distributed vLLM inference using Tensor Parallelism.
On Both Nodes:
ethtool -i <iface>).iommu=pt, pci=realloc, etc.).192.168.100.1 & .2), set MTU 9000, and trust the interface in firewall../refresh_toolbox.sh (this automatically installs the container with RDMA support and the custom librccl.so patch).start-vllm-cluster.HF_TOKEN first for gated models!)Key Note: The refresh_toolbox.sh script detects your Infiniband/RDMA devices and automatically configures the container to expose them.

To fully utilize the Strix Halo cluster, it is helpful to understand the technologies involved:

Perform these steps on the Host OS (Fedora 43) of both nodes.
Tested Host Configuration:
| Node | Kernel | OS | IP (RDMA Interface) |
|---|---|---|---|
| Node 1 | 6.18.5-200.fc43.x86_64 |
Fedora Linux 43 | 192.168.100.1/30 |
| Node 2 | 6.18.6-200.fc43.x86_64 |
Fedora Linux 43 | 192.168.100.2/30 |
Note: These specific kernel versions were verified to work. Fedora 43 is recommended.
Install the core RDMA userspace tools. You do not need proprietary Intel drivers; the in-kernel drivers work perfectly.
iceirdma (Unified driver for RoCE v2 & iWARP)sudo dnf install rdma-core libibverbs-utils perftest
rdma-core: The userspace components for the RDMA subsystem (libraries, daemons, and configuration tools).libibverbs-utils: Utilities for querying RDMA devices (e.g., ibv_devinfo).perftest: A suite of benchmarks (e.g., ib_write_bw, ib_send_lat) to verify RDMA bandwidth and latency.Use ethtool to check the current firmware version of your Intel E810 card.
ethtool -i enp194s0np0
Recommended Firmware:
Ensure your firmware is at least as new as the version shown below (Firmware 4.91...). If your firmware is older, please update it using the Intel® Ethernet NVM Update Tool for E810 Series.
Example Output:
driver: ice
version: 6.18.5-200.fc43.x86_64
firmware-version: 4.91 0x800214b5 1.3909.0
expansion-rom-version:
bus-info: 0000:c2:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
This guide assumes a subnet of 192.168.100.0/30.
Identify your interface:
Run ip link to find your 100GbE card (e.g., enp194s0np0).
Node 1 (Head - 192.168.100.1):
# Bring link up
sudo ip link set enp194s0np0 up
# Assign IP
sudo ip addr add 192.168.100.1/30 dev enp194s0np0
# Set MTU (Jumbo Frames)
sudo nmcli connection modify "rdma0" ethernet.mtu 9000
sudo nmcli connection up "rdma0"
Node 2 (Worker - 192.168.100.2):
# Bring link up
sudo ip link set enp194s0np0 up
# Assign IP
sudo ip addr add 192.168.100.2/30 dev enp194s0np0
# Set MTU
sudo nmcli connection modify "rdma0" ethernet.mtu 9000
sudo nmcli connection up "rdma0"
Verify Routing: Ensure the route exists on both:
sudo ip route add 192.168.100.0/30 dev enp194s0np0
Verify Link:
rdma link
# Output should show: state ACTIVE physical_state LINK_UP used_usec X ...
1. BIOS Settings: Set the iGPU Memory Allocation to the minimum possible (512MB). We will use the GTT (Graphics Translation Table) to dynamically allocate system memory as "Unified Memory" for the GPU.
2. Kernel Parameters: Update GRUB to enable unified memory, optimize RDMA performance, and fix PCI resource allocation.
Edit /etc/default/grub and append to GRUB_CMDLINE_LINUX:
iommu=pt pci=realloc pcie_aspm=off amdgpu.gttsize=126976 ttm.pages_limit=32505856
Explanation of Parameters:
iommu=pt: Sets IOMMU to "Pass-Through" mode. This is critical for performance, reducing overhead for both the RDMA NIC and the iGPU unified memory access.pci=realloc: Reallocates PCI BARs. Often needed on consumer platforms to properly map large address spaces for devices like the E810 or Strix Halo.pcie_aspm=off: Disables PCIe Active State Power Management. Prevents latency spikes and link negotiation issues on the 100GbE connection.amdgpu.gttsize=126976: Caps the GPU GTT size to ~124GiB (126976MB). This defines how much system RAM the GPU can address as its own "VRAM".ttm.pages_limit=32505856: Limits the Translation Table Manager to ~124GiB (in 4KB pages), matching the GTT size.3. Apply Changes:
sudo grub2-mkconfig -o /boot/grub2/grub.cfg
sudo reboot
Applications like Ray and NCCL use random high ports. It is easiest to trust the internal RDMA interface completely.
# Assign the interface to the trusted zone permanently
sudo firewall-cmd --permanent --zone=trusted --add-interface=enp194s0np0
# Reload firewall
sudo firewall-cmd --reload
The cluster management and verification scripts rely on SSH to execute commands on remote nodes. You must configure passwordless SSH between both nodes (root or sudo-enabled user).
ssh <other-node-ip> date from each node. It should print the date without asking for a password.The toolbox container provided in this repo includes a critical patch: a custom-built librccl.so that enables gfx1151 (Strix Halo) support for RDMA (https://github.com/kyuz0/rocm-systems/tree/gfx1151-rccl), which is currently missing in upstream ROCm packages. This library is automatically compiled using the build-rccl GitHub Action in this repository, which generates the artifact that is then bundled into the Docker container.
To install the toolbox on both nodes, run:
./refresh_toolbox.sh
What this does:
kyuz0/vllm-therock-gfx1151 image./dev/infiniband exists on your host./dev/dri, /dev/kfd (Required for ROCm)/dev/infiniband, --group-add rdma--ulimit memlock=-1 (Required for DMA)Before proceeding to run the cluster, verify that RDMA is active and providing low latency (~5µs vs ~70µs for Ethernet).
Run the provided verification script from the Head Node:
# Inside toolbox
/opt/compare_eth_vs_rdma.sh
Expected Results:
Path Latency Bandwidth
------------------------------------------------
Ethernet (1G LAN) 0.074 ms 0.94 Gbps
Ethernet (RoCE NIC) 0.068 ms 55.70 Gbps
RDMA (RoCE) 5.23 us 50.64 Gbps
Note the massive latency drop (milliseconds to microseconds) for RDMA.
A TUI utility, start-vllm-cluster, is provided to manage the Ray cluster and vLLM.
toolbox enter vllm
start-vllm-cluster
192.168.100.1 and Worker is 192.168.100.2.# Head
export NCCL_SOCKET_IFNAME=<rdma_iface>
ray start --head --node-ip-address=192.168.100.1 ...
# Worker
ray start --address=192.168.100.1:6379 ...
2.0 GPU).Once the cluster is active (checked via Option 3):
Meta-Llama-3.1-8B-Instruct).2 (one GPU per node).131072).YES if you are restarting after a crash.YES.Important Gotchas:
google/gemma-2-27b-it are "gated" and require you to request access on Hugging Face.export HF_TOKEN=your_token_here
start-vllm-cluster
If you see link issues, ensure your Intel E810 firmware is up to date using the Intel standard tools.
gfx1151 support in upstream RCCL.If you do not have dedicated 100GbE RDMA network cards, you can directly connect the two nodes using a high-quality Thunderbolt 4 / USB4 cable. This will create a thunderbolt0 network interface.
While it lacks the ultra-low microprocessor-level latency of RDMA, it provides significantly more bandwidth than standard 1GbE/5GbE Ethernet and is easier to configure.
Note:
thunderbolt-netrelies on standard OS kernel TCP/IP stacks.
1. Establish Connection: Connect the nodes directly using a certified Thunderbolt 4 or USB4 cable. Verify the link is active:
ip link show thunderbolt0
2. Network Configuration (Head - Node 1):
Configure a persistent connection using nmcli with a static IP and Jumbo Frames (reduces CPU overhead).
Note: Jumbo Frames may be unsupported on some Thunderbolt host controllers.
sudo nmcli connection add type ethernet ifname thunderbolt0 con-name thunderbolt0 ipv4.method manual ipv4.addresses 192.168.2.1/24 mtu 9000
sudo nmcli connection up thunderbolt0
3. Network Configuration (Worker - Node 2):
sudo nmcli connection add type ethernet ifname thunderbolt0 con-name thunderbolt0 ipv4.method manual ipv4.addresses 192.168.2.2/24 mtu 9000
sudo nmcli connection up thunderbolt0
4. Firewall Rules: To ensure Ray and NCCL can communicate freely over this link:
# Assign the interface to the trusted zone permanently
sudo firewall-cmd --permanent --zone=trusted --add-interface=thunderbolt0
sudo firewall-cmd --reload
Our cluster scripts dynamically detect the network interface based on the provided IPs. There is no need to manually export environment variables!
toolbox enter vllmstart-vllm-cluster192.168.2.1 and the Worker IP to 192.168.2.2.thunderbolt0 as the backend network for Ray orchestration and GPU synchronization.I have added Thunderbolt support to the compare_eth_vs_rdma.sh script. Run it from inside the toolbox to see the latency and bandwidth of your Thunderbolt link compared to your other network interfaces.
You can use the -t flag to ONLY benchmark the Thunderbolt connection (or -e, -r, -i for the others):
/opt/compare_eth_vs_rdma.sh -t
The ones I've seen on aliexpress are from unknown Chinese vendors.
I was also a bit wary about Bosgame but TBH they've been great and the machine is rock solid, if a little noisier than and not as pretty as the FD. You can just buy from them directly and be fine, best computer deal out there by a mile.