Although I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit.

At least this one gave credit to the upstream projects which it used as a reference.

The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by pointing claude at the repo and the original paper and having it produce something.

Almost none of these attempts contain information that really matters, like actual benchmark tests with differen KV quantization levels (not just perplexity or KLD).

Feels 100% vibe coded in a bad way.

Llama.cpp already has KV compression and one of the turbo quant PRs will get merged at some point.

If you don’t care about the fancy 3 bit, the q8 KV compression is good enough! Don’t bother with q4

./build/bin/llama-server -m model.gguf \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -c 65536

Etc

I couldn't get the downloadable binary to work, or the binary I compiled myself:

  ./SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

Error:

  [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit
  [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5
  [SwiftLM]  Memory strategy: FULL GPU (69.6GB model, 133.4GB available)
  [SwiftLM] Download: [===================>] 100% ⠋ (66395.4 MB / 66395.4 MB) | Speed: 0.0 MB/s      
  MLX error: Failed to load the default metallib. library not found library not found library not found library not found  at /Users/runner/work/SwiftLM/SwiftLM/LocalPackages/mlx-swift/Source/Cmlx/mlx-c/mlx/c/stream.cpp:115

Anyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there..

Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach.

Am i missing something fundamental?

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro:

TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead.

SSD Expert Streaming: To fit a 122B parameter model (e.g., Qwen3.5-122B MoE) without triggering macOS VM swapping or Watchdog kernel kills, the full ~60 GB weight file remains on NVMe. Only the top-k active expert pages are streamed to the GPU per forward pass at ~9 GB/s. As a result, inference runs with only 2,694 MB of active GPU VRAM on the M5 Pro 64GB, while the OS page cache automatically handles hot-expert reuse.

By combining these two approaches, we can comfortably run massive models in memory-constrained environments on Apple Silicon.

Also tested QWEN 4B on IPHONE 13 Pro.

Code and implementation details: https://github.com/SharpAI/SwiftLM

I like this idea on expert streaming. I've been poking around fairly thoroughly at the same idea - can we fix a set of experts? when can we fix them? How long is the top-k selection "good" for in terms of number of forward passes?

One thing I've turned up in smaller models and I'm sort of winding my way toward verifying in larger ones is that if you train the MoE model from scratch with this kind of knockout / subset of experts baked in, then you get significantly better loss outcomes. In small models, it's actually better than training an MOE without conditioning on a reduced set of experts per pass.

Anyway, pretty cool. There's some Pareto-optimal curve based on memory bandwidth, amount of GPU / unified RAM and inference compute times for streaming stuff in.

Can this work on M1, M2, M3, M4?

Does this use anything from the flash-moe project?

https://github.com/Alexintosh/flash-moe

Another project without running real benchmarks. It's very easy to generate tokens, it's much harder to solve tasks locally.

At least this one gave credit to the upstream projects which it used as a reference.

The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by pointing claude at the repo and the original paper and having it produce something.

Almost none of these attempts contain information that really matters, like actual benchmark tests with differen KV quantization levels (not just perplexity or KLD).

"vibe coded" is NOT the bad thing you think it is.

Going from paper to implementation from scratch in half an hour or so is great.

"vibe coded" is NOT the bad thing you think it is.

Going from paper to implementation from scratch in half an hour or so is great.

> "vibe coded" is NOT the bad thing you think it is.

It's not inherently bad in the same way that a first draft of a novel is not inherently bad.

But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.

Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it.

Software is valuable if it has been tested and exercised properly by other people. I don't care if you vide coded it provided you then put the real work in to verify that it actually works correctly - and then include the proof that you've done that when you start widely sharing it with the world.

Right now it's impossible to tell which of these projects implementing the paper are worth spending time with.

> Going from paper to implementation from scratch in half an hour or so is great.

This repo isn’t showing that at all. Scroll to the bottom of the README and you’ll see the other project it was based on. It’s a translation of other people’s work.

There have been dozens or perhaps hundreds of vibecoded TurboQuant examples posted around the usual forums in the past few days. This one doesn’t even include anything helpful like benchmarks or tests. It’s just some proof of concept code that doesn’t even work if you try to run it.

My problem with this specific type of vibe coded project is that it’s initially presented as something more novel or polished in order to get more upvotes, karma, likes, or pad a resume. Then you read it and discover they just pointed Claude at some other projects and told it to produce something similar, then posted it as their own work.

That’s a starting spot, but how about some testing and benchmarks?

Where’s the value added if the person just tells Claude to do it and then submits a PR?

The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

If there is nothing valuable it contributes, though? i.e. its not a novel paper then only value is the whatever you personally learn from it.

The authors of the project have CC as well, so doing this is just eating their time.

Yes, this took time to test :)

We implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro:

By combining these two approaches, we can comfortably run massive models in memory-constrained environments on Apple Silicon.

Also tested QWEN 4B on IPHONE 13 Pro.

Code and implementation details: https://github.com/SharpAI/SwiftLM

what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md

Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach.

Am i missing something fundamental?

Feels 100% vibe coded in a bad way.

Llama.cpp already has KV compression and one of the turbo quant PRs will get merged at some point.

If you don’t care about the fancy 3 bit, the q8 KV compression is good enough! Don’t bother with q4

./build/bin/llama-server -m model.gguf \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -c 65536

Etc

I’ve always believed local is the future. If you consider how your iPhone has a processor that is more powerful than something very large not too long ago.

llm intelligence seems to be proportional to the ram used. All techniques like this will be used by everyone.

I couldn't get the downloadable binary to work, or the binary I compiled myself:

  ./SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

Error:

  [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit
  [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5
  [SwiftLM]  Memory strategy: FULL GPU (69.6GB model, 133.4GB available)
  [SwiftLM] Download: [===================>] 100% ⠋ (66395.4 MB / 66395.4 MB) | Speed: 0.0 MB/s      
  MLX error: Failed to load the default metallib. library not found library not found library not found library not found  at /Users/runner/work/SwiftLM/SwiftLM/LocalPackages/mlx-swift/Source/Cmlx/mlx-c/mlx/c/stream.cpp:115

Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts):

  git clone --recursive https://github.com/SharpAI/SwiftLM.git
  cd SwiftLM

  swift build -c release

  # Trick to copy in that missing mlx.metallib file
  uv run --with mlx-metal python -c "
  import importlib.metadata, pathlib, shutil
  d = importlib.metadata.distribution('mlx-metal')
  metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib'
  shutil.copy(metallib, '.build/release/')
  print(f'Copied {metallib} -> .build/release/mlx.metallib')

  # Now start the server (downloads 69GB Qwen model)
  .build/release/SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

But the server crashed when I tried to run a prompt through it:

  freed pointer was not the last allocation

git clone https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue:

# Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/

Anyway, pretty cool. There's some Pareto-optimal curve based on memory bandwidth, amount of GPU / unified RAM and inference compute times for streaming stuff in.

Another project without running real benchmarks. It's very easy to generate tokens, it's much harder to solve tasks locally.

Here is a reference https://www.sharpai.org/benchmark/ For specific tasks, local model could achieve workable level.

Does this use anything from the flash-moe project?

https://github.com/Alexintosh/flash-moe

Yes, this is a reference project, the main different is we don't use os swap ( it introduces latency, will add https://github.com/danveloper/flash-moe to the original reference as well ).

Can this work on M1, M2, M3, M4?

> "vibe coded" is NOT the bad thing you think it is.

It's not inherently bad in the same way that a first draft of a novel is not inherently bad.

But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.

If there is nothing valuable it contributes, though? i.e. its not a novel paper then only value is the whatever you personally learn from it.

> Going from paper to implementation from scratch in half an hour or so is great.

This repo isn’t showing that at all. Scroll to the bottom of the README and you’ll see the other project it was based on. It’s a translation of other people’s work.

Yes, this took time to test :)

Sure, but the problem is when you take that half hour of work and share it with other people without making clear how much effort has gone into it.

Right now it's impossible to tell which of these projects implementing the paper are worth spending time with.

The authors of the project have CC as well, so doing this is just eating their time.

Yes, I've ran it on IOS, IPHONE 13 pro beside M5 pro, I'll test it on my M2 Mini and M3 Air.

At least in the novel example the author had the decency to write what they're asking you to read.

These are more like sending someone who didn't ask you a question a LMGTFY link they didn't ask for and expecting them to read all the results. Just a complete lack of awareness and respect for the maintainers

Thanks, pure Swift was the design idea and since I found nothing could be used for my project https://www.sharpai.org then I created Swift version. Python is too heavy to be delivered with application, user mentioned they want to use MLX, that's why I've been working on it for 1-2 weeks for bug fixing and testing , then suddenly TurboQuant proposed, I had a quick integration. My 64GB M5 Pro is already good for my local security task, now it's able to use M1/M2 Mini w/ 8GB memory.

what tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md

https://www.sharpai.org/benchmark/ The MLX part is what we've done with SwiftLM, the local result is still being verified more details are on-going.

I'll add more details. We just wired up the pipeline on both MAC and IOS.

yeah this I'd like to see added to teh readme.

That’s a starting spot, but how about some testing and benchmarks?

Where’s the value added if the person just tells Claude to do it and then submits a PR?

The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.

if it works it works

we live in a wholly unoptimized world because the available resources have been so high, while the benefits of optimizing have been so low. that has flipped now and there are tons of low hanging fruit to optimize.

I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts):

  git clone --recursive https://github.com/SharpAI/SwiftLM.git
  cd SwiftLM

  swift build -c release

  # Trick to copy in that missing mlx.metallib file
  uv run --with mlx-metal python -c "
  import importlib.metadata, pathlib, shutil
  d = importlib.metadata.distribution('mlx-metal')
  metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib'
  shutil.copy(metallib, '.build/release/')
  print(f'Copied {metallib} -> .build/release/mlx.metallib')

  # Now start the server (downloads 69GB Qwen model)
  .build/release/SwiftLM \
    --model mlx-community/Qwen3.5-122B-A10B-4bit \
    --stream-experts \
    --port 5413

But the server crashed when I tried to run a prompt through it:

  freed pointer was not the last allocation

the Python mlx-metal trick is actually what's crashing it. The mlx.metallib from pip is a different version of MLX than what your Swift binary was built against. It gets past the startup error but then corrupts the GPU memory allocator at inference time → freed pointer was not the last allocation.

Use the version-matched metallib that's already in the repo:

cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/ .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 This is the exact metallib that was compiled alongside the Swift code — no version mismatch. Future pre-built releases will bundle it automatically.

Yes, I've ran it on IOS, IPHONE 13 pro beside M5 pro, I'll test it on my M2 Mini and M3 Air.

⚡️ SwiftLM

A blazingly fast, native Swift inference server that serves MLX models with a strict OpenAI-compatible API.

No Python runtime, no Global Interpreter Lock (GIL), no unnecessary memory copies. Just bare-metal Apple Silicon performance compiled to a single binary.

SwiftLM Chat iOS demo

🚀 Features

🍎 100% Native Apple Silicon: Powered natively by Metal and Swift.
🔌 OpenAI-compatible: Drop-in replacement for OpenAI SDKs (/v1/chat/completions, streaming, etc).
🧠 Smart Model Routing: Loads HuggingFace format models directly, with native Safetensors parsing.
⚡️ TurboQuantization Integrated: Custom low-level MLX Metal primitives that apply extremely fast quantization for KV caching out-of-the-box.
💾 SSD Expert Streaming: Experimental zero-copy streaming that swaps Mixture of Experts (MoE) layers directly from the NVMe SSD to the GPU command buffer without trashing macOS Unified Memory (prevents Watchdog OS kernel panics on 122B+ models).
🎛️ Granular Memory Control: Integrated Layer Partitioning (--gpu-layers) and Wisdom Auto-Calibration for squeezing massive models into RAM.

⚡️ TurboQuantization: KV Cache Compression

SwiftLM implements a hybrid V2+V3 TurboQuant architecture for on-the-fly KV cache compression. At roughly ~3.6 bits per coordinate overall, the KV cache is compressed ~3.5× vs FP16 with near-zero accuracy loss.

By combining V2 Speed with V3 Quality:

Recent reproductions of the TurboQuant algorithm (e.g., turboquant-mlx) revealed two distinct paths:

V2 (Hardware-Accelerated): Fast, but uses linear affine quantization which degrades quality at 3-bit.
V3 (Paper-Correct): Excellent quality using non-linear Lloyd-Max codebooks, but painfully slow due to software dequantization.

We built the "Holy Grail" hybrid: We ported the V3 non-linear Lloyd-Max codebooks directly into the native C++ encoding path, and process the dequantization natively in fused Metal (bggml-metal) shaders. This achieves V3 quality at V2 speeds, completely detached from Python overhead.

The Algorithm:

K-Cache (3-bit PolarQuant + 1-bit QJL) = 4.25 bits/dim

Extract L2 norm and normalize: x̂ = x / ‖x‖
Apply Fast Walsh-Hadamard Transform (WHT) rotation to distribute outliers evenly.
Quantize each coordinate using 3-bit non-linear Lloyd-Max centroids.
Compute the residual error between the original vector and the quantized approximation.
Project the residual via a random Johnson-Lindenstrauss (QJL) matrix and store the 1-bit signs. (Why QJL? QJL acts as an additional regularizer that prevents centroid resolution loss from degrading the attention dot-product.)

V-Cache (3-bit PolarQuant) = 3.125 bits/dim Because the V-cache matrix is not used for inner-product attention scoring, the QJL error correction provides no benefit. We cleanly disable QJL for the V-cache, extracting an additional 25% memory savings without sacrificing quality.

Reference implementations: turboquant-mlx | turboquant_plus | Paper: TurboQuant, Google 2504.19874

💻 Tested Hardware & Benchmarks

To reliably run massive 122B parameter MoE models over SSD streaming, SwiftLM was designed and benchmarked natively on the following hardware:

Machine: MacBook Pro, Apple M5 Pro
Memory: 64 GB Unified Memory
Model: Qwen3.5-122B-A10B-4bit
SSD: Internal Apple NVMe (Zero-Copy Streaming)

⚠️ Quantization Disclaimer: While heavier quantization shrinks the required memory footprint, 4-bit quantization remains the strict production standard for MoE models. Our metrics indicated that aggressive 2-bit quantization heavily destabilizes JSON grammars—routinely producing broken keys like \name\ instead of "name"—which systematically breaks OpenAI-compatible tool calling.

📱 SwiftLM Chat — iOS App

A native iPhone & iPad companion app that downloads MLX models directly from HuggingFace and runs inference on-device via MLX Swift.

Features

Tab UI: Chat · Models · Settings
Live download progress with speed indicator and circular progress ring
Model catalog: Qwen3, Phi-3.5, Mistral, Llama — with on-device RAM fit indicators
HuggingFace search — find any mlx-community model by name
Context-aware empty states — downloading ring, loading spinner, idle prompt
iOS lifecycle hardened — model unload only fires on true background (not notification banners); 30-second grace period on app-switch

Build & Run (iOS)

cd SwiftLMChat
python3 generate_xcodeproj.py       # Generates SwiftLMChat.xcodeproj
open SwiftLMChat.xcodeproj

Then in Xcode:

Select the SwiftLMChat target → Signing & Capabilities
Set your Team (your Apple Developer account)
Select your iPhone as the run destination
⌘R to build and run

Note for contributors: The .xcodeproj is git-ignored (it contains your personal Team ID). Run generate_xcodeproj.py after cloning to regenerate it locally. Your Team ID is never committed.

🛠️ Quick Start (macOS Server)

Fastest: Download Pre-built Binary

Download the latest release tarball from the Releases page. The archive is self-contained — default.metallib is bundled alongside the binary.

tar -xzf SwiftLM-<version>-macos-arm64.tar.gz

# Run from the extracted directory — default.metallib must be co-located with the binary
./SwiftLM --model mlx-community/Qwen2.5-3B-Instruct-4bit --port 5413

⚠️ Metal GPU Error? If you see Failed to load the default metallib, it means default.metallib is missing from the directory you are running SwiftLM from. Make sure you run the binary from the extracted folder and do not move the binary without also moving default.metallib alongside it.

Build from Source

# Must clone recursively — default.metallib ships inside the mlx-swift submodule
git clone --recursive https://github.com/SharpAI/SwiftLM
cd SwiftLM
swift build -c release

default.metallib is a pre-built artifact inside the mlx-swift submodule, version-matched to the Swift binary. Copy it next to the binary before running:

cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \
   .build/release/

.build/release/SwiftLM \
  --model mlx-community/Qwen3.5-122B-A10B-4bit \
  --stream-experts \
  --port 5413

⚠️ Do NOT use Python's mlx-metal package as a source for mlx.metallib.
While uv run --with mlx-metal python -c "...shutil.copy(metallib, ...)" will get the server to start, the pip mlx-metal package is a different version of MLX than what this binary was compiled against. The version mismatch causes GPU kernel ABI corruption during inference, producing a freed pointer was not the last allocation crash. Always use the metallib from LocalPackages/mlx-swift/ — it is the only version-matched artifact for this build.

(Add --stream-experts when running oversized MoE models like Qwen3.5 122B to bypass macOS virtual memory swapping and stream expert layers directly from NVMe.)

📡 API Endpoints

Endpoint	Method	Description
`/health`	GET	Server health + loaded model capabilities
`/v1/models`	GET	List available models
`/v1/chat/completions`	POST	Chat completions (LLM and VLM support, multi-turn, system prompts)

💻 Usage Examples

Chat Completion (Streaming)

Drop-in compatible with standard OpenAI HTTP consumers:

curl http://localhost:5413/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen3.5-122B-A10B-4bit",
    "stream": true,
    "messages": [
      {"role": "system", "content": "You are Aegis-AI, a local home security agent. Output strictly in JSON format."},
      {"role": "user", "content": "Clip 1: Delivery person drops package at 14:02. Clip 2: Delivery person walks away down driveway at 14:03. Do these clips represent the same security event? Output a JSON object with a `duplicate` boolean and a `reason` string."}
    ]
  }'

⚙️ CLI Options

Option	Default	Description
`--model`	(required)	HuggingFace model ID or local path
`--port`	`5413`	Port to listen on
`--host`	`127.0.0.1`	Host to bind
`--max-tokens`	`2048`	Max tokens limit per generation
`--gpu-layers`	`model_default`	Restrict the amount of layers allocated to GPU hardware
`--stream-experts`	`false`	Enable experimental SSD streaming for MoE model expert matrices

📦 Requirements

macOS 14.0+
Apple Silicon (M1/M2/M3/M4/M5)
Xcode Command Line Tools
Metal Toolchain (xcodebuild -downloadComponent MetalToolchain)

📄 Dependencies & License

Built entirely on the hard work of the Apple MLX community.

mlx-swift — Apple MLX framework for Swift
Hummingbird — Event-driven Swift HTTP server
flash-moe — Reference for SSD Expert Streaming

🙏 TurboQuant Credits

The TurboQuant KV cache compression implemented in SwiftLM is directly based on the following open-source work and research:

TheTom/llama-cpp-turboquant — The primary reference for the C and Metal GPU implementation. The turbo-wht.h Fast Walsh-Hadamard kernel, WHT sign arrays (seed=42), Lloyd-Max centroid tables, and the ggml-turbo-quant.c quantize/dequantize logic were ported directly from this repository into our MLX C++ and Metal backend.
TheTom/turboquant_plus — Python reference implementation used to validate the algorithm math, codebook construction (Lloyd's algorithm for N(0, 1/d)), and KV cache integration design.
TurboQuant Paper — "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate", Zandieh et al., AISTATS/ICLR 2026. The two-stage PolarQuant + QJL algorithm described in Section 3 and Appendix A is the mathematical foundation of this implementation.
amirzandieh/QJL — Original Quantized Johnson-Lindenstrauss (QJL) 1-bit residual correction implementation by the paper authors.

MIT License

At least in the novel example the author had the decency to write what they're asking you to read.

llm intelligence seems to be proportional to the ram used. All techniques like this will be used by everyone.

git clone https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue:

# Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/

Yes, this is a reference project, the main different is we don't use os swap ( it introduces latency, will add https://github.com/danveloper/flash-moe to the original reference as well ).

Here is a reference https://www.sharpai.org/benchmark/ For specific tasks, local model could achieve workable level.

https://www.sharpai.org/benchmark/ The MLX part is what we've done with SwiftLM, the local result is still being verified more details are on-going.

yeah this I'd like to see added to teh readme.

I'll add more details. We just wired up the pipeline on both MAC and IOS.

I’ve always believed local is the future. If you consider how your iPhone has a processor that is more powerful than something very large not too long ago.

I've ran this on an IPHONE 13 pro (6GB) memory, QWEN 3 1.7B runs good. So local will get more intelligent for the task you want it done soon or already.

if it works it works

I agree that benchmarks would be great, but thats only relevant to this one topic, not the overall agentic coded pull request concept itself

It's relevant in that it's an example that people are doing the easy part - the coding - and skipping the hard part - the benchmarking and proving it works and provides value.

A PR without evidence it works and expectations for the benefits using the new feature would bring is kind of worthless.

It might work, but what's the point is sharing it if anyone can do the same in those 30 minutes with minimal effort?

> if it works it works

If it works in one case that doesn't mean it works consistently or well in the general case

I've made lots of things with Claude Code that just work... until I do things in a slightly different order and the whole thing explodes

Use the version-matched metallib that's already in the repo:

I've ran this on an IPHONE 13 pro (6GB) memory, QWEN 3 1.7B runs good. So local will get more intelligent for the task you want it done soon or already.

It's relevant in that it's an example that people are doing the easy part - the coding - and skipping the hard part - the benchmarking and proving it works and provides value.

A PR without evidence it works and expectations for the benefits using the new feature would bring is kind of worthless.

It might work, but what's the point is sharing it if anyone can do the same in those 30 minutes with minimal effort?

> if it works it works

If it works in one case that doesn't mean it works consistently or well in the general case

I've made lots of things with Claude Code that just work... until I do things in a slightly different order and the whole thing explodes

Hacker Times

Hacker Times

TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS

Discussion

Discussion

⚡️ SwiftLM

🚀 Features

⚡️ TurboQuantization: KV Cache Compression

By combining V2 Speed with V3 Quality:

The Algorithm:

💻 Tested Hardware & Benchmarks

📱 SwiftLM Chat — iOS App

Features

Build & Run (iOS)

🛠️ Quick Start (macOS Server)

Fastest: Download Pre-built Binary

Build from Source

📡 API Endpoints

💻 Usage Examples

Chat Completion (Streaming)

⚙️ CLI Options

📦 Requirements

📄 Dependencies & License

🙏 TurboQuant Credits