Kimi K3 Architecture Overview and Notes

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."

It just baffles me that this even works at all. Doesn't it just become a token soup? Is attention that precise that a second token can tell its the second token just because it learns to accumulate something in the embedding space without any sort of inductive bias?

Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack

i like your detailed breakdown. Thanks. <3

Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.

Interesting that they went NoPE everywhere — everyone else hedges with RoPE in the local layers. Feels like the linear-attention stuff (Kimi Delta) is quietly doing the positional work so they can get away with it. Curious to see if it holds up at frontier scale.

i like your detailed breakdown. Thanks. <3

Great breakdown. After using Kimi extensively, it's fascinating to see how architectural choices like KDA and NoPE translate into such strong real-world performance. Really impressive engineering.

Kimi Delta Attention (KDA), despite having "Attention" in the name, isn't really attention at all in any conventional sense. It's more like an RNN which can be efficiently parallelized during training. It's a very small modification to Gated DeltaNet, which can be described as an RNN whose hidden state acts like a small, editable attention memory.

Because it's RNN-like, it has an inherent idea that X comes before Y which comes before Z in the sequence XYZ. Transformers, by default, don't have that. They operate on sets, unordered collections of unique items. They have no idea where those items are in relation to each-other so you have to clue them in.

Because There are 3 KDA layers per attention layer, and 3 KDA layers before the first attention layer, every single token position is going to be able to learn information about where it is in the sequence before the first actual attention layer.

RoPE is actually a bit destructive, so being able to omit it like this is very convenient. Models like Gemma-4 have a similar structure with 5:1 Sliding Window Attention (SWA) layers for every global attention layer. These are cheap, shitty attention layers which handle local information and which go in-between the big powerful ones, KDA serves the same role in this model. In Gemma only the SWA layers have RoPE while the global attention layers omit it. SWA is actual attention, even if it only operates on a small sliding window, so it needs the positional embedding. KDA isn't, so it doesn't.

>Curious to see if it holds up at frontier scale.

I don't know how much more frontier scale you can get than this, but yes, there's no reason why that wouldn't work at larger scales. Honestly, more parameters just makes it easier for the KDA layers to communicate that positional information better.

Is frontier scale larger than this? Kimi K3 seems to benchmark in the same range as Opus and Fable. I would have expected they are all in the 2-4T range, with quality of the training and architecture differences as the major differentiators

It has some weird side effects though. for example KV-caches are implemented in fixed incremental token blocks (1024 from the providers I used) instead of simply caching up to the most recent input prompt input. It results in up to 1023 additional input (cache miss) tokens per inference.

This comment smells LLM generated.

Sabastian Raschka is one of the great LLM researchers/authors. I highly recommend his substack

This seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!

Never heard of him before but I was amazed how straight to the point yet very easy to grasp it was

I second that. I've always enjoyed his work!

"Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead."

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

As a sibling comment points out you don't strictly need positional embeddings for decoder-only causal transformers. You definitely need it for non-causal ones (e.g. the encoder of the original transformer paper!).

And yes accumulation is a good intuition for what's going on. You could imagine a part of the attention head that just kept writing to the same part of the residual stream causing that to keep accumulating (simply via attention summation) as more input tokens come in thereby functioning as a kind of index without the need for any positional encoding.

Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.

At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.

>Curious to see if it holds up at frontier scale.

This seems like a rare gem in a space swamped with lazily LLM generated summaries. Added to my RSS feed!

Never heard of him before but I was amazed how straight to the point yet very easy to grasp it was

Linear layers use decays (like IIR filters) that naturally provide relative positions. Full attention layers can then be free to develop concepts that attend to each other regardless of distance.

I second that. I've always enjoyed his work!

At least the SSM layers should have relative PE built in via recurrence, in a hybrid attention model like this.

Sounds a lot like running the Qwen3.5/3.6-series models at home: you need checkpoints for the recurrent state (GDN in the case of Qwen). You avoid the miss for the common case of 100% prefix match (e.g. during interleaved tool calls and thinking) by keeping an additional checkpoint for the actual last generated token. If you're a cloud provider serving many concurrent clients then you might prefer to skip that complexity and always take the 1k worst-case prefill hit.

SWA has a similar issue. Unless you keep the entire KV prefix lying around (which is not unreasonable: you retain flop + bandwidth benefits but lose capacity benefits), you need to start 1 window back from the rollback point, in order to refill the sliding window before going into normal prefill.

This comment smells LLM generated.

Huh, which model? I didn't pick up any of the tells. It seemed normal enough to me. (I vouched it, oops.)

The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.

> Seems to benchmark

yes, but in human usage the differences show up

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

And adding to that, there is also the recurrent state in the Kimi Delta Attention. I wouldn't call it position information but more sth like "position sensitivity"

Huh, which model? I didn't pick up any of the tells. It seemed normal enough to me. (I vouched it, oops.)

Telling which model wrote it is much harder than telling that some model wrote it.

The number of active parameters is vastly different. Deepseek CEO hinted that he estimates it as an order of magnitude difference in one of his recent interviews.

> Seems to benchmark

yes, but in human usage the differences show up

Would you happen to have a link to that interview? Sounds like an interesting read.

Telling which model wrote it is much harder than telling that some model wrote it.

Would you happen to have a link to that interview? Sounds like an interesting read.

It was posted to HN a few days back

1. https://news.ycombinator.com/item?id=49019012 (original chinese)

2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])

It was posted to HN a few days back

1. https://news.ycombinator.com/item?id=49019012 (original chinese)

2. https://news.ycombinator.com/item?id=49052912 (translated english [pdf])

The Kimi K3 architecture figure for yesterday’s big open-weight model release, along with some observations and thoughts.

Yes, it looks relatively complicated, but it’s essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)
The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it’s already very crowded, but that’s essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
Kimi K3’s overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).
The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.
Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.
Kimi K3 now also has native multimodal support, which is great!

There are several other interesting training tidbits in the technical report, but that’s it from the architecture front so far. A really great release overall.

Composite Kimi K3 architecture diagram with Kimi Delta Attention, gated multi-head latent attention, Attention Residuals, LatentMoE, and benchmark comparisons

Figure 1. Kimi K3 architecture and release-time benchmark comparisons. See K3 in the architecture gallery for more details.

Source: website version of my Substack note.

Hacker Times

Hacker Times

Kimi K3 Architecture Overview and Notes

Discussion

Discussion

Read Next