Since it seems to just produce broken and nonsensical sentences (at least based on the one example given) I'm not sure if it does work at this scale.
Anyway, as written this passage doesn't really make a whole lot of sense (the point is that it produces broken sentences?), and given that it was almost certainly written by an AI, it demonstrates that the architecture doesn't work especially well at any scale (I kid, I kid).
It (v3) mostly only says hello and bye, but I guess for 25k parameters you can't complain. (I think the rather exuberant copy is probably the product of Claude et al.)
(Or maybe not, if it doesn't perform better than random, I haven't actually tried it out yet. Some more examples would have been nice!)
I wonder how far you could push this while still staying period correct, e.g. by adding a REU (RAM Expansion Unit), or even a GeoRAM (basically a REU on steroids).
SuperCPU would also be an option, but for me it's always blurring the line of "what is a C64" a bit too much, and it likely just makes it faster anyway.
If it turns out that LLM-like models can produce genuinely useful outputs on something as constrained as a Commodore 64—or even more convincingly, if someone manages to train a capable model within the limits of hardware from that era—it would suggest we may have left a lot of progress on the table. Not just in terms of efficiency, but in how we framed the problem space for decades.
Brings back memories
On a plain vanilla C64, the Transformer cannot really show what it's capable of doing. An implementation using 2 bit per weight (vectorized) could be slightly better, perhaps.
Have not heard much about it since launch. Although, now that I look, it seems they are just shipping now.
https://www.commodore.net/product-page/commodore-64-ultimate...
I'm not sure what the venn diagram of knowledge to understand what that sentence is suggesting looks like, it's probably more crowded in the intersection than one might think.
It does also make me wonder what you could do with somewhat more powerful retro hardware. I'd love to see what a transformer running on a PSX or an N64 could do.
ELIZA was not written in assembler, but (different versions) in COMIT, FORTRAN and LISP.
(Came here to say an update to Eliza could really mess with the last person still talking to her.)
YOU> hey
C64> HELLO! RE SOUNDS ME. MEFUL!
Very, very cool project though!
YOU> HELP I'M DROWNING
C64> YOU' HERE!
YOU> OH NO I'M ON FIRE
C64> IGLAY!
YOU> IM BEING SWALLOWED BE A SNAKE
C64>
YOU> BIRDS ARE NIPPING ON ME
C64> YOU
Maybe there is deeper wisdom in there that we have yet to unearth
A real transformer running on a 1 MHz Commodore 64.
.-------.
| O O |
| V |
|..|---|..|
# SOUL PLAYER C64
25K PARAMETERS. 2 LAYERS. REAL TRANSFORMER.
LOADED OFF A FLOPPY DISK.
YOU> hey
C64> HELLO! RE SOUNDS ME. MEFUL!
A 2-layer decoder-only transformer - the same architecture behind ChatGPT, Claude, and Gemini - implemented in hand-written 6502/6510 assembly and running on an unmodified Commodore 64. ~25,000 int8 parameters. Real multi-head causal self-attention, real softmax, real RMSNorm. About 60 seconds per token. The whole thing fits on a floppy disk with room to spare.
2 layers, 4 attention heads × 8 dims, 32-dimensional embeddings, 64 FFN hidden units. ~25,000 parameters quantized to int8 with per-tensor shift scaling. The key breakthrough was fixing the softmax score normalization - shifting attention scores by 14 bits instead of 17 gives the 128-entry exp lookup table enough dynamic range to produce meaningful attention weights. Without this fix, the integer attention was essentially uniform across all positions, making the model blind regardless of architecture or training.
Grab disk/soulplayer.d64 and load it in any C64 emulator (VICE recommended):
LOAD"SOULPLAYER",8,1
RUN
Type a short message in lowercase, press RETURN, wait. The border flashes while it thinks. Each token gets a SID blip. A full response takes a few minutes. Type q to quit.
Tip: The model understands lowercase letters, spaces, and punctuation (
. , ! ? ' : ; -). Capital letters become unknown tokens.
This is the fun part. Write a corpus, train a model, build a floppy.
pip install numpy torch
Create a text file with one exchange per line in <SEP>input<SEP>response<SEP> format:
<SEP>hello<SEP>hey! nice to see you!<SEP>
<SEP>i'm sad<SEP>i hear you. i care about you.<SEP>
<SEP>tell me a joke<SEP>why did the bit flip? it was tired!<SEP>
Keep exchanges short - the model has a 20-token context window. See data/example_corpus.txt for a starter.
python train.py data/example_corpus.txt
This trains a BPE tokenizer (128 tokens), trains the QAT transformer, exports models/soul.bin and models/tokenizer.json. Takes a few minutes on GPU.
Every 500 epochs, you'll see both the float and int8 inference output side by side - what the model learned vs what the C64 will actually produce. The best checkpoint is saved based on int8 quality, not float loss. All checkpoints are saved to models/checkpoints/ for cherry-picking.
Options:
python train.py data/my_corpus.txt --epochs 30000 --output models/
python train.py # uses built-in emotional support corpus
Training resumes automatically if checkpoints exist from a previous run.
python build.py
This assembles all 6502/6510 routines, embeds your trained weights, and writes disk/soulplayer.prg and disk/soulplayer.d64.
x64 disk/soulplayer.d64 # VICE emulator
Or flash the .d64 to a real 1541 floppy for hardware.
python soulchat.py # uses models/soul.bin
python soulchat.py models/soul.bin # custom soul
Runs the same integer arithmetic as the C64, just faster.
python test.py # full suite (~90 tests, ~30 seconds)
python test.py --quick # skip 6502/6510 assembly tests
Tests verify the entire chain: float reference → integer reference → memory-faithful shadow → 6502/6510 assembly routines → build round-trip.
soulplayer-c64/
├── train.py - train a model + export weights
├── build.py - assemble the C64 binary
├── test.py - run all tests
├── soulchat.py - chat in your terminal
│
├── data/
│ └── example_corpus.txt
├── models/
│ ├── soul.bin - pre-trained weights (25KB, int8)
│ ├── tokenizer.json - BPE tokenizer (128 tokens)
│ └── checkpoints/ - all saved training checkpoints
├── disk/
│ ├── meful.d64 - original release, disk image
│ └── meful.prg - original release, raw PRG
│ ├── soulplayer.d64 - ready-to-run disk image
│ └── soulplayer.prg - raw PRG
└── src/ - the engine
├── numerics.py - ground truth: fixed-point math + forward pass
├── soul_io.py - .bin weight file format
├── shadow.py - memory-faithful Python shadow of the 6502/6510
├── assembler.py - mini 6502 assembler (labels, patches, far branches)
├── cpu6502.py - minimal 6502 interpreter for testing
├── asm_matvec.py - 6502 matrix-vector multiply
├── asm_rms_norm.py - 6502 RMSNorm (integer sqrt + divide)
├── asm_attn_head.py - 6502 attention head (LUT softmax)
├── asm_simple.py - 6502 embed, residual, relu, argmax
└── build.py - PRG + D64 assembler
| Vocab | 128 tokens (4 special + 34 chars/punct + 90 BPE merges) |
| Embedding | 32 dimensions |
| Layers | 2 |
| Attention | 4 heads × 8 dims per head |
| FFN | 64 hidden units |
| Context | 20 tokens |
| Parameters | ~25,000 (all int8) |
| Weight size | 25 KB |
| Decoding | Greedy (argmax) |
Each layer: RMSNorm → multi-head causal self-attention → residual → RMSNorm → ReLU MLP → residual. Final RMSNorm → output projection → argmax.
All activations are Q8.8 fixed-point (int16). Weights are int8 with per-tensor power-of-2 shifts. Biases are int16 pre-scaled to the matmul accumulator. Softmax uses a 128-entry exp lookup table with >>14 score normalization. The 6502 has no multiply instruction - everything is shift-and-add.
$0801-$20FF code + tokenizer tables (~6 KB)
$2100-$85A0 weights (25.3 KB)
$8600-$9D00 activation buffers (5.8 KB)
$C000-$C3FF token buffer, input, scratch
$D000- VIC-II, SID, CIA (I/O)
The model uses quantization-aware training (QAT). During training, weights pass through FakeQuantI8 - fake-quantized with continuous float scaling and straight-through gradient estimation. The deliberate mismatch between training's continuous scale and export's power-of-2 shift grid acts as implicit noise, forcing the model to learn weights with wider logit margins that survive the quantization gap. Biases are fake-quantized with simple fq(). Every matmul gets a × 0.5 post-shift simulating the 6502's >> 1.
Label smoothing (0.15) prevents the model from sharpening logit distributions beyond what int8 arithmetic can reliably distinguish. The training loop evaluates the actual integer forward pass (numerics.forward()) every 500 epochs and saves the best checkpoint by int8 argmax accuracy, not float loss.
The training output shows float and int8 inference side by side - what the model learned vs what the C64 will produce.
<UNK>. Stick to lowercase.GNU General Public License v3. See LICENSE.
The future came back for the past. And now it has a soul.
Even cooler would have been to have the 6502 directly generated from the LLM.
This is a giveaway for AI generation, from the docstring to the terrible opcode dispatch (Claude sucks at assembly or low-level optimization): https://github.com/gizmo64k/soulplayer-c64/blob/main/src/cpu...
A human would use a proper dispatch table and wouldn't make excuses for a sloppy implementation ("Python is fast enough").
Besides, the author has an art and design background, which doesn't seem to match the deep knowledge of Transformers or assembly required for such a project.