mapwolf 53°30′ N · 8°00′ W

Research

Field notes

Independent experiments on how language models work. I build the controls before I believe a result, and I publish the ones that fail.

01

Attention & geometry

19 experiments

Novel LLM architectures

Question
Every modern language model compares words with a dot product. Is that the best way to route information, or only the most obvious one? Seven rounds tested spectral representations, Hopfield memory theory, p-adic geometry, Fourier Neural Operators and Grassmannian similarity.
Finding
Standard attention has a sharp failure point. When the number of items to retrieve approaches the embedding dimension (d/N = 1), accuracy drops from about 100% to about 20% in a single step. Comparing subspaces instead of single vectors (Grassmannian attention) reaches 97.3% where standard attention gets 19.1%, and moves the boundary out at least twice as far. A version with identical Q/K parameter counts confirms the gain comes from the geometry.
Scope
A side thread pinned down a likely mechanism for hallucination, where similar keys blend their values together. The ideas that failed are written up as carefully as the one that worked.

Python · PyTorch

View the repo on GitHub →

  • Non-Euclidean geometry for LLMs

    The wider set of experiments the attention study grew out of, testing alternative geometries for language-model attention.

  • HyperAttn-Nano

    Gives each attention head its own learnable curvature in hyperbolic space. Four phases lost to a Euclidean baseline. A fifth beat it by about 11 perplexity at training length, then degraded faster on longer contexts.

  • Frequency-domain weight compression

    Tests whether GPT-2's weight matrices compress in the frequency domain with a 2D FFT. They don't. There is no meaningful low-frequency concentration to exploit.

02

SSM / Mamba

post-mortem

The MambaE answer leak

Question
Plain Mamba can't follow a two-step chain of facts (A points to B, B points to C) in a single pass. My variant, MambaE, added a bidirectional junction and went from the chance floor to 100%, with fewer parameters than the baseline (1.48M against 1.86M).
Finding
The answer always sat at the same position near the end of each sequence, and a model that can read backwards might simply be looking at it. Three independent controls removed that leak in different ways (truncating the input, a leak-free junction, and an eval-time knockout). All three dropped MambaE straight back to chance.
Scope
The repo reproduces both the false win and the controls that killed it. A negative result you can trust is worth more than a positive one you can't.

Python · PyTorch · state-space models

View the repo on GitHub →

  • Recurrent-depth Mamba

    Reuses one Mamba block several times instead of stacking many. On an easy task it needs about 3.3× fewer parameters for roughly a quarter more training. On a harder one, halving the parameters costs reliability.

  • Generated weights for Mamba-2

    Tests whether a weight patch generated from the input helps Mamba-2. It doesn't. The patch turns out to be redundant with the model's own state.

  • SSM retrieval efficiency

    Measures how much cheaper it is to place a retrieved fact next to the answer than to make Mamba-2 recall it from context. About 70× cheaper to learn at K=8, and 470× to 690× at K=32.

03

Fundamentals & applied

fast weights

Fast-weight reasoning

Question
Transformers follow chains of facts (Anna's key is red, the red key opens the shed) because attention lets every word look at every other word. That cost grows with the square of the context. Fast weights, a small memory a layer writes to and reads back mid-pass, are far cheaper. Can they do the same job?
Finding
Not fully. The first two chapters, FENRIR and FREKI, show that plain stacked memory layers can't compose a chain on their own, and what it takes to make one form. The third changed only the training. Random filler between the facts and the question forced the model onto its memory, restoring a 4 of 5 pass rate at six hops where the plain recipe produced no healthy runs.
Scope
Some of the best numbers are single runs and are labelled that way. The chain skill did not carry over into ordinary language-model training, and the filler trick teaches the model to expect filler at test time.

Python · PyTorch

View the repo on GitHub →

applied ML

Weather nowcasting

Question
A thirty euro weather station forecasts with the Zambretti rule from the 1920s. It reads the barometer, checks the trend and picks one of 26 canned forecasts. Given the same readings, can a model do better?
Finding
Yes, by a clear margin. Gradient-boosted trees over about 40 features (current values, recent history, pressure and humidity trends, time and season) were trained on ten years of hourly data and tested on the two most recent years. Rain PR-AUC is 0.67 against Zambretti's 0.33, and the probabilities are well calibrated.
Scope
One station sees one point on the map, so the target is the gadget on the shelf rather than the national forecast. A leakage test checks that no feature ever sees past the moment it predicts from.

Python · scikit-learn

View the repo on GitHub →

  • Passive observer

    A gradient-free next-bit predictor built from scratch in pure Python, and a look at what actually makes an online predictor good.

04

Tools

tool

Alexandria

Why
I wanted one place for papers, docs and notes that an AI agent could search as easily as I can, without sending anything to a cloud service.
What it does
Alexandria ingests PDFs, HTML, text and markdown from folders or URLs, removes duplicates with three content-hash layers, and retrieves with reciprocal rank fusion over BM25 (SQLite FTS5) and vector search (sqlite-vec). Everything lives in one portable SQLite file. The same 20 tools are exposed over MCP (stdio and Streamable HTTP) and in a browser UI.
Also
New documents get suggested categories and tags from user-written anchors, with no training step.

Python · SQLite (FTS5 + sqlite-vec) · MCP · sentence-transformers

View the repo on GitHub →