RTFM GitHub

Research and method

The benchmark, its dataset and its scoring are in the repository. So are the runs that went badly — publishing only the flattering ones would make the number meaningless.

What is measured

One question, two ways of answering it, identical retrieval in both. The only difference is what comes back.

facts

The production path. A question goes in; a pack of facts with provenance comes out. Measured: how many tokens that pack costs whoever reads it.

full-docs

The same question and the same retrieved sources, but the agent reads those documents whole — what it would have to do without a fact layer.

No model is called in either arm, which is why the run is offline, free and repeatable. Loading the entire corpus into a context window is deliberately not implemented as a baseline: it is not a thing anyone should do, and measuring against it would flatter us.

Latest run — 2026-07-29

Mode Questions Marker recall Forbidden markers Honest refusal Version accuracy Provenance Tokens the agent reads Tokens RTFM spends
facts 22 73% 0% 2 of 2 1 of 1 90% 13 948 0
full-docs 22 86% 5% 2 of 2 1 of 1 90% 20 781 0

Run: 2026-07-29 · Source: docs/eval/results/2026-07-29-cc033.md

How to read that table, including the bad parts

Marker recall is not correctness

Scoring matches keywords from the dataset against the returned claims. It cannot tell a correct answer from one that merely contains the right nouns, and it under-credits a right answer phrased differently. Read it as “did not obviously miss”, not “is correct”.

The fact layer scores worse on recall — 73% against 86%

Reading whole documents finds more markers than our packs do, because a whole document contains every word in it. This is a real weakness of mechanical retrieval, it is our own number, and it is the thing most worth improving.

The two arms do not carry the same knowledge: 13 948 against 20 781

A pack is ranked against the question and cut to a budget, so it carries the documents behind the facts that survived that cut. The full-docs arm carries every document retrieval touched, whole. Roughly a third fewer tokens is therefore “less to read”, not “the same knowledge, cheaper” — and the pack says how much it held back, so the rest can be asked for.

Both honest-refusal probes pass

Two questions in the set have no answer in the corpus. Before the generator was removed, both were failed — the system invented a licence price and a person who does not exist. Mechanically they now come back empty.

Forbidden markers are not hallucinations

The column counts dataset strings that mean the answer went wrong. In full-docs they appear because a whole document happens to contain them, not because anything was invented — nothing generates text in either arm.

Run it yourself

No key, no network, no account. It takes seconds.

Reproduce the table above 
git clone https://github.com/uaskpro/rtfm && cd rtfm
pip install -e .
rtfm reindex

python scripts/benchmark.py --mode facts --mode full-docs

The dataset is docs/eval/questions.yaml; its header states what the scoring can and cannot see. Results land in docs/eval/results/.

A result that went the other way — 2026-07-30

Semantic search was added so that a question in one language could find a fact written in another, without a hand-written dictionary. It does that. It also broke the property this system exists for, so it is switched off.

Retrieval Accuracy Hallucinations Honest refusal Cross-language Per question
Lexical (shipped) 68% 9% 2 of 2 needs a dictionary 48 ms
Hybrid, k=1 36% 18% 0 of 2 20 of 20 5 010 ms
Hybrid, k=60 41% 18% 0 of 2 20 of 20 4 898 ms

Corpus: 2 403 entities · Source: docs/adr/0023-verifiability-is-not-relevance.md

Why it is off

Honest refusal went from 2 of 2 to 0 of 2

Both questions with no answer in the corpus came back answered. That is the one failure this project treats as disqualifying, and no gain elsewhere buys it back.

Nearest is not relevant

Nearest-neighbour search returns neighbours in order of distance for every question, including one nobody wrote about: there is always a closest match, and on a dense multilingual model it looks convincing. The admission rule read that ordering as significance. It is now measured against the corpus instead — a candidate has to be nearer than the base itself normally gets — and whether that is enough is a measurement nobody has taken yet on the production corpus.

The cross-language result is real, and kept

Twenty of twenty cross-language questions were answered with the synonym dictionary emptied. That is why the feature is retained, off by default, with the numbers it has to show before it comes back on.

A hundred times slower

48 ms against 4.9 s per question, most of it spent re-checking that quotes still resolved in their documents — a check that turned out to filter nothing, because a fact being verifiable says nothing about whether it answers the question.

What is still open

No third-party comparison yet

Runs against mem0 and Zep are wired into the script but were skipped: one needs an API key, the other a running Neo4j and a loaded corpus. Until they are stood up honestly, no number is reported for either.

One corpus, and it is our own

The golden set is built from this project’s documentation. That makes it easy to check by hand and easy to overfit to. A second, unrelated corpus is needed before any of these figures generalise.

Keyword scoring is blunt

A human verdict would be better and is not reproducible at this cost. The dataset keeps reference answers in prose alongside the markers so the scoring can be re-checked by hand.