Benchmark

The field is full of incomparable vendor numbers - the same system scores 62% and 92% depending on who ran it. Bezial does the opposite: lead with a hard, human-curated benchmark and publish everything needed to reproduce the score. The harness is built and lives in the repo.

What it reports

Benchmark	Role	Why
LongMemEval	Primary - lead with it	Human-curated, 500 questions, five abilities including temporal reasoning and knowledge-updates - the exact strengths of an editable temporal graph.
Full-context baseline	The bar	Whole history to GPT-4o scores ~60.6%. Beat it to prove retrieval earns its keep.
Oracle retrieval	The ceiling	Relevant sessions only, ~87.0%. The target to chase.
LOCOMO	Comparability only	Widely quoted, but with ~6.4% answer-key errors and lenient judging. Never lead with it.

Public baselines are reported as targets to beat and ceilings to chase, never as Bezial’s own numbers, and they render in their own labelled section - never merged into Bezial’s accuracy row.

How the harness works

One dataset-neutral pipeline:

1. A per-dataset loader normalizes the source JSON into a common shape.
2. Each haystack’s sessions are ingested through the file-is-truth write path - the harness never rewrites or pre-extracts the conversation, and it advances an injected clock to each session’s date so the bi-temporal model reflects the real conversation timeline.
3. Each question is answered through permission-filtered recall(..., synthesize: true).
4. An LLM judge scores the synthesized answer.

The harness only ever reads recall - it never touches the graph - so the load-bearing invariant holds.

Reproducibility is the trust move

Every run emits the harness, the published judge prompt, and the raw per-question transcripts (question, gold, synthesized answer, source citations, and the judge’s verbatim verdict). A reproducible score pairs perfectly with the inspectability Bezial already sells: if you can read the memory and re-run the evaluation, there is nothing to take on faith.

Cost is gated. The default run is a small fixture sample scored by an offline, deterministic heuristic judge (zero network, zero cost) that proves the harness is wired correctly and keeps CI fast and free. A ~$200 authoritative LongMemEval pass requires a real LLM judge over the whole dataset, behind explicit --full and --allow-paid flags. A run scored by a non-LLM judge, or a sampled run, is marked validation-only and cannot be presented as a benchmark result.

Caution

Extractor honesty. The bundled reference extractor is deterministic and only parses Bezial’s own note syntax - it cannot extract facts from natural-language prose. Real LongMemEval / LOCOMO sessions are prose, so a CLI-driven paid pass over them would surface no facts and score near zero; the CLI warns prominently before any paid run. An authoritative prose result requires the production temporal-graph extractor and synthesizer (Graphiti’s LLM extraction behind the extractor seam), injected through the library API - see Architecture.

The restore rate from the forgetting subsystem is deliberately not a published benchmark number - it is an internal tuning signal, and never appears in a benchmark report.

Sources

• LongMemEval - the primary benchmark and its baseline/oracle figures.
• LOCOMO - useful for comparability only; see published audits of its answer-key error rate.

Caution

Benchmark costs are order-of-magnitude and depend on model choice. Verify figures before quoting them formally.