Technical Dossier // Production Research

Work & Research

Autonomous agent architectures, long-term memory evaluation, AI benchmark analysis, and production engineering notes.

[tmux:0 research/longmemeval] pid: 4821

STATUS: COMPLETE

My Memory System Cheated To Beat LongMemEval Until I Fixed It

Last week I ran Weft, my homegrown memory layer, through LongMemEval and it scored 69.0% overall, 72.1% task-averaged. I was pleased. I shouldn't have been. The number was a lie, and the system that produced it was destroying the test data and calling it a win.

#coding/ai#memory

$ python eval_weft.py --dataset=longmemeval

[INFO] Initializing Oracle evaluator

[WARN] False 69.0% -> True 43.6%

[PERF] Memory rot at >32k tokens

[OK] Markdown fence bug resolved

[tmux:1 research/advisor-bench] pid: 6109

STATUS: BENCHMARK_DONE

I Benchmarked Anthropic's Advisor Strategy on Task Decomposition. The Expensive Model Was the Worst.

Anthropic's Advisor Strategy promises near-Opus intelligence at near-Sonnet cost. A server-side tool that pairs a cheap executor model with an expensive advisor. I benchmarked it in task decomposition.

#coding/ai#benchmarking

$ python run_advisor_bench.py

[INFO] Task decomposition: 42 graphs

[WARN] Opus advisor: 3.4x latency

[COST] +41% token spend over Sonnet

[OK] Sonnet standalone won benchmark