OpenClaw, Hermes Agent and Prime Agent don't meaningfully learn from experience.

On Continual-ARC, our new benchmark at @letsunifyai, none of them were able to memorize anything of value and none were able to generalize to new tasks. Training a small 9B model's weights did both.

The paper, the code, and replays of every run:

Paper: unify.ai/papers/continu…
Code: github.com/unifyai/contin…
Site: continual-arc.unify.ai

Notes against weights, on one open model: Qwen3.5-9B, taught by GLM-5.3 Flash.

Notes saved nothing measurable on new puzzles. Trained on GLM's moves (DAgger), Qwen spent 57% less than untrained on puzzles it had met, 29% less on 100 new ones.

Section 4 · continual-arc.unify.ai/experiments/qd

How it works: 25 ARC puzzles keep returning over a stream of 200 visits, each time with a new grid, so caching answers is useless.

After the stream, two tests: new grids of the puzzles it has seen, and 100 puzzles it has never seen.

Section 2 · continual-arc.unify.ai/tasks

We ran all three on GPT-6 Luna. They solve 93 to 95% of puzzles at high effort, but never get cheaper as puzzles return. Remove their memory, and nothing changes.

A baseline that saves a program per puzzle drops straight away and stays down.

Appendix I · continual-arc.unify.ai/leaderboard

Told exactly what to keep, a checked program per puzzle, all three memorized, saving 17 to 62% of what perfect memory would.

But none got better at puzzles they'd never seen. The memory was there all along. The judgment wasn't.

Appendix F · continual-arc.unify.ai/leaderboard

Why? Prime Agent never stored anything for later (its Python skills also failed to import for us). Hermes Agent opened a skill on 524 of 525 returns, but mostly general tips. OpenClaw kept every conversation, but rarely went back to look.

Appendix E · continual-arc.unify.ai/runs/arc1/H-he…

Big caveat: one domain and a few models, none the strongest, and stronger ones may well keep more.

But I think real learning on the job will need the weights to change, with the harness becoming the thing that trains the model, not a growing pile of markdown files.

Section 6

Think your harness or training method learns? Run it on the public stream, then get it verified: we rerun your code on 50 new puzzles nobody outside unify has seen.

Feedback of all kinds welcome, especially on where the benchmark falls short 🙏

continual-arc.unify.ai/submit