ADEMark v1 is out.

Same Mac mini, same tasks, same model, bare CLI as the baseline.

Did not expect most of them to use fewer tokens than the CLI on its own. Or one to sit at 16% CPU doing nothing.

First of many, updates come way more often from here.

ademark.dev

Nothing launches faster than the bare CLI. Expected, these are full apps.

Once they're open it flips. 9 of 10 get you a new thread faster than the CLI, Emdash in half the time.

Diri is the fastest to relaunch and to show the first word. T3 Code is the slowest to the first word.

Then I ran 5 agents at once and typed while they streamed.

Almost every app had less typing lag than the CLI. T3 Code was the smoothest at 48ms.

Memory is where they split. super.engineering stayed under 1 GB, Emdash went past 3.5 GB.

Left 5 idle threads open in each one and walked away.

super.engineering sat at 134 MB. Emdash at 2.3 GB. Traycer kept burning 15.8% CPU on nothing.

And the first request alone carries 10.5K tokens of setup in Paseo, 54.8K in Claude desktop.

This is the one that surprised me most.

Same question, same model. Paseo used 73K tokens. The bare CLI used 168K. Traycer used 357K.

The system prompt and tools get sent again on every request, so a heavy app costs you on every single turn.

Last test was the real one. A lead and 4 workers building a project against 5 hidden tests.

Synara passed all 5 in 5.2 min, faster than a single agent in the bare CLI. super.engineering passed all 5 too.

Cloudroom finished with main unchanged. bb got cut at 45 min.

Orchestration isn't free either. Every one of them spent more tokens than the solo run. Synara the least, bb 2.5x without finishing.

Method and raw data are on ademark.dev, and you can run it on your own machine.