# Tag: experiments · Developer

> The items with the tag, newest first.

- [Stitch against Claude Design, and the correction after the score](https://yotamfromm.dev/developer/blog/design-ab-stitch.md): In a blind A/B test the Opus + Stitch lane beat Opus + Claude Design 3 to 2 on my vote and 12 to 6 with the AI judges, and only afterwards did the transcripts show that Stitch had returned almost none of the screens it was asked for. This is how I set up the trial, what it really measured, and how to run one that counts what a tool gives back. · [Plain text](https://yotamfromm.dev/developer/blog/design-ab-stitch.txt) · lang: en
- [Do my design books make Opus a better designer? A blind trial](https://yotamfromm.dev/developer/blog/design-ab-trove.md): I ran plain Opus against Opus reading my Trove design books, three blind runs each, on the brief for this site. The books removed the generic AI look and added polish, but every Trove site made its two modes look alike, cost about a dollar more per run, and lost the vote. · [Plain text](https://yotamfromm.dev/developer/blog/design-ab-trove.txt) · lang: en
- [Five routes to a site design, and plain Opus won](https://yotamfromm.dev/developer/blog/design-bakeoff-five-routes.md): I built my site's home page from one frozen brief with plain Opus and four design tools, then a sixth time with my Trove plugin, and had blind judges score the screenshots. Plain Opus scored highest in 7.1 minutes for $1.61, my top-ranked tool tied for last, and two judging flaws had to be fixed in code along the way. · [Plain text](https://yotamfromm.dev/developer/blog/design-bakeoff-five-routes.txt) · lang: en
- [Four tasks, three setups, hidden tests: plain Opus matched my /work process](https://yotamfromm.dev/developer/blog/hidden-tests-trial.md): On four real backlog items scored by tests the agents could not see, plain Opus passed every hidden test my tuned /work process passed, in 7 minutes and $2.46 against 53 minutes and $8.92, in notional API dollars. This is how I set up the trial so the agents could not game it, what it showed, and why I retired /work. · [Plain text](https://yotamfromm.dev/developer/blog/hidden-tests-trial.txt) · lang: en
- [Trove vs PageIndex: finding the exact page in a book](https://yotamfromm.dev/developer/blog/trove-vs-pageindex.md): My academic-writer plugin must cite the exact page that supports each claim, so I tested Trove against PageIndex on an English book, the Hebrew Bible and Hebrew sources. They tied on the exact page for English and the Hebrew sources, where PageIndex cost $0.086 a query against Trove's $0.096, and Trove's clear loss on the Bible came from a contents range that ended a chapter one page early. · [Plain text](https://yotamfromm.dev/developer/blog/trove-vs-pageindex.txt) · lang: en
- [Token-saving tools: I measured before I adopted](https://yotamfromm.dev/developer/blog/token-tools-trial.md): Before installing two community-recommended token-saving tools, I measured them over 80 headless runs alongside the output style I already use. All three cost more per task than the default at the same success rate, and a check that only tested whether files existed let an invented output style through. · [Plain text](https://yotamfromm.dev/developer/blog/token-tools-trial.txt) · lang: en
- [A typed judge in shadow: what I measured and why I promoted nothing](https://yotamfromm.dev/developer/blog/jev-typed-judge-trial.md): I tested Jev, a model that answers closed questions with a typed answer and a probability, in shadow at real decision points in my workflow. It was a good gate against needless skill loads, but it did no better than the planner's own labels on outcomes and picked skills worse than the model that reads the skill list, so nothing was promoted and the bake-off against Sonnet 5 still has no verdict. · [Plain text](https://yotamfromm.dev/developer/blog/jev-typed-judge-trial.txt) · lang: en
- [I read phalanx's source before I ran it](https://yotamfromm.dev/developer/blog/phalanx-trial.md): My catalogue held 64 AI tools and I had tested none, so I put one multi-agent orchestrator through a source read and two free test tiers. Its documented first command would have written a file into my own slash-command folder without asking; its stall monitor was worth copying. · [Plain text](https://yotamfromm.dev/developer/blog/phalanx-trial.txt) · lang: en

The full page: https://yotamfromm.dev/developer/tags/experiments/