Yotam Fromm
A typed judge in shadow: what I measured and why I promoted nothing
I tested Jev, a model that answers closed questions with a typed answer and a probability, in shadow at real decision points in my workflow. It was a good gate against needless skill loads, but it did no better than the planner's own labels on outcomes and picked skills worse than the model that reads the skill list, so nothing was promoted and the bake-off against Sonnet 5 still has no verdict.