Discussion about this post

User's avatar
Hellospark's avatar

Scoring the kept code a second time, unchanged, is a clever way to measure how noisy your own eval is. An experiment only counts when it beats the best checkpoint by more than that noise, so a gain smaller than the noise never gets mistaken for progress. Distilling each night's decision log into a 200-line learnings file keeps what was tried in one readable place.

Ready for more?