save the work, then mark it done

The tone translator has an agent that mines real usage for failing translations and proposes new eval cases — the thing that keeps the test set growing. One run did all of its work (“Reviewed 238, proposed 38 new cases”), then crashed on the very last step: a dropped socket on a call that counts how many rows are left, there purely for the summary line. The 38 proposals were gone. Worse, the agent had already advanced its “seen” watermark, so a re-run would skip those 238 rows forever. A transient blip on a cosmetic call had thrown away a batch of paid judge work.

Two faults compounded. First, a best-effort cosmetic call was allowed to be fatal — a network error on a row count crashed an otherwise-successful run, when it should return “unknown” and warn. Second, and the real one: the code marked the inputs consumed before it saved the output. The sequence was advance-the-watermark, then write the proposals — and the crash landed in the gap. Mark-as-seen ran before save-the-work.

There was a sibling to it. The watermark is a single monotonic timestamp — “everything up to here is done.” That shape can only ever encode a contiguous prefix; it has no way to say “all of these except the one in the middle that errored.” So when a row’s judge call failed and the loop moved on, advancing the mark to the newest fetched row silently dropped the errored one from ever being mined again. The fix is to freeze the mark at the first failure, even when later rows succeeded — a monotonic cursor must never step past a unit of work that didn’t actually complete.

Both are the same rule stated twice: record that you’ve consumed an input only after the work behind it is durably real, and never let a cursor claim more than it can prove. The trade you accept is recoverable duplicates over silent loss — a frozen watermark re-processes a few clean rows next run, which is strictly better than a gap you can’t see. It’s at-least-once processing and checkpoint ordering, learned the expensive way, in about forty lines of a side project’s agent.