kb-eval-nix: a failed .drv closure push fails the eval (SPEC §7.3)

kris commented

2026-09-27 22:12:47 +02:00

Owner

What

A failed .drv closure push fails the eval. With a cache-push target configured, publishing the evaluated derivation closure that fails every retry now fails the eval whatever KB_CACHE_PUSH_ON_FAILURE says. The error names the target and the push error ("publishing the evaluated derivation closure to failed; no other node could build this graph, so the eval fails rather than be cached" + the "failed after 3 attempts: …" cause). No target: unchanged. Logic in publish_derivations() (run.rs).
Honest log summary. The evaluated line is logged after publishing, with reconciled (locally-held outputs pushed so their derivations could be pruned — the old, misleading pushed=) and published (roots whose .drv closure reached the target).
Docs: OnFailure governs output pushes only; SPEC §7.3; forgejo-setup.md on_failure row.

Why

Live incident: the ares agent's attic token lacked push permission. Its evals logged WARN … derivation closure not published … AccessError, exited 0 and were cached (an eval is pure), so tasks placed on nuxbox failed downstream with "no substituter can build " — and since a cached eval replay never re-pushes derivations, not even a rerun could recover; only a new tree did. warn trades a missing output for a green task; it doesn't fit the drv closure, without which no other node can build anything in the graph.

Tests

run::tests::a_failed_derivation_push_fails_the_eval_even_under_warn (always-failing push command under warn → error naming the closure, retries and target; no target → Ok(0); working push → Ok(1)).
e2e (needs nix; skipped in the sandbox): an_unreachable_push_target_fails_the_evaluation now expects failure under warn; a_locally_held_path_is_published_before_it_is_pruned reads reconciled/published.

Operational note: the evaluated log field pushed= is now reconciled=.

🤖 Generated with Claude Code

### What - **A failed `.drv` closure push fails the eval.** With a cache-push target configured, publishing the evaluated derivation closure that fails every retry now fails the eval whatever `KB_CACHE_PUSH_ON_FAILURE` says. The error names the target and the push error ("publishing the evaluated derivation closure to <target> failed; no other node could build this graph, so the eval fails rather than be cached" + the "failed after 3 attempts: …" cause). No target: unchanged. Logic in `publish_derivations()` (`run.rs`). - **Honest log summary.** The `evaluated` line is logged after publishing, with `reconciled` (locally-held outputs pushed so their derivations could be pruned — the old, misleading `pushed=`) and `published` (roots whose `.drv` closure reached the target). - Docs: `OnFailure` governs output pushes only; SPEC §7.3; forgejo-setup.md `on_failure` row. ### Why Live incident: the ares agent's attic token lacked push permission. Its evals logged `WARN … derivation closure not published … AccessError`, exited 0 and were **cached** (an eval is pure), so tasks placed on nuxbox failed downstream with "no substituter can build <drv>" — and since a cached eval replay never re-pushes derivations, not even a rerun could recover; only a new tree did. `warn` trades a missing *output* for a green task; it doesn't fit the drv closure, without which no other node can build anything in the graph. ### Tests - `run::tests::a_failed_derivation_push_fails_the_eval_even_under_warn` (always-failing push command under `warn` → error naming the closure, retries and target; no target → `Ok(0)`; working push → `Ok(1)`). - e2e (needs nix; skipped in the sandbox): `an_unreachable_push_target_fails_the_evaluation` now expects failure under `warn`; `a_locally_held_path_is_published_before_it_is_pruned` reads `reconciled`/`published`. Operational note: the `evaluated` log field `pushed=` is now `reconciled=`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kris added 1 commit

2026-09-27 22:12:47 +02:00

kb-eval-nix: a failed .drv closure push fails the eval (SPEC §7.3)

krisbuild/kris/krisbuild/nix/test-deps cached

Details

krisbuild/kris/krisbuild/nix/hello cached

Details

krisbuild/kris/krisbuild/nix/workspace-deps cached

Details

krisbuild/kris/krisbuild/nix/world cached

Details

krisbuild/kris/krisbuild/nix/test cached

Details

krisbuild/kris/krisbuild/nix/clippy cached

Details

krisbuild/kris/krisbuild/nix/build succeeded

Details

krisbuild/kris/krisbuild/nix/kb-check succeeded

Details

krisbuild/kris/krisbuild krisbuild kris/krisbuild: all tasks succeeded

Details

krisbuild/queue merged

94f09318b3

With a push target configured, publishing the evaluated derivation closure
failing every retry now fails the eval regardless of
KB_CACHE_PUSH_ON_FAILURE, with an error naming the target and the push
error. `warn` is an output-push policy: without the derivations no other
node can build any task of the graph, and an eval is pure, so a success
was cached and replayed with its drvs still missing - not even a rerun
recovered, only a new tree. Without a target nothing changes.

The `evaluated` summary now logs after publishing and says what it
counts: `reconciled` (locally-held outputs pushed so their derivations
could be pruned, formerly the misleading `pushed`) and `published` (roots
whose .drv closure reached the target; 0 without one).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented

2026-09-27 22:12:51 +02:00

Author

Owner

@krisbuild r+

krisbuild commented

2026-09-27 22:18:26 +02:00

Collaborator

The pull request head's own build failed (graph 658).

nix/build: exit-code on nuxbox, exit 1 — log

Push a fix, or comment @krisbuild retry to rerun the failed tasks and requeue.

The pull request head's own build failed ([graph 658](http://192.168.0.2:1337/ui/graphs/658)). - `nix/build`: exit-code on nuxbox, exit 1 — [log](http://192.168.0.2:1337/ui/instances/18191) Push a fix, or comment `@krisbuild retry` to rerun the failed tasks and requeue.

krisbuild commented

2026-09-27 22:33:26 +02:00

Collaborator

Removed from the merge queue: the pull request was closed.