Re-evaluate when a cached eval's drvs are missing from the cache (SPEC §6.1) #17

Manually merged

krisbuild merged 4 commits from eval-drv-verify into main 2026-09-27 19:57:32 +02:00

Conversation 2 Commits 4 Files changed 7 +378 -8

kris commented 2026-09-27 18:43:09 +02:00

Owner

Copy link

When a cached eval/expand result is about to be reused, the control plane now checks that the .drv files it emitted are actually in the cache (SPEC §6.1, §7.3). Until now a cached eval whose drvs never reached the cache (for example because attic watch-store was backlogged) was replayed forever, and every dependent failed with "derivation not present locally and not fetchable". Retrying couldn't recover (seen on graphs 573, 575 and 580).

• Stale-eval check: missing_drv() probes the narinfo of every emitted NixDrv .drv on the configured substituters, up to 16 at a time. If every substituter answers 404, the hit is stale and the eval runs again.
• Fail open: on substituter errors, after 15 s, or with no substituters configured, the cached result is used as before.
• Probe cache: positive answers are kept in memory for 1 h, so a warm cache costs one probe per drv per hour.
• Reruns: a rerun no longer carries over unfinished tasks emitted by a pure expander. The expander emits them again, so they are ordered after the (re-)evaluation.
• Tests: new tests/stale_eval.rs covers the 404 re-eval, 200 replay with one probe in total, 503 fail-open, and a failed_only rerun that recovers.

🤖 Generated with Claude Code

When a cached eval/expand result is about to be reused, the control plane now checks that the `.drv` files it emitted are actually in the cache (SPEC §6.1, §7.3). Until now a cached eval whose drvs never reached the cache (for example because `attic watch-store` was backlogged) was replayed forever, and every dependent failed with "derivation not present locally and not fetchable". Retrying couldn't recover (seen on graphs 573, 575 and 580). - **Stale-eval check:** `missing_drv()` probes the narinfo of every emitted `NixDrv` `.drv` on the configured substituters, up to 16 at a time. If every substituter answers 404, the hit is stale and the eval runs again. - **Fail open:** on substituter errors, after 15 s, or with no substituters configured, the cached result is used as before. - **Probe cache:** positive answers are kept in memory for 1 h, so a warm cache costs one probe per drv per hour. - **Reruns:** a rerun no longer carries over unfinished tasks emitted by a pure expander. The expander emits them again, so they are ordered after the (re-)evaluation. - **Tests:** new `tests/stale_eval.rs` covers the 404 re-eval, 200 replay with one probe in total, 503 fail-open, and a `failed_only` rerun that recovers. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kris added 1 commit 2026-09-27 18:43:09 +02:00

Stale cached eval: re-evaluate when its .drv files are not in the cache ... 871c098a18

A cached eval/expand result is replayed only after a narinfo HEAD for each
emitted NixDrv's .drv. An authoritative 404 from every substituter makes the
hit stale (SPEC §6.1) and the eval runs again; an unreachable or slow cache
keeps the hit (fail open, 15s bound), and positive answers are remembered for
an hour so a warm cache costs one probe per derivation, not per graph.

Reruns no longer carry tasks a pure expander emitted that did not complete in
the source: the expander (replayed or re-run) emits them again, so a
re-evaluation precedes the tasks it repairs instead of racing them. Without
this a failed_only rerun of a graph whose drvs never reached attic could
never recover.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris added 1 commit 2026-09-27 19:13:04 +02:00

substitution: collapse the stale-eval check into a let-chain (clippy) ... 17a94bdcb4

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris added 2 commits 2026-09-27 19:51:27 +02:00

Merge remote-tracking branch 'origin/main' into eval-drv-verify 6bd17341df

tests: fold stale_eval into the control plane's single test binary ... 4923aacb1e

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented 2026-09-27 19:54:14 +02:00

Author

Owner

Copy link

Merged main (#18) and moved stale_eval into tests/it/. Previous head was fully green (graph 588).

@krisbuild r+ p=5

Merged main (#18) and moved `stale_eval` into `tests/it/`. Previous head was fully green (graph 588). @krisbuild r+ p=5

krisbuild referenced this pull request from a commit 2026-09-27 19:57:27 +02:00

Auto merge of #17 - Re-evaluate when a cached eval's drvs are missing from the cache (SPEC §6.1), r=kris

krisbuild manually merged commit 5fcfc8d3fb into main 2026-09-27 19:57:32 +02:00

krisbuild commented 2026-09-27 19:57:32 +02:00

Collaborator

Copy link

Merged as 5fcfc8d3fb.

Merged as 5fcfc8d3fb15.

kris referenced this pull request 2026-09-27 22:38:44 +02:00

Per-node attic push tokens issued by the control plane (SPEC §7.3.1) #13

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kris/krisbuild!17

No description provided.