Node health: requeue infra failures, quarantine failing nodes, infra vs verdict statuses #8

Manually merged

krisbuild merged 6 commits from node-health into main 2026-09-27 21:29:39 +02:00

Conversation 10 Commits 6 Files changed 20 +1423 -104

kris commented 2026-09-27 16:57:51 +02:00

Owner

Copy link

Node health, from the 2026-09-27 ares outage (disk full → every job failed with spawn-error for ~9h while ares kept getting work).

• Infra vs verdict classification (scheduler/health.rs::classify, new SPEC §3.3.1). Unknown classes are verdicts (never retried, never held against a node). kb-agent: spawn-error now strictly means the payload never started; a failure waiting on a started process is the new wait-error.
• Requeue infra failures on another node (#8): in the same transaction as the failure event; the failed attempt ends lost with its exit kept, a new attempt avoids that node (hard exclusion only while another connected node could take it). Pre-payload failures requeue for any effect class, post-payload only for pure/idempotent. Budget infra_retries (default 2). Failed-run usage is dropped so it doesn't skew learned requests.
• Unhealthy nodes take no new work (#7): free-disk floor min_free_disk_mb (default 4096), and a circuit breaker after quarantine_after (3) consecutive node-attributed infra failures with a quarantine_secs (600) cool-off, then probation (one task) → one success closes it. Derived from instance history each pass; never touches nodes.state, so manual drains are unaffected. Shown on /ui/nodes, /api/nodes and the queue's "why not scheduled".
• Status text (#11): failed: exit 1 on ares vs infra: spawn-error on ares, and … (retrying on another node) while a requeue is pending. Per-task statuses dedupe by (state, attempt).

No new tables/columns; one additive index. docs/forgejo-setup.md documents the four config keys; docs/TODO.md has the open items.

Review notes / follow-ups: git-checkout counts against the node, so a rev uncheckoutable everywhere can quarantine all nodes for one cool-off (documented, bounded). free_disk_mb measures the work dir only, not /nix/store.

🤖 Generated with Claude Code

Node health, from the 2026-09-27 ares outage (disk full → every job failed with `spawn-error` for ~9h while ares kept getting work). - **Infra vs verdict classification** (`scheduler/health.rs::classify`, new SPEC §3.3.1). Unknown classes are verdicts (never retried, never held against a node). kb-agent: `spawn-error` now strictly means the payload never started; a failure waiting on a started process is the new `wait-error`. - **Requeue infra failures on another node (#8)**: in the same transaction as the failure event; the failed attempt ends `lost` with its exit kept, a new attempt avoids that node (hard exclusion only while another connected node could take it). Pre-payload failures requeue for any effect class, post-payload only for pure/idempotent. Budget `infra_retries` (default 2). Failed-run usage is dropped so it doesn't skew learned requests. - **Unhealthy nodes take no new work (#7)**: free-disk floor `min_free_disk_mb` (default 4096), and a circuit breaker after `quarantine_after` (3) consecutive node-attributed infra failures with a `quarantine_secs` (600) cool-off, then probation (one task) → one success closes it. Derived from instance history each pass; never touches `nodes.state`, so manual drains are unaffected. Shown on `/ui/nodes`, `/api/nodes` and the queue's "why not scheduled". - **Status text (#11)**: `failed: exit 1 on ares` vs `infra: spawn-error on ares`, and `… (retrying on another node)` while a requeue is pending. Per-task statuses dedupe by (state, attempt). No new tables/columns; one additive index. docs/forgejo-setup.md documents the four config keys; docs/TODO.md has the open items. Review notes / follow-ups: `git-checkout` counts against the node, so a rev uncheckoutable everywhere can quarantine all nodes for one cool-off (documented, bounded). `free_disk_mb` measures the work dir only, not `/nix/store`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kris added 2 commits 2026-09-27 16:57:51 +02:00

kb-agent: spawn-error only before the payload starts; wait failures are wait-error ... df2e0925c0

proc::run now classifies its own failures: a failed spawn(2) is spawn-error,
a failure waiting on a started child is wait-error (after tearing the tree
down). The control plane relies on spawn-error meaning nothing of the payload
ran to requeue even effectful tasks after an infrastructure failure
(SPEC §3.3.1).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Node health: requeue infra failures elsewhere, disk floor, circuit breaker ... 46693bafd4

One classification (scheduler/health.rs, SPEC §3.3.1) decides whether a
failure class is the task's verdict or an infrastructure failure, and whether
the payload had started:

- An agent-reported infra failure becomes a new attempt in the same
  transaction (the old one ends lost, its exit kept), avoiding every node that
  failed it while another node could take the task, within infra_retries
  (default 2). Pre-payload classes requeue even effectful tasks. The failed
  run's zero usage is not learned from.
- A node reporting less than min_free_disk_mb (default 4096) free, or whose
  last quarantine_after (default 3) outcomes were node-attributed infra
  failures, takes no new work; the quarantine lifts after quarantine_secs
  (default 600) into probation (one task at a time) until a success. Derived
  from instances each pass (new index instances_by_node), never written to
  nodes.state, so a manual drain is untouched. Shown on /ui/nodes,
  GET /api/nodes and the queue's why-not.
- Per-task statuses read 'failed: exit 1 on ares' vs 'infra: spawn-error on
  ares', and a requeued task stays pending as '... (retrying on another
  node)'; a new attempt re-posts its pending status.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented 2026-09-27 16:58:04 +02:00

Author

Owner

Copy link

@krisbuild r+

@krisbuild r+

krisbuild referenced this pull request from a commit 2026-09-27 17:36:08 +02:00

Auto merge of #8 - Node health: requeue infra failures, quarantine failing nodes, infra vs verdict statuses, r=kris

krisbuild commented 2026-09-27 18:02:52 +02:00

Collaborator

Copy link

Merge candidate failed: candidate graph failed

Merge candidate failed: candidate graph failed

kris commented 2026-09-27 18:04:02 +02:00

Author

Owner

Copy link

@krisbuild retry

The candidate failure was infra: nix/build was placed on nuxbox, but its .drv (evaluated on ares) was never pushed to attic. The closure has been pushed by hand; see the PR thread for the cache-push follow-up.

@krisbuild retry The candidate failure was infra: `nix/build` was placed on nuxbox, but its `.drv` (evaluated on ares) was never pushed to attic. The closure has been pushed by hand; see the PR thread for the cache-push follow-up.

krisbuild commented 2026-09-27 18:38:36 +02:00

Collaborator

Copy link

Removed from the merge queue: the merge conflicts in crates/kb-control-plane/src/ws.rs, crates/kb-core/src/protocol.rs.

Removed from the merge queue: the merge conflicts in crates/kb-control-plane/src/ws.rs, crates/kb-core/src/protocol.rs.

kris added 2 commits 2026-09-27 18:50:34 +02:00

Merge origin/main into node-health ... b007c04bf1

ws.rs: main's rejected-Outputs path (set_exit + failed) alongside the infra
settle_failure path. protocol.rs: both sides' ExitInfo class docs. main's
expansion-invalid / outputs-invalid are verdicts in the SPEC §3.3.1 table.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kb-agent: an unfetchable .drv fails drv-fetch before realise ... 3e63954dda

A NixDrv whose .drv is neither on the node nor in any substituter used to run
nix build anyway and fail as an exit-code verdict. It is now the infra class
drv-fetch, raised before the payload runs and not held against the node, so
the control plane requeues it where the .drv exists (SPEC §3.3.1; classified
in scheduler/health.rs alongside the merge of main).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented 2026-09-27 18:51:18 +02:00

Author

Owner

Copy link

Merged main in (conflicts in ws.rs / protocol.rs resolved; main's new expansion-invalid/outputs-invalid classes classified as verdicts) and added the drv-fetch infra class for an unfetchable .drv. Reviewed.

@krisbuild r+

Merged main in (conflicts in ws.rs / protocol.rs resolved; main's new `expansion-invalid`/`outputs-invalid` classes classified as verdicts) and added the `drv-fetch` infra class for an unfetchable `.drv`. Reviewed. @krisbuild r+

krisbuild referenced this pull request from a commit 2026-09-27 19:38:57 +02:00

Auto merge of #8 - Node health: requeue infra failures, quarantine failing nodes, infra vs verdict statuses, r=kris

kris commented 2026-09-27 19:44:29 +02:00

Author

Owner

Copy link

@krisbuild r-

Yielding the queue slot to #18 (faster test suite); will merge main into this branch and re-approve once it lands.

@krisbuild r- Yielding the queue slot to #18 (faster test suite); will merge main into this branch and re-approve once it lands.

kris added 2 commits 2026-09-27 19:54:50 +02:00

Merge remote-tracking branch 'origin/main' into worktree-agent-a253c698783613350 9f1a0a1d69

tests: node_health joins the single tests/it binary ... d4fb56e386

tests/node_health.rs becomes the tests/it::node_health module (SPEC-free
layout change from #18); its fail_class/heartbeat_disk helpers already sit in
tests/it/common and are used, so nothing needs a dead_code allow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented 2026-09-27 19:55:20 +02:00

Author

Owner

Copy link

Merged main (#18) in; node_health integration tests moved into the single tests/it binary. Reviewed.

@krisbuild r+

Merged main (#18) in; `node_health` integration tests moved into the single `tests/it` binary. Reviewed. @krisbuild r+

krisbuild referenced this pull request from a commit 2026-09-27 20:20:28 +02:00

Auto merge of #8 - Node health: requeue infra failures, quarantine failing nodes, infra vs verdict statuses, r=kris

krisbuild commented 2026-09-27 20:24:57 +02:00

Collaborator

Copy link

The merge candidate's build failed (graph 625).

• nix/kb-check: exit-code on nuxbox, exit 1 — log

Comment @krisbuild retry to requeue with a fresh candidate on the current base.

The merge candidate's build failed ([graph 625](http://192.168.0.2:1337/ui/graphs/625)). - `nix/kb-check`: exit-code on nuxbox, exit 1 — [log](http://192.168.0.2:1337/ui/instances/17740) Comment `@krisbuild retry` to requeue with a fresh candidate on the current base.

kris commented 2026-09-27 20:25:48 +02:00

Author

Owner

Copy link

@krisbuild retry

Infra again: nix/kb-check landed on nuxbox, its .drv (evaluated on ares) was not in attic. Pushed the closure by hand; the lasting fix is agent cache push (pixienix krisbuild-agent-push).

@krisbuild retry Infra again: `nix/kb-check` landed on nuxbox, its `.drv` (evaluated on ares) was not in attic. Pushed the closure by hand; the lasting fix is agent cache push (pixienix `krisbuild-agent-push`).

krisbuild referenced this pull request from a commit 2026-09-27 20:54:26 +02:00

Auto merge of #8 - Node health: requeue infra failures, quarantine failing nodes, infra vs verdict statuses, r=kris

krisbuild manually merged commit fdbb9c5aa9 into main 2026-09-27 21:29:39 +02:00

krisbuild commented 2026-09-27 21:29:39 +02:00

Collaborator

Copy link

Merged as fdbb9c5aa9.

Merged as fdbb9c5aa986.

kris referenced this pull request from a commit 2026-09-27 22:02:08 +02:00

Merge origin/main (#10 enrollment, #8 node health, #22)

kris referenced this pull request from a commit 2026-09-27 22:03:36 +02:00

Merge origin/main into lu/protocol-handshake

kris referenced this pull request from a commit 2026-09-27 22:34:22 +02:00

Merge remote-tracking branch 'origin/lu/acked-delivery' into lu/run-supervisor

kris referenced this pull request 2026-09-27 22:35:38 +02:00

Runs outlive the agent: run supervisor, run directories, adoption, late effectful verdicts (SPEC §7.5) #28

kris referenced this pull request 2026-09-27 22:38:44 +02:00

Per-node attic push tokens issued by the control plane (SPEC §7.3.1) #13

krisbuild referenced this pull request from a commit 2026-09-27 23:28:07 +02:00

Auto merge of #28 - Runs outlive the agent: run supervisor, run directories, adoption, late effectful verdicts (SPEC §7.5), r=kris

kris referenced this pull request 2026-10-01 00:21:06 +02:00

Node health: don't quarantine the last node able to take work when others tripped with it #36

kris referenced this pull request 2026-10-01 00:30:59 +02:00

Node health: don't quarantine the last node able to take work when others tripped with it #36

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kris/krisbuild!8

No description provided.