Measure NixDrv builds from the realiser log; partial samples are lower bounds #25

Manually merged

krisbuild merged 2 commits from nixdrv-usage into main 2026-09-27 22:06:36 +02:00

Conversation 2 Commits 2 Files changed 8 +794 -154

kris commented 2026-09-27 20:32:40 +02:00

Owner

Copy link

NixDrv tasks read as idle (cpu_secs: 0.02, peak_cpus: 0, partial: true), for two reasons:

• Nix 2.34's nix build --json carries no timing or CPU.
• The build-cgroup watcher drops CPU whenever the node runs more than one instance, which on a busy agent is nearly always.

As a result, learned requests collapsed to the 0.25-cpu floor and --cores to 1, and crane's CARGO_BUILD_JOBS/RUST_TEST_THREADS followed. That is why graph 582 compiled on one core for 30+ minutes.

Agent

• Realiser flags: the realiser runs with --log-format internal-json --debug. The daemon's per-build builder for '<drv>' terminated …, user CPU …, system CPU … line, read from the build's cgroup, only reaches the client at debug verbosity.
• nix::log::NixLog: records each derivation build's span (activity type 105 start/stop) and its CPU. It renders the stream back into the plain name> line build log that --print-build-logs gave, keeping info-level messages, errors and warnings and dropping debug chatter. Daemon text arrives re-logged at level 0, so a level-0 message is shown only if it carries error info or reads as an error or warning.
• proc::run: folds the builds in once stderr is drained. CPU goes into cpu_secs, and each build is spread evenly over its span into the 2 s windows peak_cpus is taken from (UsageCurve::fold_daemon_builds). The watcher's CPU stays out of a NixDrv's windows so nothing is counted twice; the watcher still supplies memory.
• Cache level: built-vs-substituted now comes from the log's build activities. The pre-realise validity snapshot remains the fallback.
• Cleanup: nix.rs becomes nix/mod.rs + nix/log.rs, and the JSON-timing parse_build_result path is removed.

Control plane

• Partial samples: a partial sample is treated as a lower bound. With three exact samples only those are learned from. Otherwise all samples are, and the result may raise the payload default but never go below it. This keeps exec tasks' tree-measured CPU on busy nodes instead of discarding it.
• Stats: /api/stats/tasks reports partial_samples.

SPEC

§6.2.1 now matches: where NixDrv CPU comes from, how spans enter peak_cpus, and the partial-sample rule.

Tests

• New unit tests for log rendering and accounting, for spreading spans over windows, for proc::run with a fake realiser, and for the sizing rule.
• A real-Nix test builds a nonce'd busy derivation. It asserts that the build's CPU reaches the Usage event, that the cache level is cold, that the builder line is rendered as name> …, and that no raw @nix JSON leaks. It ran against the daemon on ares; in the Nix sandbox, where nix isn't available, it skips.
• cargo clippy --workspace --all-targets -D warnings is clean and cargo test --workspace passes.

Limitations and follow-ups

• A single derivation's peak_cpus is its average parallelism: a cargo derivation that compiles on every core and links on one reads as somewhere in between. Finer curves would need the watcher to attribute build cgroups by build activity.
• Memory is still attributed only when the task is alone on the node.
• Once this is deployed and grants look sane, the RUST_TEST_THREADS pin from #23 and its TODO entry can go.

🤖 Generated with Claude Code

NixDrv tasks read as idle (`cpu_secs: 0.02, peak_cpus: 0, partial: true`), for two reasons: - Nix 2.34's `nix build --json` carries no timing or CPU. - The build-cgroup watcher drops CPU whenever the node runs more than one instance, which on a busy agent is nearly always. As a result, learned requests collapsed to the 0.25-cpu floor and `--cores` to 1, and crane's `CARGO_BUILD_JOBS`/`RUST_TEST_THREADS` followed. That is why graph 582 compiled on one core for 30+ minutes. ### Agent - **Realiser flags:** the realiser runs with `--log-format internal-json --debug`. The daemon's per-build `builder for '<drv>' terminated …, user CPU …, system CPU …` line, read from the build's cgroup, only reaches the client at debug verbosity. - **`nix::log::NixLog`:** records each derivation build's span (activity type 105 start/stop) and its CPU. It renders the stream back into the plain `name> line` build log that `--print-build-logs` gave, keeping info-level messages, errors and warnings and dropping debug chatter. Daemon text arrives re-logged at level 0, so a level-0 message is shown only if it carries error info or reads as an error or warning. - **`proc::run`:** folds the builds in once stderr is drained. CPU goes into `cpu_secs`, and each build is spread evenly over its span into the 2 s windows `peak_cpus` is taken from (`UsageCurve::fold_daemon_builds`). The watcher's CPU stays out of a NixDrv's windows so nothing is counted twice; the watcher still supplies memory. - **Cache level:** built-vs-substituted now comes from the log's build activities. The pre-realise validity snapshot remains the fallback. - **Cleanup:** `nix.rs` becomes `nix/mod.rs` + `nix/log.rs`, and the JSON-timing `parse_build_result` path is removed. ### Control plane - **Partial samples:** a `partial` sample is treated as a lower bound. With three exact samples only those are learned from. Otherwise all samples are, and the result may raise the payload default but never go below it. This keeps exec tasks' tree-measured CPU on busy nodes instead of discarding it. - **Stats:** `/api/stats/tasks` reports `partial_samples`. ### SPEC §6.2.1 now matches: where NixDrv CPU comes from, how spans enter `peak_cpus`, and the partial-sample rule. ### Tests - New unit tests for log rendering and accounting, for spreading spans over windows, for `proc::run` with a fake realiser, and for the sizing rule. - A real-Nix test builds a nonce'd busy derivation. It asserts that the build's CPU reaches the Usage event, that the cache level is `cold`, that the builder line is rendered as `name> …`, and that no raw `@nix` JSON leaks. It ran against the daemon on ares; in the Nix sandbox, where `nix` isn't available, it skips. - `cargo clippy --workspace --all-targets -D warnings` is clean and `cargo test --workspace` passes. ### Limitations and follow-ups - A single derivation's `peak_cpus` is its *average* parallelism: a cargo derivation that compiles on every core and links on one reads as somewhere in between. Finer curves would need the watcher to attribute build cgroups by build activity. - Memory is still attributed only when the task is alone on the node. - Once this is deployed and grants look sane, the `RUST_TEST_THREADS` pin from #23 and its TODO entry can go. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kris added 1 commit 2026-09-27 20:32:40 +02:00

Measure NixDrv builds from the realiser log; partial samples are lower bounds ... ebc81824ff

NixDrv tasks read as idle: Nix 2.34's `nix build --json` carries no
timing, and the build-cgroup watcher drops CPU whenever the node runs
more than one instance. Learned requests collapsed to the 0.25-cpu
floor, `--cores` to 1, and crane's build jobs and test threads with it.

Agent: the realiser runs with `--log-format internal-json --debug`.
`nix::log::NixLog` takes each build's span from its activity and its
CPU from the daemon's `builder for … terminated …, user CPU …` line,
and renders the stream back into the plain `name> line` build log
(daemon debug chatter dropped; errors and warnings kept). `proc::run`
folds the builds into the usage once stderr is drained: CPU into the
total, each build spread over its span into the `peak_cpus` windows.
The watcher's CPU stays out of a NixDrv's windows; it still supplies
memory. Built-vs-substituted comes from the log's build activities.

Control plane: a `partial` sample is a lower bound. With three exact
samples only those are learned from; otherwise all are, floored at the
payload default. The task stats report `partial_samples`.

SPEC §6.2.1 updated to match.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris added 1 commit 2026-09-27 22:00:06 +02:00

Merge commit '214f223466' into nixdrv-usage ... 4889423259

# Conflicts:
#	crates/kb-agent/src/exec.rs
#	crates/kb-agent/src/proc.rs

kris commented 2026-09-27 22:06:26 +02:00

Author

Owner

Copy link

@krisbuild r+

@krisbuild r+

krisbuild referenced this pull request from a commit 2026-09-27 22:06:29 +02:00

Auto merge of #25 - Measure NixDrv builds from the realiser log; partial samples are lower bounds, r=kris

krisbuild manually merged commit 64e57fc1c9 into main 2026-09-27 22:06:36 +02:00

krisbuild commented 2026-09-27 22:06:36 +02:00

Collaborator

Copy link

Merged as 64e57fc1c9.

Merged as 64e57fc1c953.

kris referenced this pull request from a commit 2026-09-27 22:34:22 +02:00

Merge remote-tracking branch 'origin/lu/acked-delivery' into lu/run-supervisor

kris referenced this pull request 2026-09-27 22:35:38 +02:00

Runs outlive the agent: run supervisor, run directories, adoption, late effectful verdicts (SPEC §7.5) #28

krisbuild referenced this pull request from a commit 2026-09-27 23:28:07 +02:00

Auto merge of #28 - Runs outlive the agent: run supervisor, run directories, adoption, late effectful verdicts (SPEC §7.5), r=kris

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kris/krisbuild!25

No description provided.