forge: follow-ups to the outbox and reconciliation (late webhook r+, truncated cursor, stuck candidate config, live pending at startup) #24

Manually merged

krisbuild merged 9 commits from lu/forge-followups into main 2026-09-27 22:52:49 +02:00

Conversation 2 Commits 9 Files changed 18 +1328 -231

kris commented 2026-09-27 20:26:50 +02:00

Owner

Copy link

Review follow-ups to #14 (forge outbox + reconciliation, SPEC §8.3). Builds on #14 — until it lands this diff includes it.

• Late webhook r+ is bound to the head like a reconciled one. An issue_comment webhook whose comment is more than 60 s old (redelivered by hand, stuck in Forgejo's backlog) no longer approves the current head outright: the head must have been recorded in forge_refs at least 5 s (CLOCK_SKEW_SECS) before the comment's created_at — the same rule reconciled comments use (strict: when clocks disagree, refuse); otherwise the bot asks for r+ again.
• A truncated comment listing advances: the cursor moves to the newest comment the listing reached, instead of staying put and staying truncated forever.
• A candidate whose .kb/ci.toml can't be read fails visibly: a pending krisbuild/queue status explains the retry, and after merge_queue_stall_secs (new, default 900) and ≥ 3 attempts the entry fails with a retry hint.
• The startup drop of stale pending statuses spares live subjects: kept while the graph is still running or the queue entry still approved/testing.
• Docs: SPEC §8.2/§8.3, forgejo-setup.md (new key, reconciliation, rollback, visibility), live-upgrades.md §4.4 (roll-forward outbox hazard).

Tests: reconcile::a_late_webhook_r_plus_is_bound_like_a_reconciled_one, reconcile::a_truncated_comment_listing_still_advances, merge_queue::a_persistently_unreadable_candidate_config_fails_visibly, outbox::stale_pending_is_dropped_only_when_its_subject_is_over.

🤖 Generated with Claude Code

Review follow-ups to #14 (forge outbox + reconciliation, SPEC §8.3). **Builds on #14** — until it lands this diff includes it. - **Late webhook `r+` is bound to the head like a reconciled one.** An `issue_comment` webhook whose comment is more than 60 s old (redelivered by hand, stuck in Forgejo's backlog) no longer approves the current head outright: the head must have been recorded in `forge_refs` at least 5 s (`CLOCK_SKEW_SECS`) before the comment's `created_at` — the same rule reconciled comments use (strict: when clocks disagree, refuse); otherwise the bot asks for `r+` again. - **A truncated comment listing advances**: the cursor moves to the newest comment the listing reached, instead of staying put and staying truncated forever. - **A candidate whose `.kb/ci.toml` can't be read fails visibly**: a `pending` `krisbuild/queue` status explains the retry, and after `merge_queue_stall_secs` (new, default 900) and ≥ 3 attempts the entry fails with a `retry` hint. - **The startup drop of stale `pending` statuses spares live subjects**: kept while the graph is still running or the queue entry still approved/testing. - Docs: SPEC §8.2/§8.3, forgejo-setup.md (new key, reconciliation, rollback, visibility), live-upgrades.md §4.4 (roll-forward outbox hazard). Tests: `reconcile::a_late_webhook_r_plus_is_bound_like_a_reconciled_one`, `reconcile::a_truncated_comment_listing_still_advances`, `merge_queue::a_persistently_unreadable_candidate_config_fails_visibly`, `outbox::stale_pending_is_dropped_only_when_its_subject_is_over`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kris added 7 commits 2026-09-27 20:26:50 +02:00

forge: durable outbox and webhook reconciliation (SPEC §8.3) ... 0609f7943d

Every forge write now goes through a `forge_outbox` table, written in the
same transaction as the state change that produces it: the `pending` at
graph creation (webhook and rerun), the aggregate and final per-task
statuses when a graph finishes, and each merge-queue transition's
`krisbuild/queue` status, comment and mark-merged. One worker drains it
with persisted exponential backoff (2 s doubling to 5 min, dropped after a
day or on a non-transient 4xx); a restart makes every row due at once.
Statuses coalesce on (repo, sha, context) so only the newest state is
sent; comments are deleted right after the forge accepts them (a crash in
between may post one twice) and writes to one PR keep their order. The
in-memory reporter and the driver's direct comment/merge calls are gone.
Per-task statuses of running graphs stay derived by the scheduler sweep,
and graph completion now enqueues the final ones itself, replacing the
post-completion re-sweep.

Reconciliation recovers webhooks sent while the control plane was down:
heads reported by push/PR webhooks are recorded in `forge_refs`; at
startup and every `forge_reconcile_secs` (default 300, 0 = off) each
repo's branches and open PRs are listed and a head that differs is fed
through the webhook's own handlers with `trigger.source = "reconcile"`,
so repo policy, ref filters, one-sha-one-run, kb-queue/* and the fork
policy apply unchanged. A recorded PR no longer open is fed through as
`closed`. The first run per repo only seeds. For merge-queue repos, PR
comments created since the last run are handed to the command handler,
which claims each comment id in `forge_comments` before applying it, so
a command seen by webhook and by listing is applied once.

Schema changes are additive (four new tables, no version bump).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge origin/main into lu/forge-outbox; address review of #14 ... b3777b3951

Conflict resolution: the merge queue's new failure/rerun/unqueued-base
paths enqueue their comments and statuses through the outbox; reruns
enqueue their `pending` in the rerun transaction; the in-memory
SeenComments guard is replaced by the durable `forge_comments` claim.

Review fixes:
- A reconciled `r+` approves the current PR head only if that head was
  recorded (forge_refs keeps when each sha was first seen) no later than
  the comment was written; otherwise it approves nothing and asks for
  `r+` again (SPEC §8.2 sha-bound approval).
- Recorded heads are read before listing; each reconcile feed or forget
  runs under a per-repo lock that webhook push/PR handling also holds,
  and only while the record still equals that snapshot. Seeding never
  overrides a webhook-recorded head.
- Comments are listed from max(cursor - overlap, seeded_at); a stale
  undelivered `pending` (>10 min) is dropped at startup; rollback and
  roll-forward documented in forgejo-setup.md.
- Listings page until an empty page; a truncated listing skips vanish and
  close detection and does not advance the comment cursor.
- An unreadable `.kb/ci.toml` is `OpenOutcome::Unavailable`: the head is
  not recorded as seen, and the merge queue retries instead of landing
  the candidate untested.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into worktree-agent-adf7e037f4266181e 74f599c028

tests: name the fake forge's listed-comment tuple (clippy::type_complexity) ... b11801ad87

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge origin/main into lu/forge-outbox (single integration-test binary) ... db26582a69

The forge outbox and reconcile suites move into tests/it/ as modules; the
fake forge's outbox and listing helpers merge with the git smart-HTTP
additions in it/common.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

forge: follow-ups to the outbox and reconciliation ... 7937c47052

- A webhook r+ whose comment is over a minute old (hand redelivery, forge
  backlog) is bound like a reconciled one; the head must be recorded at
  least CLOCK_SKEW_SECS before the comment, refusing when in doubt.
- A truncated comment listing advances the cursor to the newest comment
  it reached instead of re-listing an ever-growing window.
- A candidate whose .kb/ci.toml the forge keeps failing to serve posts a
  pending krisbuild/queue status saying so, and after
  merge_queue_stall_secs (default 900, at least 3 attempts) fails the
  entry with a retry hint.
- The startup drop of stale undelivered pendings keeps those whose subject
  is still live: a running graph's aggregate check, the queue status of
  an approved/testing entry.
- SPEC §8.2/§8.3, forgejo-setup.md, live-upgrades.md §4.4 updated.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into lu/forge-followups 9d2ed53c99

kris added 1 commit 2026-09-27 22:01:27 +02:00

merge queue: a displaced entry's stall starts a fresh clock ... 566e3b6609

A stall record now tracks its last attempt; attempts further apart than
two ticks plus slack start a new stall, so an entry that stopped being
first in line and returns much later is not failed on its first new
error. forgejo-setup.md: the late-r+ guard compares the forge's and the
control plane's clocks; run NTP on both.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris added 4 commits 2026-09-27 22:14:57 +02:00

Merge origin/main into lu/forge-outbox (node health, agent enrollment) ... c9dbb62f40

Conflicts: SPEC.md and db.rs (agent_enrollments beside the forge tables;
both kept), state.rs and scheduler/status.rs. The node-health status
behaviour — infra vs verdict descriptions, a requeued attempt re-posting
`pending` ("... (retrying on another node)") — now runs through the
outbox sweep: the per-task bookkeeping is keyed on (state, attempt), and
completion enqueues the final statuses in its own transaction as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Merge remote-tracking branch 'origin/lu/forge-outbox' into lu/forge-followups c6e692df6b

Merge remote-tracking branch 'origin/main' into lu/forge-followups b1852fabe5

outbox: note that the sweep re-derives a requeued attempt's retrying pending ... ff97a4a26b

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

kris commented 2026-09-27 22:37:48 +02:00

Author

Owner

Copy link

@krisbuild r+

@krisbuild r+

krisbuild referenced this pull request from a commit 2026-09-27 22:47:41 +02:00

Auto merge of #24 - forge: follow-ups to the outbox and reconciliation (late webhook r+, truncated cursor, stuck candidate config, live pending at startup), r=kris

krisbuild manually merged commit c5d5f2de35 into main 2026-09-27 22:52:49 +02:00

krisbuild commented 2026-09-27 22:52:49 +02:00

Collaborator

Copy link

Merged as c5d5f2de35.

Merged as c5d5f2de35a3.

kris referenced this pull request 2026-09-27 22:57:45 +02:00

Failure-summary PR comments with log tails; /ui/merge-queue page #27

kris referenced this pull request 2026-09-27 23:03:59 +02:00

Per-node attic push tokens issued by the control plane (SPEC §7.3.1) #13

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kris/krisbuild!24

No description provided.