gabogg/hikcentral
Watch
1
0
Fork
You've already forked hikcentral
 0
Code Issues 113 Pull requests 8 Projects Releases Packages Wiki Activity Actions

WIP: docs(tests): record confirmed local validation policy #265

Draft
gabogg wants to merge 6 commits from docs/test-policy-217 into master
pull from: docs/test-policy-217
merge into: gabogg:master
Conversation 4 Commits 6 Files changed 3 +184
gabogg commented 2026-10-03 12:06:23 +00:00
Owner
Copy link

Summary

Agents repeatedly run the full test suite locally even though remote CI independently validates every PR revision. Record the maintainer-confirmed policy from the #217 decision record and D16 confirmation: scoped local tests, reusable completed evidence for identical immutable inputs, and a commit hook that runs relevant tests only when matching evidence is absent.

This PR complements research PR #218, which contains the measurements and the #216/#210 isolation fixes. It records the policy agreed after that research; PR #218 retains its research scope, and issue #217 remains open. Implementation remains follow-up work.

Architectural impact

  • Add the accepted ADR 0009 and the consolidated policy document; link both from the documentation index.
  • Specify explicit test groups with full-suite fallback for shared or unknown changes, shared local evidence storage, immutable execution, approved-skip handling, and independent concurrent requests.
  • Retain remote full-suite CI and the head-commit merge gate. A fix after the final review requires a new review examining why validation missed the failure; testing-system defects need a separate issue.
  • Require research handoff to accurate, linked follow-up issues before research closure; the detailed checklist remains proposed. Follow-ups join a milestone only if delivery depends on them.

Executable hooks, AGENTS.md, workflow rules and the test runner are unchanged. CPU scheduling and conflicting-run diagnosis remain separate studies.

Related work and implementation handoff

  • #217: parent research and design discussion.
  • #218: measured findings and hermetic-suite prerequisites; its description links this policy and related open-test examples.
  • #241: scoped runner, evidence storage/reuse, immutable snapshots and validated selection boundaries; brief reconciled with this policy.
  • #242: evidence-aware hook and consistent agent/workflow guidance; brief reconciled with this policy.
  • #238 and #240: parallel execution and fixture optimization; reconcile withdrawn projections with the actual measured findings.
  • #235: separate remote-execution evaluation.

The related examples on #218 include #221 (order dependence), #250 (an unforced collision), #260/#184 (test brittleness), #213 (clock and transition fidelity), #76 (timezone assumptions), #261 (migration failure-path coverage), #225 (missing integration coverage), and #60 (unexecuted platform behavior). They illustrate constraints rather than additional scope closed here.

Verification

  • Merged origin/master (9a608c1) cleanly through merge commit 81d5c2a.
  • python scripts/check_docs.py passed: 45 Markdown files and 90 HTTP operations.
  • ruff check . and ruff format --check . passed (156 files formatted).
  • Installed commit hooks passed, including full pytest, on fixes commit f919ba6.
  • Docs-only draft remains draft. #218 is still unmerged; its branch research link has an explicit pending relative-path switch.

Checklist

  • Maintainer confirmed the consolidated design.
  • Record the policy and accepted ADR.
  • Link the documentation and research handoff.
  • Replace the #218 branch research link with a relative path once #218 merges.
  • Pass installed validation hooks.
  • Review the documentation PR.
  • Reconcile follow-ups #241 and #242 with acceptance criteria and necessary milestone membership.
  • Implement runner, hook and guidance through follow-up work.

This is a documentation-only draft under the repository workflow.

## Summary Agents repeatedly run the full test suite locally even though remote CI independently validates every PR revision. Record the maintainer-confirmed policy from the [#217 decision record](https://git.gaboggamer.online/gabogg/hikcentral/issues/217#issuecomment-4305) and [D16 confirmation](https://git.gaboggamer.online/gabogg/hikcentral/issues/217#issuecomment-4342): scoped local tests, reusable completed evidence for identical immutable inputs, and a commit hook that runs relevant tests only when matching evidence is absent. This PR complements research PR #218, which contains the measurements and the #216/#210 isolation fixes. It records the policy agreed after that research; PR #218 retains its research scope, and issue #217 remains open. Implementation remains follow-up work. ## Architectural impact - Add the accepted ADR 0009 and the consolidated policy document; link both from the documentation index. - Specify explicit test groups with full-suite fallback for shared or unknown changes, shared local evidence storage, immutable execution, approved-skip handling, and independent concurrent requests. - Retain remote full-suite CI and the head-commit merge gate. A fix after the final review requires a new review examining why validation missed the failure; testing-system defects need a separate issue. - Require research handoff to accurate, linked follow-up issues before research closure; the detailed checklist remains proposed. Follow-ups join a milestone only if delivery depends on them. Executable hooks, AGENTS.md, workflow rules and the test runner are unchanged. CPU scheduling and conflicting-run diagnosis remain separate studies. ## Related work and implementation handoff - #217: parent research and design discussion. - #218: measured findings and hermetic-suite prerequisites; its description links this policy and related open-test examples. - #241: scoped runner, evidence storage/reuse, immutable snapshots and validated selection boundaries; brief reconciled with this policy. - #242: evidence-aware hook and consistent agent/workflow guidance; brief reconciled with this policy. - #238 and #240: parallel execution and fixture optimization; reconcile withdrawn projections with the actual measured findings. - #235: separate remote-execution evaluation. The related examples on #218 include #221 (order dependence), #250 (an unforced collision), #260/#184 (test brittleness), #213 (clock and transition fidelity), #76 (timezone assumptions), #261 (migration failure-path coverage), #225 (missing integration coverage), and #60 (unexecuted platform behavior). They illustrate constraints rather than additional scope closed here. ## Verification - Merged origin/master (9a608c1) cleanly through merge commit 81d5c2a. - `python scripts/check_docs.py` passed: 45 Markdown files and 90 HTTP operations. - `ruff check .` and `ruff format --check .` passed (156 files formatted). - Installed commit hooks passed, including full pytest, on fixes commit f919ba6. - Docs-only draft remains draft. #218 is still unmerged; its branch research link has an explicit pending relative-path switch. ## Checklist - [x] Maintainer confirmed the consolidated design. - [x] Record the policy and accepted ADR. - [x] Link the documentation and research handoff. - [ ] Replace the #218 branch research link with a relative path once #218 merges. - [x] Pass installed validation hooks. - [ ] Review the documentation PR. - [x] Reconcile follow-ups #241 and #242 with acceptance criteria and necessary milestone membership. - [ ] Implement runner, hook and guidance through follow-up work. This is a documentation-only draft under the repository workflow.
docs(tests): require actionable research follow-up issues
All checks were successful
CI / lint-and-test (pull_request) Successful in 2m47s
Details
4c1845bd8f
gabogg reviewed 2026-10-03 12:38:17 +00:00
gabogg left a comment
Copy link

Code review, pass 1 (origin/master...4c1845b, spec: maintainer-confirmed policy from the #217 design interview)

Result: 2 P2s and 5 P3s. Not mergeable yet.

  • check_docs passes. It merges cleanly with master and with #218: they edit different lines of docs/README.md, and only #218 adds docs/research/README.md.

P2

  1. The maintainer's confirmation isn't recorded anywhere a reader can check.
    • ADR 0009 (lines 15–16) and the policy doc (lines 3–4) say "the maintainer confirmed … in the #217 interview".
    • The #217 thread ends with the 03:59 triage comment, and #242, #235 and #238–#241 contain no interview either. None of the roughly 15 settled decisions can be traced to a maintainer statement.
    • Two of them are new process rules beyond the brief:
      • a new review after a CI fix that lands after the final review (rfc lines 51–56);
      • a handoff rule before research closes (rfc lines 116–121).
    • Fix: post the confirmed decision record on #217 (once the maintainer confirms it), and link it from the ADR.
  2. The follow-up issues contradict the accepted ADR.
    • #241 still asks for a filename-based mapping from app/ to tests, which rfc lines 112–113 call unsafe.
    • #242 still offers a path-filter or status-quo option.
    • The PR's own checklist leaves "reconcile follow-ups" unticked, and under the rule this PR adds (lines 116–121) that has to happen before merge.
    • Fix: update #241 and #242 to match the policy.

P3

  1. ADR 0009 doesn't match the format of its siblings. Lines 1–5 have no date:, and the title lacks the ADR 0009: prefix that 0005–0008 use.
  2. The filename starts with rfc-, but the doc is titled and indexed as an accepted policy. Rename it.
  3. #218's research doc will move when #218 merges. Link it with a relative path then.
  4. #265 is in Milestone 3, but the milestone description doesn't name it (milestones.md, "side PR").
  5. The research-follow-up rule conflicts with #266. rfc lines 117–118 put every research follow-up into "the research milestone", but #266 makes milestone membership "only if the milestone cannot be completed without it". Align the wording.
## Code review, pass 1 (`origin/master...4c1845b`, spec: maintainer-confirmed policy from the #217 design interview) Result: **2 P2s and 5 P3s. Not mergeable yet.** - `check_docs` passes. It merges cleanly with master and with #218: they edit different lines of `docs/README.md`, and only #218 adds `docs/research/README.md`. ### P2 1. **The maintainer's confirmation isn't recorded anywhere a reader can check.** - ADR 0009 (lines 15–16) and the policy doc (lines 3–4) say "the maintainer confirmed … in the #217 interview". - The #217 thread ends with the 03:59 triage comment, and #242, #235 and #238–#241 contain no interview either. None of the roughly 15 settled decisions can be traced to a maintainer statement. - Two of them are new process rules beyond the brief: - a new review after a CI fix that lands after the final review (rfc lines 51–56); - a handoff rule before research closes (rfc lines 116–121). - Fix: post the confirmed decision record on #217 (once the maintainer confirms it), and link it from the ADR. 2. **The follow-up issues contradict the accepted ADR.** - #241 still asks for a filename-based mapping from app/ to tests, which rfc lines 112–113 call unsafe. - #242 still offers a path-filter or status-quo option. - The PR's own checklist leaves "reconcile follow-ups" unticked, and under the rule this PR adds (lines 116–121) that has to happen before merge. - Fix: update #241 and #242 to match the policy. ### P3 3. **ADR 0009 doesn't match the format of its siblings.** Lines 1–5 have no `date:`, and the title lacks the `ADR 0009:` prefix that 0005–0008 use. 4. **The filename starts with `rfc-`,** but the doc is titled and indexed as an accepted policy. Rename it. 5. **#218's research doc will move when #218 merges.** Link it with a relative path then. 6. **#265 is in Milestone 3, but the milestone description doesn't name it** (`milestones.md`, "side PR"). 7. **The research-follow-up rule conflicts with #266.** rfc lines 117–118 put every research follow-up into "the research milestone", but #266 makes milestone membership "only if the milestone cannot be completed without it". Align the wording.
docs(tests): reconcile pass 1 with recorded decisions
All checks were successful
CI / lint-and-test (pull_request) Successful in 3m23s
Details
f919ba668f
gabogg commented 2026-10-03 13:09:47 +00:00
Author
Owner
Copy link

Pass 1 fixes (f919ba6)

Addressed r50 on top of origin/master (merge commit 81d5c2a):

  • P2-1: ADR 0009 and the policy link the #217 decision record (c4305) and subsequent maintainer D16 confirmation (c4342). Detailed evidence schema/input inventory, implementation sequence and handoff checklist are explicitly proposed; confirmed CI-fix review and research handoff rules remain.
  • P2-2: Rewrote #241 around explicit versioned groups with indirect consumers and conservative fallback, removing filename-based app-to-tests mapping. Rewrote #242 as the evidence-aware hook/guidance implementation, removing path-filter and status-quo alternatives; updated its title. Ticked reconcile follow-ups.
  • P3-3/4: Added date: 2026-10-03 and ADR 0009:; renamed the policy to docs/architecture/agent-test-validation-policy.md and updated the ADR/index links.
  • P3-5: #218 remains unmerged. Linked its branch research document and recorded the pending switch to ../research/217-test-suite-speed.md once it lands; no nonexistent relative link was added.
  • P3-6: Milestone 3 names #265, its policy path, and why this side PR is required.
  • P3-7: Research follow-ups join only if the milestone cannot complete without them, matching #266. The detailed handoff checklist remains proposed.

Validation: documentation checker passed (45 Markdown files, 90 HTTP operations), Ruff lint/format checks passed, and installed commit hooks including full pytest passed. The worktree is clean and fixes are pushed. PR #265 remains a docs-only draft.

Please perform pass 2 against f919ba6, including the reconciled #241/#242 briefs and milestone description. The relative research-link switch remains conditional on #218 merging.

## Pass 1 fixes (f919ba6) Addressed r50 on top of origin/master (merge commit 81d5c2a): - **P2-1:** ADR 0009 and the policy link the #217 decision record (c4305) and subsequent maintainer D16 confirmation (c4342). Detailed evidence schema/input inventory, implementation sequence and handoff checklist are explicitly proposed; confirmed CI-fix review and research handoff rules remain. - **P2-2:** Rewrote #241 around explicit versioned groups with indirect consumers and conservative fallback, removing filename-based app-to-tests mapping. Rewrote #242 as the evidence-aware hook/guidance implementation, removing path-filter and status-quo alternatives; updated its title. Ticked reconcile follow-ups. - **P3-3/4:** Added `date: 2026-10-03` and `ADR 0009:`; renamed the policy to `docs/architecture/agent-test-validation-policy.md` and updated the ADR/index links. - **P3-5:** #218 remains unmerged. Linked its branch research document and recorded the pending switch to `../research/217-test-suite-speed.md` once it lands; no nonexistent relative link was added. - **P3-6:** Milestone 3 names #265, its policy path, and why this side PR is required. - **P3-7:** Research follow-ups join only if the milestone cannot complete without them, matching #266. The detailed handoff checklist remains proposed. Validation: documentation checker passed (45 Markdown files, 90 HTTP operations), Ruff lint/format checks passed, and installed commit hooks including full pytest passed. The worktree is clean and fixes are pushed. PR #265 remains a docs-only draft. Please perform pass 2 against f919ba6, including the reconciled #241/#242 briefs and milestone description. The relative research-link switch remains conditional on #218 merging.
gabogg reviewed 2026-10-03 18:00:43 +00:00
gabogg left a comment
Copy link

Code review, pass 2 (origin/master...f919ba6, spec: maintainer-confirmed policy from the #217 design interview)

Result: 1 P2 and 3 P3s. Close to mergeable; the P2 is a one-line link fix in the PR plus the same path in #241/#242.

  • CI is green on f919ba6 (CI / lint-and-test). scripts/check_docs.py passes in a git-archive export: 45 Markdown files, 90 HTTP operations. It checks only local links, so it can't catch the broken external link below.
  • Mergeability. git merge-tree --write-tree origin/master f919ba668f exits 0 with no conflicting files. The branch already contains master 9a608c1 through merge 81d5c2a. The PR also merges cleanly with #218 (4abfff3). Forgejo's mergeable: false comes from the PR's draft/WIP: state: Forgejo's Mergeable() returns false for work-in-progress PRs. Every other open non-draft PR without conflicts reports True, and #265 is the only draft. It clears when the PR is promoted, so it isn't a conflict.

Previous findings (r50)

# Finding Status Evidence
P2-1 No checkable maintainer confirmation FIXED ADR 0009:16-17 and policy:3-6 link #217 c4305 (decision record D1–D16) and c4342 (D16 in the maintainer's own words). Both comments exist and say what is cited. Policy:93-97 and :114-115 label the evidence schema, delivery order and handoff checklist "proposed", matching the "Proposed, not confirmed" notes in c4305 D13/D14/D16.
P2-2 #241/#242 contradict the ADR FIXED (one carried-over defect, see P2 below) #241 now requires an explicit versioned group map and forbids inferring tests from filenames. #242 drops the path-filter, smoke-only and status-quo options and gets a new title. Both cite c4305/c4342 and give a milestone justification that follows #266.
P3-3 ADR date and title prefix FIXED date: 2026-10-03, # ADR 0009: …. Matches 0005–0008.
P3-4 rfc- filename FIXED Renamed to docs/architecture/agent-test-validation-policy.md. The README index and ADR links are updated, and no rfc-agent-test reference remains.
P3-5 #218 research link NOT FIXED (the fix introduced a broken link) See P2-1 below.
P3-6 M3 description doesn't name #265 FIXED Milestone 3 now reads: "Side documentation PR #265 records ADR 0009 and the confirmed validation policy required to specify this delivery…"
P3-7 Research follow-ups vs #266 FIXED Policy:132-134 says "joins … only if the milestone cannot be completed without it" and links ../agents/milestones.md#membership. The anchor exists (## Membership, milestones.md:13). This matches c4305 D16 and c4342.

P2

  1. The #218 research link and the planned replacement path point to a file that has never existed. CONFIRMED.
    • Where: docs/architecture/agent-test-validation-policy.md:154-156.
    • What's wrong:
      • The link goes to …/src/branch/fix/test-isolation/docs/research/217-test-suite-speed.md and returns HTTP 404.
      • The real document on #218 (head 4abfff3) is docs/research/217-test-performance-and-agent-workflows.md, which returns 200. git log 4abfff3 -- docs/research/217-test-suite-speed.md is empty: that name never existed.
    • Failure scenario:
      • A reader following the only reference for the "Verified current behavior" claim (policy:79-80, shared-machine load and no CPU-per-test data) gets a 404.
      • Whoever applies the note "replace this with ../research/217-test-suite-speed.md" after #218 lands creates a broken relative link. check_docs would then fail CI.
    • Same wrong path elsewhere: the "Decision sources" sections of #241 and #242 cite it too.
    • Fix: use 217-test-performance-and-agent-workflows.md in the policy and in both issue bodies.

P3

  1. Three confirmed clauses were dropped when the decisions were condensed. CONFIRMED against c4305. #241/#242 carry all three, so this is a policy-doc fidelity gap, not a contract gap.
    • Policy:19-20 says "a failed or pending CI revision". D1 says "a failed, pending or missing CI result cannot authorize a merge". scripts/wt merge does handle a missing result (the ci_gate "not started" path dies after CI_START_GRACE). #242 req. 4 says "missing" too.
    • Policy:37-40 omits D9's "A subset pass cannot certify the full suite."
    • Policy:127-130 omits D16's confirmed clause that related open test issues are linked as examples, distinguished from work the research PR actually delivered.
  2. Policy:6 wraps inconsistently. That one line is about 120 columns, while the rest of the file wraps at about 80. This is cosmetic.
  3. The policy names #241, #242, #221 and #235 as follow-ups, but not #238 or #240. c4305 ("Consolidated confirmation and scope") lists both as direct follow-up candidates to reconcile.
    • Their current bodies still contain the withdrawn claims, and neither has been reconciled:
      • #238 cites a 25–35 s projection, cites xdist database is locked errors, and asks to "configure -n auto" without the D3/D15 measurement precondition;
      • #240 claims "80%+ … unbatched INSERT" and a 15–20 s saving.
    • Both cite the nonexistent path docs/research/test-performance-and-agent-workflows.md (missing the 217- prefix).
    • Both are ready-for-agent in M3.
    • The D16 handoff for these belongs mainly to #218. #218's research doc:315-318 already flags them, and line 318 still calls #242 "proposed hook policy, awaiting maintainer decision", which is now stale.
    • Fix: list #238/#240 in policy:151-161, or explicitly leave their reconciliation to #218. Either way, reconcile them before an AFK agent picks them up.

Prose accuracy against code (all verified at master 9a608c1 unless noted)

  • Pre-commit runs full pytest unconditionally: .pre-commit-config.yaml, local pytest hook with always_run: true and pass_filenames: false. Correct.

  • CI runs full pytest, and tests/test_frontend_modules.py invokes node --test tests/frontend/*.test.js (and asserts that node exists, so it doesn't skip). Correct.

  • scripts/wt merge (scripts/wt:251-335):

    • polls commits/$sha/status on the PR head SHA;
    • dies on failure/error/warning/cancelled, on no run after the grace period, and on timeout;
    • sends head_commit_id;
    • checks draft and mergeable but not review state.

    So "Review completion is a separate gate" is accurate.

  • The selection groups name tests that exist: test_doors_reconciliation, test_cycle_verdict, occupancy/calibration/statistics files, test_frontend_modules, test_static_assets, test_i18n.

  • ADR frontmatter/title matches 0005–0008. README index entries resolve.

  • The #218 research claims at policy:79-80 match its doc (load "observed, not controlled", lines 24 and 71; no per-test CPU or simultaneous-agent throughput).

  • The milestones anchor resolves.

## Code review, pass 2 (`origin/master...f919ba6`, spec: maintainer-confirmed policy from the #217 design interview) Result: **1 P2 and 3 P3s. Close to mergeable; the P2 is a one-line link fix in the PR plus the same path in #241/#242.** - CI is green on f919ba6 (`CI / lint-and-test`). `scripts/check_docs.py` passes in a git-archive export: 45 Markdown files, 90 HTTP operations. It checks only local links, so it can't catch the broken external link below. - **Mergeability.** `git merge-tree --write-tree origin/master f919ba668f` exits 0 with no conflicting files. The branch already contains master 9a608c1 through merge 81d5c2a. The PR also merges cleanly with #218 (4abfff3). Forgejo's `mergeable: false` comes from the PR's draft/`WIP:` state: Forgejo's `Mergeable()` returns false for work-in-progress PRs. Every other open non-draft PR without conflicts reports True, and #265 is the only draft. It clears when the PR is promoted, so it isn't a conflict. ### Previous findings (r50) | # | Finding | Status | Evidence | |---|---|---|---| | P2-1 | No checkable maintainer confirmation | FIXED | ADR 0009:16-17 and policy:3-6 link #217 c4305 (decision record D1–D16) and c4342 (D16 in the maintainer's own words). Both comments exist and say what is cited. Policy:93-97 and :114-115 label the evidence schema, delivery order and handoff checklist "proposed", matching the "Proposed, not confirmed" notes in c4305 D13/D14/D16. | | P2-2 | #241/#242 contradict the ADR | FIXED (one carried-over defect, see P2 below) | #241 now requires an explicit versioned group map and forbids inferring tests from filenames. #242 drops the path-filter, smoke-only and status-quo options and gets a new title. Both cite c4305/c4342 and give a milestone justification that follows #266. | | P3-3 | ADR date and title prefix | FIXED | `date: 2026-10-03`, `# ADR 0009: …`. Matches 0005–0008. | | P3-4 | `rfc-` filename | FIXED | Renamed to `docs/architecture/agent-test-validation-policy.md`. The README index and ADR links are updated, and no `rfc-agent-test` reference remains. | | P3-5 | #218 research link | NOT FIXED (the fix introduced a broken link) | See P2-1 below. | | P3-6 | M3 description doesn't name #265 | FIXED | Milestone 3 now reads: "Side documentation PR #265 records ADR 0009 and the confirmed validation policy required to specify this delivery…" | | P3-7 | Research follow-ups vs #266 | FIXED | Policy:132-134 says "joins … only if the milestone cannot be completed without it" and links `../agents/milestones.md#membership`. The anchor exists (`## Membership`, milestones.md:13). This matches c4305 D16 and c4342. | ### P2 1. **The #218 research link and the planned replacement path point to a file that has never existed.** CONFIRMED. - **Where:** `docs/architecture/agent-test-validation-policy.md:154-156`. - **What's wrong:** - The link goes to `…/src/branch/fix/test-isolation/docs/research/217-test-suite-speed.md` and returns **HTTP 404**. - The real document on #218 (head 4abfff3) is `docs/research/217-test-performance-and-agent-workflows.md`, which returns 200. `git log 4abfff3 -- docs/research/217-test-suite-speed.md` is empty: that name never existed. - **Failure scenario:** - A reader following the only reference for the "Verified current behavior" claim (policy:79-80, shared-machine load and no CPU-per-test data) gets a 404. - Whoever applies the note "replace this with `../research/217-test-suite-speed.md`" after #218 lands creates a broken relative link. `check_docs` would then fail CI. - **Same wrong path elsewhere:** the "Decision sources" sections of #241 and #242 cite it too. - **Fix:** use `217-test-performance-and-agent-workflows.md` in the policy and in both issue bodies. ### P3 2. **Three confirmed clauses were dropped when the decisions were condensed.** CONFIRMED against c4305. #241/#242 carry all three, so this is a policy-doc fidelity gap, not a contract gap. - **Policy:19-20 says "a failed or pending CI revision".** D1 says "a failed, pending **or missing** CI result cannot authorize a merge". `scripts/wt merge` does handle a missing result (the `ci_gate` "not started" path dies after `CI_START_GRACE`). #242 req. 4 says "missing" too. - **Policy:37-40 omits D9's "A subset pass cannot certify the full suite."** - **Policy:127-130 omits D16's confirmed clause** that related open test issues are linked as examples, distinguished from work the research PR actually delivered. 3. **Policy:6 wraps inconsistently.** That one line is about 120 columns, while the rest of the file wraps at about 80. This is cosmetic. 4. **The policy names #241, #242, #221 and #235 as follow-ups, but not #238 or #240.** c4305 ("Consolidated confirmation and scope") lists both as direct follow-up candidates to reconcile. - Their current bodies still contain the withdrawn claims, and neither has been reconciled: - #238 cites a 25–35 s projection, cites xdist `database is locked` errors, and asks to "configure `-n auto`" without the D3/D15 measurement precondition; - #240 claims "80%+ … unbatched INSERT" and a 15–20 s saving. - Both cite the nonexistent path `docs/research/test-performance-and-agent-workflows.md` (missing the `217-` prefix). - Both are `ready-for-agent` in M3. - The D16 handoff for these belongs mainly to #218. #218's research doc:315-318 already flags them, and line 318 still calls #242 "proposed hook policy, awaiting maintainer decision", which is now stale. - Fix: list #238/#240 in policy:151-161, or explicitly leave their reconciliation to #218. Either way, reconcile them before an AFK agent picks them up. ### Prose accuracy against code (all verified at master 9a608c1 unless noted) - Pre-commit runs full pytest unconditionally: `.pre-commit-config.yaml`, local `pytest` hook with `always_run: true` and `pass_filenames: false`. Correct. - CI runs full `pytest`, and `tests/test_frontend_modules.py` invokes `node --test tests/frontend/*.test.js` (and asserts that node exists, so it doesn't skip). Correct. - `scripts/wt merge` (scripts/wt:251-335): - polls `commits/$sha/status` on the PR head SHA; - dies on failure/error/warning/cancelled, on no run after the grace period, and on timeout; - sends `head_commit_id`; - checks draft and mergeable but not review state. So "Review completion is a separate gate" is accurate. - The selection groups name tests that exist: `test_doors_reconciliation`, `test_cycle_verdict`, occupancy/calibration/statistics files, `test_frontend_modules`, `test_static_assets`, `test_i18n`. - ADR frontmatter/title matches 0005–0008. README index entries resolve. - The #218 research claims at policy:79-80 match its doc (load "observed, not controlled", lines 24 and 71; no per-test CPU or simultaneous-agent throughput). - The milestones anchor resolves.
docs(tests): fix research document link for PR #218
All checks were successful
CI / lint-and-test (pull_request) Successful in 3m18s
Details
19b0c5a052
gabogg commented 2026-10-03 20:18:39 +00:00
Author
Owner
Copy link

Pass 2 fixes (19b0c5a)

Addressed r54 P2 on top of f919ba6:

  • P2: Corrected the #218 research document reference from docs/research/217-test-suite-speed.md to docs/research/217-test-performance-and-agent-workflows.md in docs/architecture/agent-test-validation-policy.md:154-156 (both the branch link and the future replacement note). Also patched the same link in the bodies of issues #241 and #242 via tea api (--data @file.json) to point to the valid path on the #218 branch.
  • P3s (P3-2, P3-3, P3-4): Already filed and tracked in #271 outside Milestone 3 per review pass 2.

Validation:

  • python3 scripts/check_docs.py passed (45 Markdown files, 90 HTTP operations).
  • Pre-commit hooks passed cleanly on commit (Ruff format, Ruff lint, and pytest full suite).
  • Branch docs/test-policy-217 pushed to origin (19b0c5a).

Please perform pass 3 review against 19b0c5a.

## Pass 2 fixes (19b0c5a) Addressed r54 P2 on top of f919ba6: - **P2:** Corrected the #218 research document reference from `docs/research/217-test-suite-speed.md` to `docs/research/217-test-performance-and-agent-workflows.md` in `docs/architecture/agent-test-validation-policy.md:154-156` (both the branch link and the future replacement note). Also patched the same link in the bodies of issues #241 and #242 via `tea api` (`--data @file.json`) to point to the valid path on the #218 branch. - **P3s (P3-2, P3-3, P3-4):** Already filed and tracked in #271 outside Milestone 3 per review pass 2. Validation: - `python3 scripts/check_docs.py` passed (45 Markdown files, 90 HTTP operations). - Pre-commit hooks passed cleanly on commit (Ruff format, Ruff lint, and pytest full suite). - Branch `docs/test-policy-217` pushed to origin (`19b0c5a`). Please perform pass 3 review against 19b0c5a.
All checks were successful
CI / lint-and-test (pull_request) Successful in 3m18s
Details
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin docs/test-policy-217:docs/test-policy-217
git switch docs/test-policy-217

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master
git merge --no-ff docs/test-policy-217
git switch docs/test-policy-217
git rebase master
git switch master
git merge --ff-only docs/test-policy-217
git switch docs/test-policy-217
git rebase master
git switch master
git merge --no-ff docs/test-policy-217
git switch master
git merge --squash docs/test-policy-217
git switch master
git merge --ff-only docs/test-policy-217
git switch master
git merge docs/test-policy-217
git push origin master
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked
Waits on the issues listed in its "Blocked by:" line before implementation

  
bug
Category: something is broken; behaviour differs from the spec or docs

  
enhancement
Category: new feature or improvement, including refactors, chores and docs

  
high-priority
Address before ordinary-priority work; independent of triage readiness and blockers

  
low-priority
Address after ordinary-priority work; independent of triage readiness and blockers

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

  
referenced
Previously closed issue reopened or referenced by an open issue; visible history marker

  
research
Needs investigation (internet or local resources) before it can be triaged down; used alongside a category and a triage role

  
wontfix
Will not be actioned

No labels
blocked
bug
enhancement
high-priority
low-priority
needs-info
needs-triage
ready-for-agent
ready-for-human
referenced
research
wontfix
Milestone
Clear milestone
No items
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
gabogg/hikcentral!265
No description provided.