feat(cli): add scripts/wt test runner with scoped change detection and compact output #241

New issue

Open

opened 2026-10-03 08:11:26 +00:00 by gabogg · 0 comments

gabogg commented

2026-10-03 08:11:26 +00:00

Owner

Goal

Implement scripts/wt test with validated scoped selection, immutable execution and reusable completed local evidence. Remote CI independently runs the full suite on every PR revision and gates merging.

Decision sources

ADR 0009 and validation policy, documentation draft #265.
Confirmed #217 decisions D1–D16 and D16 confirmation.
Research PR #218: docs/research/217-test-performance-and-agent-workflows.md (not yet merged). Withdrawn projections and older briefs are superseded by the measured findings and these decisions.

Requirements and acceptance criteria

Support changed-scope and full-suite requests with compact success summaries and complete failure diagnostics. Full local execution is an explicit option and fallback, not mandatory before every commit.
Use an explicit versioned path-to-test-group map including indirect consumers; do not infer corresponding tests from app/test filenames. Union groups for changes spanning areas. Shared initialization, configuration, schemas, dependencies, test infrastructure and unmapped paths select the full suite. Validate boundaries against recent changes before enabling selection.
Starting boundaries to validate: door/access-cycle/reconciliation/API; combined occupancy/statistics for occupancy, calibration, analytics or schedules; frontend wrapper/static-asset/i18n/relevant-security tests for frontend assets. These are not a proven complete map.
Execute an isolated immutable snapshot of observable source, configuration and applicable Git context. Match complete snapshot identity, equivalent command/settings and environment. Any source/test change invalidates earlier evidence; HEAD or index identity alone is insufficient.
Store separate completed records atomically in shared local storage outside worktrees. Reuse only completed passing evidence, explicitly reporting the earlier result and covered scope. A full pass can satisfy a subset only with unchanged matching inputs and demonstrated execution of every requested test; subset evidence cannot certify the full suite.
Require every requested test to pass except explicitly approved skips with recorded conditions. Missing tools, failed/incomplete runs and zero exit status without coverage do not qualify.
Concurrent requests run independently without joining or waiting for in-progress runs; retain serial execution within each request initially. Preserve disagreeing results for investigation; no permanent precedence, retry or blacklist policy has been confirmed.
Verify broad/unknown fallbacks, multi-area selection, changed/unchanged inputs, superset coverage, partial staging and post-format contents, skips/missing tools, independent requests and snapshot isolation with deterministic tests.

Implementation decisions still open

Design and validate the exact snapshot mechanism, fingerprint/record schema, exhaustive inputs, retention/exclusions and skip-approval mechanism. These details are proposed implementation work, not confirmed interview decisions. CPU/scheduling research does not block this initial release; limits require measurements.

Dependencies and milestone

#242 consumes this runner and evidence interface. Coordinate with #218's hermetic isolation fixes. This issue belongs to Milestone 3 because its scoped runner and evidence reuse are required to deliver the fast, hermetic agent test loop; unrelated review follow-ups stay outside unless delivery depends on them.

## Goal Implement `scripts/wt test` with validated scoped selection, immutable execution and reusable completed local evidence. Remote CI independently runs the full suite on every PR revision and gates merging. ## Decision sources - [ADR 0009](https://git.gaboggamer.online/gabogg/hikcentral/src/branch/docs/test-policy-217/docs/adr/0009-scoped-local-tests-with-independent-ci.md) and [validation policy](https://git.gaboggamer.online/gabogg/hikcentral/src/branch/docs/test-policy-217/docs/architecture/agent-test-validation-policy.md), documentation draft #265. - [Confirmed #217 decisions D1–D16](https://git.gaboggamer.online/gabogg/hikcentral/issues/217#issuecomment-4305) and [D16 confirmation](https://git.gaboggamer.online/gabogg/hikcentral/issues/217#issuecomment-4342). - Research PR #218: [docs/research/217-test-performance-and-agent-workflows.md](https://git.gaboggamer.online/gabogg/hikcentral/src/branch/fix/test-isolation/docs/research/217-test-performance-and-agent-workflows.md) (not yet merged). Withdrawn projections and older briefs are superseded by the measured findings and these decisions. ## Requirements and acceptance criteria 1. Support changed-scope and full-suite requests with compact success summaries and complete failure diagnostics. Full local execution is an explicit option and fallback, not mandatory before every commit. 2. Use an explicit versioned path-to-test-group map including indirect consumers; do not infer corresponding tests from app/test filenames. Union groups for changes spanning areas. Shared initialization, configuration, schemas, dependencies, test infrastructure and unmapped paths select the full suite. Validate boundaries against recent changes before enabling selection. 3. Starting boundaries to validate: door/access-cycle/reconciliation/API; combined occupancy/statistics for occupancy, calibration, analytics or schedules; frontend wrapper/static-asset/i18n/relevant-security tests for frontend assets. These are not a proven complete map. 4. Execute an isolated immutable snapshot of observable source, configuration and applicable Git context. Match complete snapshot identity, equivalent command/settings and environment. Any source/test change invalidates earlier evidence; HEAD or index identity alone is insufficient. 5. Store separate completed records atomically in shared local storage outside worktrees. Reuse only completed passing evidence, explicitly reporting the earlier result and covered scope. A full pass can satisfy a subset only with unchanged matching inputs and demonstrated execution of every requested test; subset evidence cannot certify the full suite. 6. Require every requested test to pass except explicitly approved skips with recorded conditions. Missing tools, failed/incomplete runs and zero exit status without coverage do not qualify. 7. Concurrent requests run independently without joining or waiting for in-progress runs; retain serial execution within each request initially. Preserve disagreeing results for investigation; no permanent precedence, retry or blacklist policy has been confirmed. 8. Verify broad/unknown fallbacks, multi-area selection, changed/unchanged inputs, superset coverage, partial staging and post-format contents, skips/missing tools, independent requests and snapshot isolation with deterministic tests. ## Implementation decisions still open Design and validate the exact snapshot mechanism, fingerprint/record schema, exhaustive inputs, retention/exclusions and skip-approval mechanism. These details are proposed implementation work, not confirmed interview decisions. CPU/scheduling research does not block this initial release; limits require measurements. ## Dependencies and milestone #242 consumes this runner and evidence interface. Coordinate with #218's hermetic isolation fixes. This issue belongs to Milestone 3 because its scoped runner and evidence reuse are required to deliver the fast, hermetic agent test loop; unrelated review follow-ups stay outside unless delivery depends on them.

gabogg added this to the Fast, hermetic agent test loop milestone

2026-10-03 08:11:26 +00:00

gabogg added the

ready-for-agent

label

2026-10-03 08:11:26 +00:00

gabogg referenced this issue

2026-10-03 08:11:56 +00:00

fix(tests): keep tests inside their sandbox and cut suite time #218

gabogg referenced this issue

2026-10-03 08:12:18 +00:00

fix(tests): keep tests inside their sandbox and cut suite time #218