fix(tests): cardholder-name retry test depends on test order #269

New issue

Open

opened 2026-10-03 12:56:24 +00:00 by gabogg · 0 comments

gabogg commented

2026-10-03 12:56:24 +00:00

Owner

Found by PR #233 (#221) while shuffling the suite: the full 604-test collection with seeds 233 and 264. The reproduction is recorded in #233's PR body.

Finding

tests/test_cardholder_name_resolution.py::test_failed_lookup_is_retried_after_a_pause_not_every_poll sees lookup calls ['69', '67'] instead of ['67'] when test_forced_open_with_card_keeps_forced_trigger runs before it. Some cardholder lookup or retry state survives between tests.

Why it is in this milestone

A hermetic test loop requires every test to pass in any order and in isolation. Without that, parallel runs (#238) and scoped test selection (#241) give results that depend on order.

Acceptance Criteria

Identify the leaked state (a module-level cache, a retry/backoff table, or a singleton) and reset it with the shared test reset (tests/occupancy_reset.py or its successor). Don't reorder or skip tests.
The two-test reproduction passes in both orders.
Shuffle tests/test_cardholder_name_resolution.py together with the door tests using 2–3 seeds, and report what you find.

Found by PR #233 (#221) while shuffling the suite: the full 604-test collection with seeds 233 and 264. The reproduction is recorded in #233's PR body. ### Finding `tests/test_cardholder_name_resolution.py::test_failed_lookup_is_retried_after_a_pause_not_every_poll` sees lookup calls `['69', '67']` instead of `['67']` when `test_forced_open_with_card_keeps_forced_trigger` runs before it. Some cardholder lookup or retry state survives between tests. ### Why it is in this milestone A hermetic test loop requires every test to pass in any order and in isolation. Without that, parallel runs (#238) and scoped test selection (#241) give results that depend on order. ### Acceptance Criteria - Identify the leaked state (a module-level cache, a retry/backoff table, or a singleton) and reset it with the shared test reset (`tests/occupancy_reset.py` or its successor). Don't reorder or skip tests. - The two-test reproduction passes in both orders. - Shuffle `tests/test_cardholder_name_resolution.py` together with the door tests using 2–3 seeds, and report what you find.

gabogg added this to the Fast, hermetic agent test loop milestone

2026-10-03 12:56:24 +00:00

gabogg added the

ready-for-agent

bug

labels

2026-10-03 12:56:24 +00:00

gabogg referenced this issue

2026-10-03 13:10:44 +00:00

fix(tests): keep tests inside their sandbox and cut suite time #218

gabogg referenced this issue from a commit

2026-10-03 13:12:37 +00:00