gabogg/hikcentral

Watch

Fork 0

test(occupancy): characterization tests and trace for single silent camera outage (#199) #248

Open

gabogg wants to merge 4 commits from test/silent-camera-trace-199 into master

gabogg commented

2026-10-03 09:01:55 +00:00

Owner

Summary

Implements Issue #199 (Phase A): documents and characterizes current system behaviour when exactly one counting camera goes silent during business hours while others report traffic, and contrasts it with a legitimately quiet camera. Changes no application runtime behaviour, preserving baseline compatibility for Phase B policy decisions under #236.

Findings per Surface (Current Behaviour & File:Line)

Passenger-flow Polling & Ingestion (app/services/occupancy_service.py:2235-2505):
- What happens: Artemis reports static cumulative counts (enterNum, exitNum). The delta calculation yields delta <= 0, so no counting events are generated for the silent camera. Because other cameras report flow, the poll succeeds with {"success": True}.
- Verdict: Silently lowers totals (facility totals undercounted; outage unnoticed).
Ingestion-Gap Detection (app/services/occupancy_service.py:2132-2148, 2415-2420):
- What happens: _passenger_flow_poll_failed_async is only invoked when the entire Artemis HTTP request fails or returns zero groups. The per-group gap check now - gap_start >= GAP_THRESHOLD_SECONDS evaluates against previous.observed_at, which updates every 1–2 seconds.
- Verdict: Ignores outage (0 anomalies recorded in ingestion_anomalies).
Cycle Completeness & Data Trust (app/services/occupancy_service.py:1040-1113):
- What happens: evaluate_cycle_integrity_async assesses facility-wide totals. Active hours (\ge 10) and footfall (\ge 5000) checks pass from reporting cameras. No data quality flags fire.
- Verdict: Ignores outage (is_trusted=True, data_trust_score=100, cycle_completeness_score=100).
Cycle Verdict (app/db/occupancy_repository.py:95-128, app/services/occupancy_service.py:1500-1530):
- What happens: The nocturnal calibration run logs the cycle with is_trusted=1 and trust_status='TRUSTED'. The ranked verdict priority treats the day as trusted.
- Verdict: Ignores outage.
Data-Quality Marker (app/services/analytics_service.py:218-230, 871-908):
- What happens: day_quality_state evaluates has_data=True, excluded=False, trusted=True, gap_estimated=False, returning "OK". Period quality reports estimate_days=0, unreliable_days=0.
- Verdict: Ignores outage (no warning marker displayed; presented as clean measurement).
Statistics Comparison & Weekday Baseline (app/services/analytics_service.py:292-330, 348-395):
- What happens: Period comparisons run under basis="TOTAL" with depressed visitor counts. get_usual_weekday_baseline_dates_async selects the candidate day as a trusted baseline sample because it has data, no gaps, and trusted verdict.
- Verdict: Silently lowers totals & corrupts baselines (skewing future comparisons downward).
Calibration Learning (app/db/occupancy_repository.py:2253-2276, app/services/occupancy_service.py:1880-1920):
- What happens: get_trusted_calibration_history_async selects the cycle because is_trusted=1 and FLAG_INGESTION_GAP is absent. The distorted I/O ratio enters the EWMA for exit multiplier k.
- Verdict: Silently distorts calibration learning (biasing future live occupancy).
Multi-Camera Group Attributon (app/services/occupancy_service.py:2438-2442):
- What happens: Ingestion divides the group delta evenly (divmod(span_count, len(cams))). If Camera 1 is silent and Camera 2 records 20 passages, each receives 10 passages.
- Verdict: Silent camera receives phantom counts and appears active, while the functioning camera is undercounted by 50%.
Camera Weights & Outage Thresholds:
- None exists: No weights, expected contribution percentages, or outage thresholds exist in app/config.py, database schemas, or service constants.
Legitimately Quiet Camera Contrast:
- A camera at an open secondary door registering zero physical passages produces identical pipeline outputs across all stages; the system cannot distinguish between a silent hardware fault and a quiet door.

Deliverables

Findings Document: docs/research/199-single-camera-outage-behaviour.md covering Q1 (offline API analysis and open questions for #236), Q2 (state and stage-by-stage trace), Q3 (weights/thresholds), quiet-camera contrast, and multi-camera groups.
Research Index: docs/research/README.md linked from docs/README.md.
Characterization Tests: tests/test_silent_camera_trace.py pinning pre-#236 behaviour across all pipeline surfaces.

Verification Evidence

Full Pytest Suite: 537 passed (100% green via commit hook).
Frontend Node Tests: 242 passed, 0 failed (node --test tests/frontend/*.test.js).
Linter & Formatter: rtk ruff check . and rtk ruff format --check . clean (0 errors, 151 files formatted).
Documentation Integrity: python3 scripts/check_docs.py clean (44 Markdown files and 80 HTTP operations checked).

Closes #199

## Summary Implements **Issue #199 (Phase A)**: documents and characterizes current system behaviour when exactly one counting camera goes silent during business hours while others report traffic, and contrasts it with a legitimately quiet camera. Changes no application runtime behaviour, preserving baseline compatibility for Phase B policy decisions under #236. ## Findings per Surface (Current Behaviour & File:Line) 1. **Passenger-flow Polling & Ingestion** ([`app/services/occupancy_service.py:2235-2505`](app/services/occupancy_service.py#L2235-L2505)): - **What happens**: Artemis reports static cumulative counts (`enterNum`, `exitNum`). The delta calculation yields `delta <= 0`, so no counting events are generated for the silent camera. Because other cameras report flow, the poll succeeds with `{"success": True}`. - **Verdict**: **Silently lowers totals** (facility totals undercounted; outage unnoticed). 2. **Ingestion-Gap Detection** ([`app/services/occupancy_service.py:2132-2148, 2415-2420`](app/services/occupancy_service.py#L2132-L2148)): - **What happens**: `_passenger_flow_poll_failed_async` is only invoked when the entire Artemis HTTP request fails or returns zero groups. The per-group gap check `now - gap_start >= GAP_THRESHOLD_SECONDS` evaluates against `previous.observed_at`, which updates every 1–2 seconds. - **Verdict**: **Ignores outage** (0 anomalies recorded in `ingestion_anomalies`). 3. **Cycle Completeness & Data Trust** ([`app/services/occupancy_service.py:1040-1113`](app/services/occupancy_service.py#L1040-L1113)): - **What happens**: `evaluate_cycle_integrity_async` assesses facility-wide totals. Active hours ($\ge 10$) and footfall ($\ge 5000$) checks pass from reporting cameras. No data quality flags fire. - **Verdict**: **Ignores outage** (`is_trusted=True`, `data_trust_score=100`, `cycle_completeness_score=100`). 4. **Cycle Verdict** ([`app/db/occupancy_repository.py:95-128`](app/db/occupancy_repository.py#L95-L128), [`app/services/occupancy_service.py:1500-1530`](app/services/occupancy_service.py#L1500-L1530)): - **What happens**: The nocturnal calibration run logs the cycle with `is_trusted=1` and `trust_status='TRUSTED'`. The ranked verdict priority treats the day as trusted. - **Verdict**: **Ignores outage**. 5. **Data-Quality Marker** ([`app/services/analytics_service.py:218-230, 871-908`](app/services/analytics_service.py#L218-L230)): - **What happens**: `day_quality_state` evaluates `has_data=True`, `excluded=False`, `trusted=True`, `gap_estimated=False`, returning `"OK"`. Period quality reports `estimate_days=0`, `unreliable_days=0`. - **Verdict**: **Ignores outage** (no warning marker displayed; presented as clean measurement). 6. **Statistics Comparison & Weekday Baseline** ([`app/services/analytics_service.py:292-330, 348-395`](app/services/analytics_service.py#L292-L330)): - **What happens**: Period comparisons run under `basis="TOTAL"` with depressed visitor counts. `get_usual_weekday_baseline_dates_async` selects the candidate day as a trusted baseline sample because it has data, no gaps, and trusted verdict. - **Verdict**: **Silently lowers totals & corrupts baselines** (skewing future comparisons downward). 7. **Calibration Learning** ([`app/db/occupancy_repository.py:2253-2276`](app/db/occupancy_repository.py#L2253-L2276), [`app/services/occupancy_service.py:1880-1920`](app/services/occupancy_service.py#L1880-L1920)): - **What happens**: `get_trusted_calibration_history_async` selects the cycle because `is_trusted=1` and `FLAG_INGESTION_GAP` is absent. The distorted I/O ratio enters the EWMA for exit multiplier $k$. - **Verdict**: **Silently distorts calibration learning** (biasing future live occupancy). 8. **Multi-Camera Group Attributon** ([`app/services/occupancy_service.py:2438-2442`](app/services/occupancy_service.py#L2438-L2442)): - **What happens**: Ingestion divides the group delta evenly (`divmod(span_count, len(cams))`). If Camera 1 is silent and Camera 2 records 20 passages, each receives 10 passages. - **Verdict**: Silent camera receives phantom counts and appears active, while the functioning camera is undercounted by 50%. 9. **Camera Weights & Outage Thresholds**: - **None exists**: No weights, expected contribution percentages, or outage thresholds exist in `app/config.py`, database schemas, or service constants. 10. **Legitimately Quiet Camera Contrast**: - A camera at an open secondary door registering zero physical passages produces identical pipeline outputs across all stages; the system cannot distinguish between a silent hardware fault and a quiet door. ## Deliverables 1. **Findings Document**: [`docs/research/199-single-camera-outage-behaviour.md`](docs/research/199-single-camera-outage-behaviour.md) covering Q1 (offline API analysis and open questions for #236), Q2 (state and stage-by-stage trace), Q3 (weights/thresholds), quiet-camera contrast, and multi-camera groups. 2. **Research Index**: [`docs/research/README.md`](docs/research/README.md) linked from [`docs/README.md`](docs/README.md). 3. **Characterization Tests**: [`tests/test_silent_camera_trace.py`](tests/test_silent_camera_trace.py) pinning pre-#236 behaviour across all pipeline surfaces. ## Verification Evidence - **Full Pytest Suite**: 537 passed (100% green via commit hook). - **Frontend Node Tests**: 242 passed, 0 failed (`node --test tests/frontend/*.test.js`). - **Linter & Formatter**: `rtk ruff check .` and `rtk ruff format --check .` clean (0 errors, 151 files formatted). - **Documentation Integrity**: `python3 scripts/check_docs.py` clean (44 Markdown files and 80 HTTP operations checked). Closes #199

gabogg added this to the Single silent counting camera milestone

2026-10-03 09:01:55 +00:00

gabogg added 1 commit

2026-10-03 09:01:56 +00:00

test(occupancy): characterization tests and trace for single silent camera outage (#199 )

CI / lint-and-test (pull_request) Successful in 2m40s

Details

1dd0db7489

gabogg changed title from ~~WIP: test(occupancy): characterization tests and trace for single silent camera outage (#199)~~ to test(occupancy): characterization tests and trace for single silent camera outage (#199)

2026-10-03 09:02:03 +00:00

gabogg added the

ready-for-agent

label

2026-10-03 09:02:04 +00:00

gabogg referenced this pull request

2026-10-03 09:02:23 +00:00

research(occupancy): decide the policy for a single silent counting camera #236

gabogg commented

2026-10-03 09:02:32 +00:00

Author

Owner

Requesting code review pass 1 for 016a012...1dd0db7, spec #199. Phase A investigation complete: findings documented in docs/research/199-single-camera-outage-behaviour.md, research index added, open questions for live testing posted to #236, and characterization tests pinning current behaviour across all surfaces added in tests/test_silent_camera_trace.py. Full pytest (537 passed), frontend node tests (242 passed), ruff, and check_docs pass cleanly.

gabogg reviewed

2026-10-03 11:27:39 +00:00

gabogg left a comment

Code review, pass 1 (`origin/master...1dd0db7`, spec #199 phase A)

Result: 6 P2s and 13 P3s. Not mergeable yet. This is pass 1, so fix every finding, then rebase or merge onto origin/master (9a72ff4) first, since most file:line references and one traced signature changed with #178 and #177. Request a second pass after that.

Good news:
- No production code is changed.
- The 3 tests pass on the merge with master (5.4 s).
- ruff and check_docs.py are clean.
- The doc follows the docs/research/<issue>-<slug>.md and index convention.
- The open questions were copied to #236.
The main problem: the trace covers one failure mode (a frozen counter giving zero deltas), and several stages assert values the test set up itself rather than what the pipeline produced. #236 will build policy on this write-up, so it has to describe what the code does.

Spec

P2

The "group missing from the response" mode is untested, and the write-up describes it wrongly. The spec asks for "one camera returns nothing (or zero)", and the doc says an omitted group "emits no events, and logs no errors".
- Probe on the merged tree: group gb was left out of count_list for 6 minutes, then returned.
- On return the app recorded IngestionAnomaly(group_code='gb', kind='gap', delta_in=30) and spread the backlog over the interval (is_gap, occupancy_service.py ~3478 on master).
- Gap anomalies set gap_estimated, so the day gets the ESTIMATE marker (analytics_service.py ~241, ~896).
- So "does not detect the outage at any layer" holds only for the frozen-counter (zero-delta) mode.
- Fix: trace and pin both modes, and correct the summary and stage 2.
Surfaces not traced at all:
- live occupancy (the live number while one camera is silent);
- the drift and retroactive audit (apply_retroactive_audit_async, and the drift reconcile);
- the dashboard trust tiles.
"Published totals" are checked only as raw people_counting_events rows. Assert a published aggregate.
The quiet-camera test stops at evaluate_cycle_integrity_async (test:362-368). The spec asks the quiet scenario to assert the marker, comparison eligibility and calibration inclusion too.
Stages 3–6 of the silent test hold by construction.
- Stage 3 seeds events for only two cameras (so cam-out-2 is silent too) and passes hand-picked total_in/out (test:192-215).
- Stage 4 inserts a TRUSTED log by hand (test:224-242) instead of running the nocturnal job, then calls day_quality_state with literals (test:244).
- Stage 5's -25.0 is arithmetic on the literals 6000 and 8000.
Nothing links the silent interval to the verdict. Run the real nocturnal and verdict path over the day the poll ingested. A stage that can't be driven end to end should say so, rather than claim "pins end-to-end".
Every file:line reference is stale against master, in both the doc and the test docstrings.
- Examples: evaluate_cycle_integrity_async 1040 → ~2055, is_gap 2415 → ~3478, get_trusted_calibration_history_async 2253 → ~2991.
- get_trusted_calibration_history_async's signature also changed: it now takes reset_time/now_epoch and filters on pending CALIBRATION_REEVAL.
- Stage 7 doesn't mention #177's next-day activation of learning.
- Rebase and re-anchor. Cite symbols, or pin GitHub-style line links to a commit SHA, rather than master-relative lines.

P3

Q1 overstates the evidence. The OpenAPI entry has data: {"type": "object"} with no schema. The shape in the doc (total, pageNo, pageSize) is invented. "No fields indicate sensor health" really means "the parser reads only enterNum/exitNum; the docs specify nothing more". Say that.
"previous.observed_at updates every 1–2 seconds" is true only while the group stays in the response.
The production DB claim is wrong. The doc says production is "not directly accessible over WireGuard/SSH", but it is reachable read-only over SSH via WireGuard. Write "not attempted by this agent (no access from the agent environment)". The maintainer can run the query if #236 needs it.
"Working camera cut by 50%" in the multi-camera case assumes HikCentral still sums the group while one member is down. Move it to #236's open questions; it isn't a finding.

Standards

P2

The test re-implements the shared DB fixture (tests/test_silent_camera_trace.py:44-56). The autouse isolated_camera_trace_db repeats conftest's opt-in isolated_repository_db line for line. Use pytestmark = pytest.mark.usefixtures("isolated_repository_db"), as test_business_cycle_labels.py:10 does, plus a small fixture for the reset time and manager setup.

P3

Duplicated Code: _mock_artemis (:59-75) is a verbatim copy of tests/test_counting_integrity.py:59-75. Share it.
Duplicated Code / Data Clumps: the same 5-key camera dict is upserted by hand 8 times (:127-136, 236-253, 327-344). Add a _register_camera(code, name, direction) helper.
Unused imports: ScheduleCalendar, facility_now, PeriodQuality (:23, 26, 33). Ruff misses them only because F401 is ignored under tests/*.
Raw SQL reads of people_counting_events (:110-115, 277-280, 366-369). Use a repository read where one exists.
The doc's links are anchored to lines (#L2235-L2302, etc.). Same fix as P2-5.
The commit has no Co-Authored-By trailer.

## Code review, pass 1 (`origin/master...1dd0db7`, spec #199 phase A) Result: **6 P2s and 13 P3s. Not mergeable yet.** This is pass 1, so fix every finding, then **rebase or merge onto `origin/master` (`9a72ff4`) first**, since most file:line references and one traced signature changed with #178 and #177. Request a **second pass** after that. - **Good news:** - No production code is changed. - The 3 tests pass on the merge with master (5.4 s). - ruff and `check_docs.py` are clean. - The doc follows the `docs/research/<issue>-<slug>.md` and index convention. - The open questions were copied to #236. - **The main problem:** the trace covers one failure mode (a frozen counter giving zero deltas), and several stages assert values the test set up itself rather than what the pipeline produced. #236 will build policy on this write-up, so it has to describe what the code does. ## Spec ### P2 1. **The "group missing from the response" mode is untested, and the write-up describes it wrongly.** The spec asks for *"one camera returns nothing (or zero)"*, and the doc says an omitted group "emits no events, and logs no errors". - Probe on the merged tree: group `gb` was left out of `count_list` for 6 minutes, then returned. - On return the app recorded `IngestionAnomaly(group_code='gb', kind='gap', delta_in=30)` and spread the backlog over the interval (`is_gap`, `occupancy_service.py` ~3478 on master). - Gap anomalies set `gap_estimated`, so the day gets the ESTIMATE marker (`analytics_service.py` ~241, ~896). - So "does not detect the outage at any layer" holds only for the frozen-counter (zero-delta) mode. - Fix: trace and pin **both** modes, and correct the summary and stage 2. 2. **Surfaces not traced at all:** - live occupancy (the live number while one camera is silent); - the drift and retroactive audit (`apply_retroactive_audit_async`, and the drift reconcile); - the dashboard trust tiles. "Published totals" are checked only as raw `people_counting_events` rows. Assert a published aggregate. 3. **The quiet-camera test stops at `evaluate_cycle_integrity_async`** (test:362-368). The spec asks the quiet scenario to assert the marker, comparison eligibility and calibration inclusion too. 4. **Stages 3–6 of the silent test hold by construction.** - Stage 3 seeds events for only two cameras (so `cam-out-2` is silent too) and passes hand-picked `total_in/out` (test:192-215). - Stage 4 inserts a `TRUSTED` log by hand (test:224-242) instead of running the nocturnal job, then calls `day_quality_state` with literals (test:244). - Stage 5's `-25.0` is arithmetic on the literals 6000 and 8000. Nothing links the silent interval to the verdict. Run the real nocturnal and verdict path over the day the poll ingested. A stage that can't be driven end to end should say so, rather than claim "pins end-to-end". 5. **Every file:line reference is stale against master**, in both the doc and the test docstrings. - Examples: `evaluate_cycle_integrity_async` 1040 → ~2055, `is_gap` 2415 → ~3478, `get_trusted_calibration_history_async` 2253 → ~2991. - `get_trusted_calibration_history_async`'s signature also changed: it now takes `reset_time`/`now_epoch` and filters on pending `CALIBRATION_REEVAL`. - Stage 7 doesn't mention #177's next-day activation of learning. - Rebase and re-anchor. Cite symbols, or pin GitHub-style line links to a commit SHA, rather than master-relative lines. ### P3 6. **Q1 overstates the evidence.** The OpenAPI entry has `data: {"type": "object"}` with no schema. The shape in the doc (`total`, `pageNo`, `pageSize`) is invented. "No fields indicate sensor health" really means "the parser reads only `enterNum`/`exitNum`; the docs specify nothing more". Say that. 7. **"`previous.observed_at` updates every 1–2 seconds"** is true only while the group stays in the response. 8. **The production DB claim is wrong.** The doc says production is "not directly accessible over WireGuard/SSH", but it is reachable read-only over SSH via WireGuard. Write "not attempted by this agent (no access from the agent environment)". The maintainer can run the query if #236 needs it. 9. **"Working camera cut by 50%" in the multi-camera case** assumes HikCentral still sums the group while one member is down. Move it to #236's open questions; it isn't a finding. ## Standards ### P2 10. **The test re-implements the shared DB fixture** (`tests/test_silent_camera_trace.py:44-56`). The autouse `isolated_camera_trace_db` repeats conftest's opt-in `isolated_repository_db` line for line. Use `pytestmark = pytest.mark.usefixtures("isolated_repository_db")`, as `test_business_cycle_labels.py:10` does, plus a small fixture for the reset time and manager setup. ### P3 11. **Duplicated Code: `_mock_artemis`** (:59-75) is a verbatim copy of `tests/test_counting_integrity.py:59-75`. Share it. 12. **Duplicated Code / Data Clumps:** the same 5-key camera dict is upserted by hand 8 times (:127-136, 236-253, 327-344). Add a `_register_camera(code, name, direction)` helper. 13. **Unused imports:** `ScheduleCalendar`, `facility_now`, `PeriodQuality` (:23, 26, 33). Ruff misses them only because F401 is ignored under `tests/*`. 14. **Raw SQL reads of `people_counting_events`** (:110-115, 277-280, 366-369). Use a repository read where one exists. 15. **The doc's links are anchored to lines** (`#L2235-L2302`, etc.). Same fix as P2-5. 16. **The commit has no `Co-Authored-By` trailer.**

gabogg added 3 commits

2026-10-03 13:54:24 +00:00

Merge remote-tracking branch 'origin/master' into test/silent-camera-trace-199 df17de2421

fix(occupancy): address review pass 1 on silent camera outage characterization (#199 ) a1f22ba815

- Pin both outage modes: frozen counter (Mode 1) and omitted group returning after gap (Mode 2)
- Trace missing surfaces: live occupancy, runtime drift, retroactive headcount audit, trust tiles, and published aggregates
- Drive stages 3-6 through the real nocturnal calibration job and ranked cycle verdict
- Complete legitimately quiet camera scenario across markers, comparisons, baselines, and calibration learning
- Re-anchor file and line references to symbols and commit SHA
- Refine Q1 evidence to enterNum/exitNum only, correct observed_at and production DB wording
- Move 50% multi-camera claim to #236 open questions
- Use isolated_repository_db fixture, extract shared _mock_artemis to conftest, and add _register_camera helper
- Remove unused imports and raw SQL queries on people_counting_events

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into test/silent-camera-trace-199

CI / lint-and-test (pull_request) Successful in 3m28s

Details

99cd4b3032

Resolve tests/conftest.py to preserve #233's reset_test_occupancy_state
and test isolation while keeping shared _mock_artemis helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

CI / lint-and-test (pull_request) Successful in 3m28s

Details

This pull request can be merged automatically.

This branch is out-of-date with the base branch

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin test/silent-camera-trace-199:test/silent-camera-trace-199

git switch test/silent-camera-trace-199

Merge

Merge the changes and update on Forgejo.

Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.

git switch master

git merge --no-ff test/silent-camera-trace-199

git switch test/silent-camera-trace-199

git rebase master

git switch master

git merge --ff-only test/silent-camera-trace-199

git switch test/silent-camera-trace-199

git rebase master

git switch master

git merge --no-ff test/silent-camera-trace-199

git switch master

git merge --squash test/silent-camera-trace-199