feat(statistics): data-quality marker inputs for the deck (records skip unreliable days, per-bucket gap flag) #129

New issue

Closed

opened 2026-09-26 10:04:51 +00:00 by gabogg · 0 comments

gabogg commented

2026-09-26 10:04:51 +00:00

Owner

Route changes the statistics deck's data-quality marker needs. Decided with the maintainer on 2026-09-26 while settling the open points of the deck RFC (#80); the marker's design is in the RFC. The verdict rule itself (which verdict makes a day excluded or trusted) is #128; this issue builds on it.

Marker model (context, decided)

Each business day that is not a Closed Day is in one of five states:

State	Condition	Marker tier
OK	trusted verdict, no gap estimate	—
Estimated	an ingestion gap was estimated (`gap_estimated`)	estimate
Unverified	no current verdict yet, or pending audit (`trusted=false`, not excluded)	estimate
Excluded	the ranked verdict excludes it (auto, manual or quarantine; #128)	unreliable
Missing	`has_data=false` and not closed	unreliable

The marker ignores min_data_trust (no trust-score threshold, as decided in #106). The field stays for admin use.

Changes

Records skip unreliable days. In get_statistics_summary_async (analytics_service.py), totals and averages keep including marked days, but the records never come from an unreliable (excluded or missing) day:
- peak (highest people inside);
- best_day (month) and busiest_day (week).
If every covered day is unreliable, return the record as absent (null, omitted like the other optional sections) rather than quoting a broken day. Missing days already have no data, so in practice this is "skip excluded days".
Per-bucket gap flag on the intraday route. Add gap_estimated: bool to HourlyFlowBucket (app/schemas/statistics.py), using the same gap-overlap rule as the daily series' HourlyVisitors.gap_estimated, at the requested bucket_minutes (15, 30 or 60). The Day view hatches those buckets on "visitors by hour" and "people inside over the day". The response-level gap_estimated stays.
Tier counts on the period quality. PeriodQuality has gap_estimated_days and excluded_days. Add counts so the top-bar badge can show both tiers without fetching the daily series:
- unverified_days: no current verdict, or not trusted and not excluded;
- missing_days: no data, not closed.
These must use the same day classification as the daily rows. A day in several states counts once, in its worst tier. Document the precedence: excluded > missing > unverified > estimated.
Gap intervals for the detail panel. The marker's detail panel (numpad *) lists each marked day in plain language, e.g. TUE 15 · counter gap 14:00–15:10, estimated. Expose each estimated gap's start and end per daily row (e.g. gaps: [{start_epoch, end_epoch}], only when gap_estimated), reusing the gap records the gap rule already reads.
Update docs/api/README.md for the new fields and the records rule.

Tests

A period where the highest peak or busiest day is on an excluded day: the record comes from the best non-excluded day; totals still include the excluded day. All days excluded: the record is omitted.
A day with a gap from 14:05 to 14:50: at bucket_minutes=15, exactly the overlapping buckets are flagged; at 60, only the 14:00 bucket.
Tier counts over a period mixing all five states, including a day that is both gap-estimated and excluded (counted once, as excluded).

Depends on #128 (the ranked verdict defines "excluded" and "unverified"). Refs #80, #106.

🤖 Generated with Claude Code

Route changes the statistics deck's **data-quality marker** needs. Decided with the maintainer on 2026-09-26 while settling the open points of the deck RFC (#80); the marker's design is in the RFC. The verdict rule itself (which verdict makes a day excluded or trusted) is #128; this issue builds on it. ## Marker model (context, decided) Each business day that is not a Closed Day is in one of five states: | State | Condition | Marker tier | |---|---|---| | OK | trusted verdict, no gap estimate | — | | Estimated | an ingestion gap was estimated (`gap_estimated`) | **estimate** | | Unverified | no current verdict yet, or pending audit (`trusted=false`, not excluded) | **estimate** | | Excluded | the ranked verdict excludes it (auto, manual or quarantine; #128) | **unreliable** | | Missing | `has_data=false` and not closed | **unreliable** | The marker ignores `min_data_trust` (no trust-score threshold, as decided in #106). The field stays for admin use. ## Changes - [ ] **Records skip unreliable days.** In `get_statistics_summary_async` (`analytics_service.py`), totals and averages keep including marked days, but the records never come from an unreliable (excluded or missing) day: - `peak` (highest people inside); - `best_day` (month) and `busiest_day` (week). If every covered day is unreliable, return the record as absent (`null`, omitted like the other optional sections) rather than quoting a broken day. Missing days already have no data, so in practice this is "skip excluded days". - [ ] **Per-bucket gap flag on the intraday route.** Add `gap_estimated: bool` to `HourlyFlowBucket` (`app/schemas/statistics.py`), using the same gap-overlap rule as the daily series' `HourlyVisitors.gap_estimated`, at the requested `bucket_minutes` (15, 30 or 60). The Day view hatches those buckets on "visitors by hour" and "people inside over the day". The response-level `gap_estimated` stays. - [ ] **Tier counts on the period quality.** `PeriodQuality` has `gap_estimated_days` and `excluded_days`. Add counts so the top-bar badge can show both tiers without fetching the daily series: - `unverified_days`: no current verdict, or not trusted and not excluded; - `missing_days`: no data, not closed. These must use the same day classification as the daily rows. A day in several states counts once, in its worst tier. Document the precedence: excluded > missing > unverified > estimated. - [ ] **Gap intervals for the detail panel.** The marker's detail panel (numpad `*`) lists each marked day in plain language, e.g. `TUE 15 · counter gap 14:00–15:10, estimated`. Expose each estimated gap's start and end per daily row (e.g. `gaps: [{start_epoch, end_epoch}]`, only when `gap_estimated`), reusing the gap records the gap rule already reads. - [ ] Update `docs/api/README.md` for the new fields and the records rule. ## Tests - A period where the highest peak or busiest day is on an excluded day: the record comes from the best non-excluded day; totals still include the excluded day. All days excluded: the record is omitted. - A day with a gap from 14:05 to 14:50: at `bucket_minutes=15`, exactly the overlapping buckets are flagged; at 60, only the 14:00 bucket. - Tier counts over a period mixing all five states, including a day that is both gap-estimated and excluded (counted once, as excluded). Depends on #128 (the ranked verdict defines "excluded" and "unverified"). Refs #80, #106. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

gabogg added the

ready-for-agent

label

2026-09-26 10:04:51 +00:00

gabogg referenced this issue from a commit

2026-09-26 10:08:57 +00:00

docs(statistics-deck): settle the open points and fold in the 2026-09-25/26 decisions

gabogg referenced this issue

2026-09-26 10:09:32 +00:00

docs(statistics-deck): zero-scroll display model and the projector paradox #80

gabogg referenced this issue

2026-09-26 10:09:34 +00:00