feat(statistics): data-quality marker inputs for the deck #138
No reviewers
Labels
No labels
blocked
bug
enhancement
high-priority
low-priority
needs-info
needs-triage
ready-for-agent
ready-for-human
referenced
research
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
gabogg/hikcentral!138
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/deck-marker-inputs"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #129
Stacked on #137 (#128, the ranked cycle verdict), which defines "excluded" and "unverified". The base is
fix/cycle-verdict-ranking, so the diff shows only #129. Merge #137 first; Forgejo retargets this PR tomasterwhen that branch is deleted.Summary
These are the route inputs the statistics deck's data-quality marker needs (RFC §4.1, decided 2026-09-26):
peak,busiest_day(week),best_day(month) andbusiest_hour(day) never come from an excluded day, and are omitted when every covered day is excluded. Totals and averages still include marked days.HourlyFlowBucketof the presenter hourly route carriesgap_estimated, so the Day view can hatch exactly the gap buckets at 15, 30 or 60 minutes.missing_days,unverified_days, and the tier totalsunreliable_daysandestimate_days, which is what the top-bar badge shows.DailyStatistics.gapslists each estimated ingestion gap as{start_epoch, end_epoch}, clipped to the cycle, for the marker's detail panel ("counter gap 14:00–15:10").Architectural impact
app/services/analytics_service.py:day_quality_state()is the one rule behind the tier counts, withESTIMATE_STATESandUNRELIABLE_STATES;_gaps_within()is the one gap-overlap rule, shared by the daily rows, their hours and the intraday buckets. It replaces the inlineany(...).bucket=hour. The hourly series reads them once per cycle.app/schemas/statistics.py):GapInterval;DailyStatistics.gaps(required, empty when there are none);HourlyFlowBucket.gap_estimated;PeriodQualitygainsmissing_days,unverified_days,estimate_daysandunreliable_days, whichSelectablePeriodinherits, so the picker can mark periods;StatisticsSummary.peakbecomes optional and is omitted like the other unavailable sections. An entirely closed period still returnspeakwith zero people inside.gap_estimated_daysandexcluded_dayskeep counting the raw flags, so existing clients see no change in meaning./api/analytics/timeseries/hourlygets the extra bucket key too; it is additive.Beyond the issue's literal list: besides
unverified_daysandmissing_days, I added the two tier totals, because the badge shows exactly those and the client can't derive them from the raw flag counts (a day can be both gap-estimated and excluded).busiest_hourfollows the records rule too.Verification
New
tests/test_statistics_marker_inputs.py(8 tests, real in-memory SQLite, no mocks); all failed before the change:peakandbusiest_hour;pytest: 402 passed, 1 skipped (on top of #137). Frontend: 83/83. Ruff, pre-commit andscripts/check_docs.pypassed.Conflict note: PR #132 edits the same summary and period-quality code. Whichever lands later rebases.
Checklist
HourlyFlowBucket.gap_estimated.PeriodQuality(worst state, counted once).DailyStatistics.gaps.🤖 Generated with Claude Code
Standards
(a) Documented Standards Violations
None. The diff is fully compliant with repository standards:
AGENTS.md): SQL queries remain encapsulated in repository methods; anomaly fetching is hoisted cleanly inanalytics_service.py.docs/standards/code-standards.md): Python 3.11+ syntax, explicit type annotations, Pydantic v2 models inapp/schemas/statistics.py.CONTEXT.md): Follows the data-quality marker terminology (missing_days,unverified_days,estimate_days,unreliable_days).docs/standards/code-standards.md): Offline, deterministic tests intests/test_statistics_marker_inputs.py; test suite is 100% green.docs/standards/git-and-workflow.md): Branch namefeat/deck-marker-inputsand conventional commit adhere to guidelines.(b) Baseline Smells (Judgement Calls)
app/services/analytics_service.pyday_quality_state(*, has_data: bool, excluded: bool, trusted: bool, gap_estimated: bool)passes 4 booleans together. Unpacks dictionary fields at a single call site inget_statistics_period_quality_async; grouping into a small helper type would be cleaner.app/services/analytics_service.py_gaps_withinqueries multiple properties ofIngestionAnomaly(kind,start_epoch,end_epoch) to perform interval clipping. Could be a method onIngestionAnomaly(e.g.anomaly.clip_to(start, end)), though defensible as a module-level translation helper toGapInterval.Spec
(a) Missing or partial requirements
"Expose each estimated gap's start and end per daily row (e.g. gaps: [{start_epoch, end_epoch}], only when gap_estimated)"app/schemas/statistics.py,DailyStatistics.gapsis typed aslist[GapInterval]and defaults/returns empty list[]on days without gaps rather than being omitted/null.(b) Behaviour in the diff not asked for (scope creep)
"Add counts so the top-bar badge can show both tiers without fetching the daily series: unverified_days... missing_days..."estimate_daysandunreliable_daysroll-ups toPeriodQualityin addition to the requested fields (author documented this as badge inputs in PR description).busiest_houron single-day summary"totals and averages keep including marked days, but the records never come from an unreliable (excluded or missing) day: peak... best_day (month) and busiest_day (week)."get_statistics_summary_async,busiest_houris also suppressed when the day is excluded (if hour and reliable:).(c) Requirements implemented but implementation looks wrong
"A day in several states counts once, in its worst tier. Document the precedence: excluded > missing > unverified > estimated."PeriodQuality.gap_estimated_daysandexcluded_daysretain raw flag accumulation (gap_days += int(row["has_gap"])), so a day that is both excluded and gap-estimated increments both counts. (The PR author noted this was intentional for backward compatibility, while tier totals use the worst-state model).Summary: Standards: 2 findings (worst: 4-boolean Data Clump in
day_quality_state); Spec: 4 findings (worst: dual-counting in raw period quality flags when a day is both excluded and gap-estimated).0459c65e10to903c49323eReview fixes:
903c493All findings from the review are addressed or clarified below.
Standards
day_quality_state(*, has_data, excluded, trusted, gap_estimated)_gaps_withinqueries multiple properties ofIngestionAnomalyGapIntervalrecords.Spec
DailyStatistics.gapsreturns[]on non-gap days instead of null/omitteddocs/api/README.md.estimate_days,unreliable_days) onPeriodQualitybusiest_houron single-day summarypeak,busiest_day,best_day).gap_estimated_daysandexcluded_daysretain raw flag accumulation for backward compatibility; new badge counters strictly use the single worst-state tier model.Integration & Rebase
origin/fix/cycle-verdict-ranking, resolving conflicts inapp/services/analytics_service.pyanddocs/api/README.md. Preserved typed_shared_labelfrom PR #132.tests/test_statistics_marker_inputs.pywithScheduleCalendarforget_statistics_period_quality_async.Code Review — Second Pass (PR #138)
Diff reviewed:
master...feat/deck-marker-inputs(commits:a7a05b5,903c493)Standards
(a) Documented Standards Compliance
docs/standards/code-standards.md§2): Pass 1 finding resolved.EstimatedGapIntervalinapp/schemas/statistics.pygives gap intervals explicit Pydantic domain modeling;PeriodQualityandHourlyFlowBucketmatch spec contracts.AGENTS.md§1, §2): Deep module conventions respected; marker logic is encapsulated inanalytics_service.pywith no SQL leakage into schemas or presentation layers.AGENTS.md§2): Fully async, explicit Python 3.11+ type annotations throughout. All 416 tests pass,ruff checkandruff formatare clean.(b) Baseline Smells (Judgement Calls)
app/services/analytics_service.py:186):DayQualityState = Literal["EXCLUDED", "MISSING", "UNVERIFIED", "ESTIMATED", "OK"]works cleanly inside the service; could optionally become a sharedStrEnuminapp/schemas/statistics.pyif needed by future exports.app/schemas/statistics.py:30-40):= 0default to the new tier counter fields (missing_days,unverified_days,estimate_days,unreliable_days) aligns withclosed_days: int = 0.Spec
(a) Requirements Missing or Partial
None. All 5 spec changes from Issue #129 and all 3 test scenarios are fully implemented and verified:
peak,best_day,busiest_day, and Day viewbusiest_hour) skip unreliable days and omit when all days are unreliable; closed periods maintainpeak=0.gap_estimated: boolonHourlyFlowBucketwith exact boundary clipping.PeriodQuality(unverified_days,missing_days,estimate_days,unreliable_days) with documented precedence (excluded > missing > unverified > estimated).gaps: list[EstimatedGapInterval]exposed on daily series rows whengap_estimated=True.docs/api/README.md.(b) Behaviour Not Asked For (Scope Creep)
estimate_daysandunreliable_daysinPeriodQuality(P3):(c) Requirements Implemented That Look Wrong
None. Precedence resolution and interval handling are accurate. All 416 tests pass (100% green).
One-line summary
Follow-up
Per directive, all non-blocking P3 cleanup suggestions have been consolidated into follow-up issue #143 (follow-up(statistics): P3 cleanups from PR #138 review).
feat(statistics): data-quality marker inputs for the deck (#129)to feat(statistics): data-quality marker inputs for the deck