gabogg/hikcentral
Watch
1
0
Fork
You've already forked hikcentral
 0
Code Issues 113 Pull requests 8 Projects Releases Packages Wiki Activity Actions

[data-veracity] Portal attribution is fabricated: group deltas are split evenly across member cameras #28

New issue
Closed
opened 2026-09-21 13:43:28 +00:00 by gabogg · 4 comments
gabogg commented 2026-09-21 13:43:28 +00:00
Owner
Copy link

Filed from a data-veracity audit of the ingestion and aggregation pipeline on master, carried out against the KPI set that the Executive Statistics Deck (PR #20 / RFC-ARCH-2026-004) intends to publish. Each issue names the deck KPIs it corrupts.

Problem

Artemis returns passenger counts per resource group, not per camera. The sync invents a per-camera breakdown by integer division:

# app/services/occupancy_service.py:1903-1909
base_in = delta_in // num_in
rem_in  = delta_in %  num_in
for idx, (c_code, c_name) in enumerate(in_cams):
    c_delta = base_in + (1 if idx < rem_in else 0)

Every member camera of a group therefore receives the same share of that group's traffic, plus at most one remainder unit. Over a full cycle each camera in an n-camera group converges to exactly 1/n of the group total, regardless of how many people actually walked through it.

The deck's Portal Traversal Attribution panel (present in all three horizons — RFC-ARCH-2026-004 §4.1.4, §4.2.4, §4.3.4) renders flow_share_pct per portal. With this ingestion those percentages are an artifact of integer division, not a measurement. A three-camera group will always read 33.3% / 33.3% / 33.3%.

It is also mis-directed, not just evenly smeared

Membership of in_cams / out_cams is decided by keyword matching on the camera name:

# app/services/occupancy_service.py:341  infer_camera_direction_and_zone
is_entrance = "ENTRADA" in uname or "ACCESO" in uname or "INGRESS" in uname
is_exit     = "SALIDA"  in uname or "EGRESS" in uname

ACCESO is the ordinary Spanish word for a portal, and in this facility the portals are named exactly that — ACCESO NORTE, ACCESO SUR (the RFC's own examples). Both are classified ENTRANCE and land only in in_cams. ESTACIONAMIENTO matches nothing, falls through to BIDIRECTIONAL, and lands in both lists (:1825-1826).

Result for a group {ACCESO NORTE, ACCESO SUR, ESTACIONAMIENTO}: out_cams == [ESTACIONAMIENTO], so 100% of facility egress is attributed to the parking camera, while the two main doors are recorded as entrance-only. The portal table then shows two portals with zero exits and one portal carrying every exit in the mall — none of which happened.

Worse: a group with no bidirectional or exit-named camera

"out_cams": out_cams or [(g_code, g_name)]

If keyword matching leaves out_cams empty, events are written with camera_index_code = <resource group code>. See the separate orphan-camera-code issue — that path is a runaway.

KPIs corrupted

Portal Attribution (Day / Week / Month), flow_share_pct, is_asymmetric_anomaly, the NATURAL_INGRESS_PORTAL / NATURAL_EGRESS_PORTAL topology described in CONTEXT.md §3, and any per-zone breakdown built on zone_name.

Suggested fix

  1. Stop inferring direction from names. counting_cameras.direction_type is already a persisted, operator-editable column — make it authoritative and have the name heuristic only seed a suggestion at discovery time, flagged for confirmation.
  2. Find out whether this HikCentral version exposes a per-camera passenger flow endpoint (/people/advance/... by resourceIndexCode). If it does, ingest per camera and delete the division entirely.
  3. If it does not, then per-camera attribution is not measurable and the deck must say so: relabel the panel "Group Attribution", aggregate to the resource group, and drop flow_share_pct at camera granularity rather than publishing a computed-looking number that carries no information.

Needs confirmation

Whether per-camera passenger flow is available on the deployed HikCentral version. That single answer decides between fix (2) and fix (3), and fix (3) changes the RFC.

> Filed from a data-veracity audit of the ingestion and aggregation pipeline on `master`, carried out against the KPI set that the Executive Statistics Deck (PR #20 / `RFC-ARCH-2026-004`) intends to publish. Each issue names the deck KPIs it corrupts. ## Problem Artemis returns passenger counts **per resource group**, not per camera. The sync invents a per-camera breakdown by integer division: ```python # app/services/occupancy_service.py:1903-1909 base_in = delta_in // num_in rem_in = delta_in % num_in for idx, (c_code, c_name) in enumerate(in_cams): c_delta = base_in + (1 if idx < rem_in else 0) ``` Every member camera of a group therefore receives the **same** share of that group's traffic, plus at most one remainder unit. Over a full cycle each camera in an n-camera group converges to exactly `1/n` of the group total, regardless of how many people actually walked through it. The deck's **Portal Traversal Attribution** panel (present in all three horizons — `RFC-ARCH-2026-004` §4.1.4, §4.2.4, §4.3.4) renders `flow_share_pct` per portal. With this ingestion those percentages are an artifact of integer division, not a measurement. A three-camera group will always read 33.3% / 33.3% / 33.3%. ## It is also mis-directed, not just evenly smeared Membership of `in_cams` / `out_cams` is decided by keyword matching on the camera **name**: ```python # app/services/occupancy_service.py:341 infer_camera_direction_and_zone is_entrance = "ENTRADA" in uname or "ACCESO" in uname or "INGRESS" in uname is_exit = "SALIDA" in uname or "EGRESS" in uname ``` `ACCESO` is the ordinary Spanish word for a portal, and in this facility the portals are named exactly that — `ACCESO NORTE`, `ACCESO SUR` (the RFC's own examples). Both are classified `ENTRANCE` and land **only in `in_cams`**. `ESTACIONAMIENTO` matches nothing, falls through to `BIDIRECTIONAL`, and lands in **both** lists (`:1825-1826`). Result for a group `{ACCESO NORTE, ACCESO SUR, ESTACIONAMIENTO}`: `out_cams == [ESTACIONAMIENTO]`, so **100% of facility egress is attributed to the parking camera**, while the two main doors are recorded as entrance-only. The portal table then shows two portals with zero exits and one portal carrying every exit in the mall — none of which happened. ## Worse: a group with no bidirectional or exit-named camera ```python "out_cams": out_cams or [(g_code, g_name)] ``` If keyword matching leaves `out_cams` empty, events are written with `camera_index_code = <resource group code>`. See the separate orphan-camera-code issue — that path is a runaway. ## KPIs corrupted Portal Attribution (Day / Week / Month), `flow_share_pct`, `is_asymmetric_anomaly`, the `NATURAL_INGRESS_PORTAL` / `NATURAL_EGRESS_PORTAL` topology described in `CONTEXT.md` §3, and any per-zone breakdown built on `zone_name`. ## Suggested fix 1. **Stop inferring direction from names.** `counting_cameras.direction_type` is already a persisted, operator-editable column — make it authoritative and have the name heuristic only seed a *suggestion* at discovery time, flagged for confirmation. 2. Find out whether this HikCentral version exposes a **per-camera** passenger flow endpoint (`/people/advance/...` by `resourceIndexCode`). If it does, ingest per camera and delete the division entirely. 3. If it does not, then per-camera attribution is **not measurable** and the deck must say so: relabel the panel "Group Attribution", aggregate to the resource group, and drop `flow_share_pct` at camera granularity rather than publishing a computed-looking number that carries no information. ## Needs confirmation Whether per-camera passenger flow is available on the deployed HikCentral version. That single answer decides between fix (2) and fix (3), and fix (3) changes the RFC.
gabogg commented 2026-09-21 13:57:33 +00:00
Author
Owner
Copy link

Blocks #14 Candidate 2, and affects a shipped view

Two additions to scope, raised while cross-checking against #14:

  1. This blocks #14 Candidate 2 (Deepen the Backend Telemetry Streaming Seam). That candidate proposes streaming atomic PassageFluxVector (camera, direction, count, timestamp). The camera field has no real signal behind it until this issue is resolved, so the answer here — per-camera passenger flow available upstream, or not — decides whether Candidate 2 ships as PassageFluxVector or as GroupFluxVector. See #14 (comment).

  2. This is already visible in production, not just in the proposed stats deck. The delivered CommandDeckAdapter Directional Flux Stream Feed renders per-camera >>> IN / <<< OUT lines fed by the recent_event payload broadcast at app/services/occupancy_service.py:540-552, whose camera_name comes straight from the even split. With this facility's portal naming (ACCESO NORTE, ACCESO SUR → ENTRANCE; ESTACIONAMIENTO → BIDIRECTIONAL), out_cams collapses to the parking camera alone — so every <<< OUT line currently scrolling in the operator feed is attributed to it.

Suggest prioritising this above the rest of the #26–#36 set on the strength of (2).

## Blocks #14 Candidate 2, and affects a shipped view Two additions to scope, raised while cross-checking against #14: 1. **This blocks #14 Candidate 2** (*Deepen the Backend Telemetry Streaming Seam*). That candidate proposes streaming atomic `PassageFluxVector (camera, direction, count, timestamp)`. The `camera` field has no real signal behind it until this issue is resolved, so the answer here — per-camera passenger flow available upstream, or not — decides whether Candidate 2 ships as `PassageFluxVector` or as `GroupFluxVector`. See https://git.gaboggamer.online/gabogg/hikcentral/issues/14#issuecomment-839. 2. **This is already visible in production, not just in the proposed stats deck.** The delivered `CommandDeckAdapter` Directional Flux Stream Feed renders per-camera `>>> IN` / `<<< OUT` lines fed by the `recent_event` payload broadcast at `app/services/occupancy_service.py:540-552`, whose `camera_name` comes straight from the even split. With this facility's portal naming (`ACCESO NORTE`, `ACCESO SUR` → `ENTRANCE`; `ESTACIONAMIENTO` → `BIDIRECTIONAL`), `out_cams` collapses to the parking camera alone — so every `<<< OUT` line currently scrolling in the operator feed is attributed to it. Suggest prioritising this above the rest of the #26–#36 set on the strength of (2).
gabogg commented 2026-09-22 14:52:15 +00:00
Author
Owner
Copy link

✅ Design settled (grilling session)

The needs-info is answered: the Artemis catalog exposes no per-camera passenger-flow endpoint — only group-level (people/resourceGroupRealTimeCount, people/advance/resourceGroupList) and a spatial camera heatmap (people/statisticsHeatMapByTime). Per-camera attribution is therefore not measurable on this HikCentral. Fix (2) is impossible; we take fix (3) + fix (1).

Settled spec

  • Group Attribution (fix 3): relabel the deck panel "Group Attribution"; compute flow_share_pct across resource groups; list member cameras as metadata with no per-camera percentage. This removes the fabricated even-split (base_in = delta_in // num_in, :1903-1909) and changes the RFC (§4.1.4 / §4.2.4 / §4.3.4).
  • Authoritative direction_type (fix 1): counting_cameras.direction_type (database.py:230, operator-editable) becomes the source of truth for in_cams / out_cams. The name heuristic (infer_camera_direction_and_zone, :341) only seeds a suggestion at discovery, written with needs_confirmation. The sync never re-derives direction from the camera name (kills the ACCESO-as-entrance-only misdirection that put 100% of egress on the parking camera). Existing rows backfilled needs_confirmation = true for a one-time operator review.
  • Domain documentation: state the invariant that the system counts per camera-group, not per camera; a per-camera figure requires the one-camera-per-group workaround on the HikCentral side. Goes in CONTEXT.md §3 and the RFC.

Deliverables

  • User-facing explanation of the deck change (panel rename + no per-camera %), so operators understand why the granularity changed.
  • ADR (short, optional per owner's lean): "Passenger-flow attribution is per camera-group, not per camera" — records the HikCentral limitation, the one-camera-per-group workaround, and the RFC panel change.
  • Unblocks the orphan-camera-code cleanup issue (see the linked issue; it is Blocked by #28).

Acceptance criteria (supersede "Needs confirmation")

  • Deck panel relabelled "Group Attribution"; flow_share_pct at resource-group granularity; no per-camera %.
  • direction_type authoritative for in_cams/out_cams; name heuristic discovery-only with needs_confirmation; existing rows backfilled needs_confirmation = true.
  • CONTEXT.md §3 + RFC document per-camera-group counting.
  • User-facing explanation published; ADR recorded.
  • Tests: a group {ACCESO NORTE, ACCESO SUR, ESTACIONAMIENTO} with operator-set directions attributes egress correctly, not 100% to the parking camera.

Re-tagged ready-for-agent.

## ✅ Design settled (grilling session) The `needs-info` is **answered**: the Artemis catalog exposes **no per-camera passenger-flow endpoint** — only group-level (`people/resourceGroupRealTimeCount`, `people/advance/resourceGroupList`) and a *spatial* camera heatmap (`people/statisticsHeatMapByTime`). Per-camera attribution is therefore **not measurable** on this HikCentral. Fix (2) is impossible; we take fix (3) + fix (1). ### Settled spec - **Group Attribution (fix 3)**: relabel the deck panel **"Group Attribution"**; compute `flow_share_pct` **across resource groups**; list member cameras as metadata with **no per-camera percentage**. This removes the fabricated even-split (`base_in = delta_in // num_in`, `:1903-1909`) and **changes the RFC** (§4.1.4 / §4.2.4 / §4.3.4). - **Authoritative `direction_type` (fix 1)**: `counting_cameras.direction_type` (`database.py:230`, operator-editable) becomes the source of truth for `in_cams` / `out_cams`. The name heuristic (`infer_camera_direction_and_zone`, `:341`) only **seeds a suggestion at discovery**, written with `needs_confirmation`. The sync **never re-derives direction from the camera name** (kills the `ACCESO`-as-entrance-only misdirection that put 100% of egress on the parking camera). Existing rows **backfilled `needs_confirmation = true`** for a one-time operator review. - **Domain documentation**: state the invariant that the system counts per **camera-group**, not per camera; a per-camera figure requires the **one-camera-per-group** workaround on the HikCentral side. Goes in `CONTEXT.md` §3 and the RFC. ### Deliverables - **User-facing explanation** of the deck change (panel rename + no per-camera %), so operators understand why the granularity changed. - **ADR** (short, optional per owner's lean): *"Passenger-flow attribution is per camera-group, not per camera"* — records the HikCentral limitation, the one-camera-per-group workaround, and the RFC panel change. - **Unblocks the orphan-camera-code cleanup issue** (see the linked issue; it is `Blocked by #28`). ### Acceptance criteria (supersede "Needs confirmation") - [ ] Deck panel relabelled "Group Attribution"; `flow_share_pct` at resource-group granularity; no per-camera %. - [ ] `direction_type` authoritative for `in_cams`/`out_cams`; name heuristic discovery-only with `needs_confirmation`; existing rows backfilled `needs_confirmation = true`. - [ ] `CONTEXT.md` §3 + RFC document per-camera-group counting. - [ ] User-facing explanation published; ADR recorded. - [ ] Tests: a group `{ACCESO NORTE, ACCESO SUR, ESTACIONAMIENTO}` with operator-set directions attributes egress correctly, not 100% to the parking camera. Re-tagged `ready-for-agent`.
gabogg commented 2026-09-22 14:52:22 +00:00
Author
Owner
Copy link

Unblocks #51 (orphan camera code) — that cleanup is Blocked by #28 and must land after this.

Unblocks #51 (orphan camera code) — that cleanup is `Blocked by #28` and must land after this.
gabogg commented 2026-09-22 16:31:11 +00:00
Author
Owner
Copy link

⚠️ Amendment to the settled spec — fix (1) is superseded

From the grilling session of 2026-09-22, which scoped this work into two PRs. The fix (3) half of the settled spec (group-level attribution, panel relabel, no per-camera %) is unchanged. This amends fix (1) only.

What the settled spec said

Authoritative direction_type (fix 1): counting_cameras.direction_type becomes the source of truth for in_cams / out_cams.

Why that no longer applies

Artemis returns current_artemis_in / current_artemis_out per group, already split by direction (occupancy_service.py:1898-1899, from resourceGroupRealTimeCount at :1838). The in_cams / out_cams lists exist for exactly one purpose: deciding which camera rows receive that group's directional delta via the even split.

Fix (3) deletes the even split. Once events attribute to the group, there is nothing left for a camera-membership split to decide — so making direction_type authoritative for it is repairing machinery that is being removed. Keeping a correct-but-unused derivation would be speculative generality against a per-camera capability the Artemis catalog says does not exist.

What replaces it

  • The in_cams / out_cams derivation (:1814-1832) and its heuristic call (:1819) are deleted, not corrected.
  • direction_type becomes camera metadata — an operator-maintained label for what a camera physically is — not a counting input.
  • infer_camera_direction_and_zone (:341) stays discovery-only at :422 / :484, still writing needs_confirmation. needs_confirmation drops from correctness-critical to a display nicety; it is retained, but it is no longer an AC blocker.
  • The reconciliation baseline (:1891-1896) becomes a direct group lookup. This is what made #29's runaway possible — an absent camera silently summed to 0 — so the fix closes that by construction rather than by guard alone.

Two further findings from the same session

  1. occupancy_controller.py:365 holds a second, independent derivation: in_cams = [c for c in cams if c["direction_type"] in ("ENTRANCE", "BIDIRECTIONAL")] or cams. So direction_type has already been authoritative in the controller while the sync used the name heuristic — the two have diverged for as long as both have existed. It also carries the same or cams fallback shape that produced the orphan bug. Folded into phase 2.

  2. The group was never persisted. grep -rn "resource_group\|resourceGroupIndexCode" app/db/ app/schemas/ returns zero hits — the group→camera map is rebuilt transiently from Artemis each sync and discarded. Group-level flow_share_pct therefore requires a schema migration that the original scoping did not identify. That migration is phase 1.

Where the work now lives

  • Phase 1 — camera-group schema, orphan guard and migration (closes #29). Additive and defensive; no behavioural change.
  • Phase 2 — the cutover (closes this issue and #35). Group-level events, the six aggregates moving to the group join, the panel relabel, CONTEXT.md §3 and ADR 0005.

Split because emitting group-level events while the aggregates still inner-join counting_cameras would drop every event and take all published KPIs to zero between the two merges.

The RFC acceptance criteria are dropped: docs/architecture/rfc-executive-statistics-deck.md exists only inside draft PR #20 and cannot be edited outside it. ADR 0005 and CONTEXT.md carry the decision; PR #20 rebases and reconciles its own §4.x panels.

🤖 Generated with Claude Code

## ⚠️ Amendment to the settled spec — fix (1) is superseded From the grilling session of 2026-09-22, which scoped this work into two PRs. The **fix (3)** half of the settled spec (group-level attribution, panel relabel, no per-camera %) is **unchanged**. This amends **fix (1)** only. ### What the settled spec said > **Authoritative `direction_type` (fix 1)**: `counting_cameras.direction_type` becomes the source of truth for `in_cams` / `out_cams`. ### Why that no longer applies Artemis returns `current_artemis_in` / `current_artemis_out` **per group, already split by direction** (`occupancy_service.py:1898-1899`, from `resourceGroupRealTimeCount` at `:1838`). The `in_cams` / `out_cams` lists exist for exactly one purpose: deciding which camera rows receive that group's directional delta via the even split. Fix (3) deletes the even split. Once events attribute to the group, **there is nothing left for a camera-membership split to decide** — so making `direction_type` authoritative for it is repairing machinery that is being removed. Keeping a correct-but-unused derivation would be speculative generality against a per-camera capability the Artemis catalog says does not exist. ### What replaces it - The `in_cams` / `out_cams` derivation (`:1814-1832`) and its heuristic call (`:1819`) are **deleted**, not corrected. - `direction_type` becomes **camera metadata** — an operator-maintained label for what a camera physically is — not a counting input. - `infer_camera_direction_and_zone` (`:341`) stays **discovery-only** at `:422` / `:484`, still writing `needs_confirmation`. `needs_confirmation` drops from correctness-critical to a display nicety; it is retained, but it is no longer an AC blocker. - The reconciliation baseline (`:1891-1896`) becomes a direct group lookup. This is what made #29's runaway possible — an absent camera silently summed to `0` — so the fix closes that by construction rather than by guard alone. ### Two further findings from the same session 1. **`occupancy_controller.py:365` holds a second, independent derivation**: `in_cams = [c for c in cams if c["direction_type"] in ("ENTRANCE", "BIDIRECTIONAL")] or cams`. So `direction_type` has *already* been authoritative in the controller while the sync used the name heuristic — the two have diverged for as long as both have existed. It also carries the same `or cams` fallback shape that produced the orphan bug. Folded into phase 2. 2. **The group was never persisted.** `grep -rn "resource_group\|resourceGroupIndexCode" app/db/ app/schemas/` returns zero hits — the group→camera map is rebuilt transiently from Artemis each sync and discarded. Group-level `flow_share_pct` therefore requires a schema migration that the original scoping did not identify. That migration is phase 1. ### Where the work now lives - **Phase 1** — camera-group schema, orphan guard and migration (closes #29). Additive and defensive; no behavioural change. - **Phase 2** — the cutover (closes this issue and #35). Group-level events, the six aggregates moving to the group join, the panel relabel, `CONTEXT.md` §3 and ADR 0005. Split because emitting group-level events while the aggregates still inner-join `counting_cameras` would drop every event and take **all published KPIs to zero** between the two merges. The RFC acceptance criteria are dropped: `docs/architecture/rfc-executive-statistics-deck.md` exists only inside draft PR #20 and cannot be edited outside it. ADR 0005 and `CONTEXT.md` carry the decision; PR #20 rebases and reconciles its own §4.x panels. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
master
docs/test-policy-217
feat/historical-flow-publication
feat/historical-flow-curation
fix/holiday-stamping-followups
chore/schedule-p3-followups-186
fix/test-isolation
test/silent-camera-trace-199
feat/historical-flow-statistics
No results found.
Labels
Clear labels   
blocked
Waits on the issues listed in its "Blocked by:" line before implementation

  
bug
Category: something is broken; behaviour differs from the spec or docs

  
enhancement
Category: new feature or improvement, including refactors, chores and docs

  
high-priority
Address before ordinary-priority work; independent of triage readiness and blockers

  
low-priority
Address after ordinary-priority work; independent of triage readiness and blockers

  
needs-info
Waiting on reporter for more information

  
needs-triage
Maintainer needs to evaluate this issue

  
ready-for-agent
Fully specified, ready for an AFK agent

  
ready-for-human
Requires human implementation

  
referenced
Previously closed issue reopened or referenced by an open issue; visible history marker

  
research
Needs investigation (internet or local resources) before it can be triaged down; used alongside a category and a triage role

  
wontfix
Will not be actioned

No labels
blocked
bug
enhancement
high-priority
low-priority
needs-info
needs-triage
ready-for-agent
ready-for-human
referenced
research
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
gabogg/hikcentral#28
No description provided.