gabogg/hikcentral

Watch

Fork 0

feat(occupancy): reviewed monthly historical passenger-flow backfill #160

New issue

Open

opened 2026-09-27 15:24:29 +00:00 by gabogg · 1 comment

gabogg commented

2026-09-27 15:24:29 +00:00

Owner

Research question: can we import historical passenger-flow data from HikCentral to retroactively fill our database, so the statistics deck (and calibration) have history from before this system started recording?

Why

The prod DB only has counted events from 2026-09-03. Consequences:

no closed month exists yet;
every "vs last year" comparison is hidden;
the Usual Weekday Baseline often reports too few samples;
the Week of 31 Aug shows Mon–Wed as missing.

HikCentral has been counting for much longer.

What we know

The code today only reads live state from Artemis:
- …/aiapplication/v1/people/advance/resourceGroupList (group totals; occupancy_service.py:2077);
- …/people/resourceGroupRealTimeCount.
- Events arrive going forward via the monitor and webhook.
The bundled catalog (app/docs/artemis_catalog.json, app/docs/artemis_openapi.json) lists …/people/statisticsHeatMapByTime. Its description is per-camera heat map by time. Nothing is obviously a historical passenger-flow report.
HikCentral Pro's web client (reachable through our Bumblebee/ISAPI session) has people-counting reports with export. That may be a second source.
Our model stores per-group IN/OUT events with timestamps (ADR 0005: one camera per group), plus cycle-quality, calibration and audit records. Startup already runs reconcile_and_quarantine_historical_anomalies_async over history (see docs/design-history/occupancy-calibration-model-redesign.md, "Startup Backfill Requirement").

Questions to answer

Source. Which HikCentral API (Artemis OpenAPI, or the web/ISAPI report or export endpoints) returns historical people-counting data per resource group or camera? Answer from primary sources (the HikCentral Pro OpenAPI docs for our version, V2.6.3). Capture granularity (per event, per minute, per hour), retention, and paging limits.
Fidelity. Is the data per event or aggregated? If it's aggregated (e.g. hourly), how do we represent it without faking events? Options include synthetic events at bucket midpoints flagged as imported, or a separate aggregate table the statistics routes can read.
Data quality. How should imported days show in the Data-Quality Marker? For example: always Estimate, never trusted for calibration, excluded from the Usual Weekday Baseline until audited. How do they interact with Cycle Verdicts and the calibration audit trail?
Overlap and idempotence. Days already recorded must not be double-counted, and re-running the import must be safe.
Operational. Rate limits and load on the HikCentral server, and where the import runs: a one-off admin command (hikctl) or a background job.

Output

A research note under docs/ answering 1–5 from primary sources, with a recommendation. Then, if feasible, an RFC/ADR proposal and implementation issues. Needs a maintainer decision on questions 2 and 3 before implementation.

Maintainer triage — 2026-09-27

Maintainer accepts research-first scope and will revisit this when in the office; deferred for now. Do not begin importer implementation. Establish available historical source, granularity, identity, retention and overlap from primary V2.6.3 sources before deciding representation, quality/calibration/baseline eligibility or operational import behavior. Implementation remains gated on a later design interview.

Verified read-only research facts — 2026-09-27

The maintainer authorized SSH/OpenAPI inspection and publication of these facts. SSH authenticated as Administrador on the supplied host; Administrator was rejected. Installed registry versions:

HikCentral Professional: 2.6.3.20250828.
HikCentral Professional OpenAPI: 2.6.3.20250826.
Control Client: 2.6.3.0.20250826.

The bundled docs/OpenAPI Developer Guide/Video API.docx section labelled “Statics total number by time” contains the event-subscription description and /api/eventService/v1/eventSubscriptionByEventTypes contract. The generated catalog repeats this mismatch. Therefore the local catalog cannot establish absence of historical passenger-flow support.

Hikvision's official OpenAPI reference lists POST /artemis/api/aiapplication/v1/people/statisticsTotalNumByTime. This confirms a historical candidate in vendor documentation, not its deployed V2.6.3 contract or availability.

Windows denied access during installed-document discovery. No historical endpoint request was sent, no secrets were published, and no server configuration/data was changed. Request/response fields, granularity, retention, timezone/boundaries, paging, licensing and overlap remain unverified.

Retain needs-triage, research-first scope and the maintainer's office deferral. Do not implement an importer; settle representation, quality and operational behavior only after matching primary documentation and read-only probes establish the facts.

Follow-up: deployed historical route confirmed — 2026-09-27

After the maintainer asked whether the route could be checked through SSH, a temporary tunnel to the deployed Artemis listener on port 9016 and the app's existing HMAC signing code were used for read-only queries. The tunnel was closed afterward.

POST /artemis/api/aiapplication/v1/people/statisticsTotalNumByTime returned code: 0, msg: Success on deployed OpenAPI 2.6.3.20250826. A bounded request for one member camera of a people-counting group used pageNo: 1, pageSize: 1, cameraIndexCodes: <one camera ID>, statisticsType: 0 (hour), and startTime/endTime covering 2026-08-31 12:00–13:00 in facility time. The response had completeness: 1 and two aggregate rows with time, cameraIndexCode, enterNum, exitNum. The returned timestamps were exactly 12:00 and 13:00. No camera identity or counts are published here.

This confirms historical hourly people-counting aggregates exist before the app's 2026-09-03 recording start, for at least one camera on 2026-08-31. It does not establish event-level history, all-camera coverage or retention depth. The two returned boundary rows despite requested pageSize: 1 make endpoint inclusivity and pagination behavior explicit research tasks; do not infer a safe importer loop yet. Group-to-camera mapping and deduplication against existing local data remain unresolved.

Hikvision's official people-counting OpenAPI example documents the same request fields and aggregate shape. Continue research-first; retain needs-triage and the maintainer's importer-design deferral.

Confirmed implementation scope — 2026-09-27

The maintainer completed the design interview and explicitly authorized implementation planning, decomposition and publication. This resolution supersedes the earlier research-first deferral and needs-triage wording above. The deployed Artemis 2.6.3 route was verified to return historical camera-hour IN/OUT aggregates on 2026-07-31 and 2026-08-31. Its boundary rows are inclusive at the tested hour; page 1 and 2 repeated rows despite pageSize: 1. Use bounded ranges and source-key deduplication, not page progression as a completion signal. Hikvision's people-counting guide requires the report to be generated first.

End-to-end behavior

One-time admin-triggered retrieval of a selected completed facility-calendar month, for all expected counting cameras. Default to the present count of 12; validate count/identity differences. Fetching stores durable source-attributed camera-hour aggregates and revisions in private staging, never in passage-event rows and never in statistics.
An admin curates the month over multiple sittings in a private web workspace: confirm historical weekly hours and dated exceptions, add holidays/events, review expected-camera coverage, resolve genuine zeros from HikCentral, and mark each open business day Complete, Partial or Missing. An absent report/row is not zero traffic. A fully missing local day may be recovered from valid HikCentral history. A month may be approved with unresolved Partial/Missing days when reasons are recorded and the gaps stay visible.
Approval publishes the reviewed monthly revision, counts and context atomically. Local observations own any camera-hour with local counts; source aggregates may fill wholly unobserved hours. Partly observed hours remain unresolved. Changed upstream counts create a reviewable revision; the current published month remains unchanged until reapproval. Preserve snapshots and actor/time/reason audit history.
Approved history uses a separate long-lived hourly aggregate table; durable local daily/hourly rollups preserve year-over-year totals after raw-event pruning. The deck shows counts and hourly flow with source and coverage markers. Complete imported days can supply count/hourly-flow usual-weekday baselines after independent quality checks and ordinary-day eligibility. Partial days show an Estimate marker. Do not invent peaks, dwell, dayparts or 15/30-minute timing from hourly totals.
Any imported or mixed-source business day stays out of calibration, both immediate nightly multiplier updates and subsequent EWMA/variance/sample history, even when its counts are usable for statistics. Historical corrections update statistics/baselines, preserve prior versions, and do not rewrite applied calibration.

Child issues and order

#162 — retrieve and stage historical HikCentral flow.
#163 — curate monthly historical flow drafts, blocked by #162 and coordinated with #113/#114/#161.
#164 — publish reviewed historical flow without calibrating, blocked by #163.
#165 — present approved historical flow and baselines, blocked by #164.

Related existing work: #113 (historical schedule), #114 (schedule activation), #161 (holiday/event context), #109 (Closed Day calibration), #58 (shared flow-query semantics), #62 (future multi-camera groups), #157 (statistics performance), and #73 (i18n identifiers). #62 remains trigger-based; the others are coordination points or explicit prerequisites as stated on the children. The four child issues are independently scoped and ready-for-agent; dependencies do not waive implementation gates. #160 is their ready-for-agent umbrella. The repository RFC and one draft PR carry the full operation specification.

Draft implementation PR: #166 — reviewed historical passenger-flow backfill. The PR currently contains the agreed RFC and glossary; implementation will follow on the same branch.

**Research question:** can we import historical passenger-flow data from HikCentral to retroactively fill our database, so the statistics deck (and calibration) have history from before this system started recording? ## Why The prod DB only has counted events from **2026-09-03**. Consequences: - no closed month exists yet; - every "vs last year" comparison is hidden; - the Usual Weekday Baseline often reports too few samples; - the Week of 31 Aug shows Mon–Wed as missing. HikCentral has been counting for much longer. ## What we know - The code today only reads live state from Artemis: - `…/aiapplication/v1/people/advance/resourceGroupList` (group totals; `occupancy_service.py:2077`); - `…/people/resourceGroupRealTimeCount`. - Events arrive going forward via the monitor and webhook. - The bundled catalog (`app/docs/artemis_catalog.json`, `app/docs/artemis_openapi.json`) lists `…/people/statisticsHeatMapByTime`. Its description is per-camera heat map by time. Nothing is obviously a historical passenger-flow report. - HikCentral Pro's web client (reachable through our Bumblebee/ISAPI session) has people-counting reports with export. That may be a second source. - Our model stores per-group `IN`/`OUT` events with timestamps (ADR 0005: one camera per group), plus cycle-quality, calibration and audit records. Startup already runs `reconcile_and_quarantine_historical_anomalies_async` over history (see docs/design-history/occupancy-calibration-model-redesign.md, "Startup Backfill Requirement"). ## Questions to answer 1. **Source.** Which HikCentral API (Artemis OpenAPI, or the web/ISAPI report or export endpoints) returns historical people-counting data per resource group or camera? Answer from primary sources (the HikCentral Pro OpenAPI docs for our version, V2.6.3). Capture granularity (per event, per minute, per hour), retention, and paging limits. 2. **Fidelity.** Is the data per event or aggregated? If it's aggregated (e.g. hourly), how do we represent it without faking events? Options include synthetic events at bucket midpoints flagged as imported, or a separate aggregate table the statistics routes can read. 3. **Data quality.** How should imported days show in the Data-Quality Marker? For example: always Estimate, never trusted for calibration, excluded from the Usual Weekday Baseline until audited. How do they interact with Cycle Verdicts and the calibration audit trail? 4. **Overlap and idempotence.** Days already recorded must not be double-counted, and re-running the import must be safe. 5. **Operational.** Rate limits and load on the HikCentral server, and where the import runs: a one-off admin command (`hikctl`) or a background job. ## Output A research note under `docs/` answering 1–5 from primary sources, with a recommendation. Then, if feasible, an RFC/ADR proposal and implementation issues. Needs a maintainer decision on questions 2 and 3 before implementation. ## Maintainer triage — 2026-09-27 Maintainer accepts research-first scope and will revisit this when in the office; deferred for now. Do not begin importer implementation. Establish available historical source, granularity, identity, retention and overlap from primary V2.6.3 sources before deciding representation, quality/calibration/baseline eligibility or operational import behavior. Implementation remains gated on a later design interview. ## Verified read-only research facts — 2026-09-27 The maintainer authorized SSH/OpenAPI inspection and publication of these facts. SSH authenticated as `Administrador` on the supplied host; `Administrator` was rejected. Installed registry versions: - HikCentral Professional: **2.6.3.20250828**. - HikCentral Professional OpenAPI: **2.6.3.20250826**. - Control Client: **2.6.3.0.20250826**. The bundled `docs/OpenAPI Developer Guide/Video API.docx` section labelled “Statics total number by time” contains the event-subscription description and `/api/eventService/v1/eventSubscriptionByEventTypes` contract. The generated catalog repeats this mismatch. Therefore the local catalog cannot establish absence of historical passenger-flow support. Hikvision's [official OpenAPI reference](https://enpinfo.hikvision.com/hkwsen/unzip/20240201150721_74009_doc/) lists `POST /artemis/api/aiapplication/v1/people/statisticsTotalNumByTime`. This confirms a historical candidate in vendor documentation, **not** its deployed V2.6.3 contract or availability. Windows denied access during installed-document discovery. No historical endpoint request was sent, no secrets were published, and no server configuration/data was changed. Request/response fields, granularity, retention, timezone/boundaries, paging, licensing and overlap remain unverified. Retain **needs-triage**, research-first scope and the maintainer's office deferral. Do not implement an importer; settle representation, quality and operational behavior only after matching primary documentation and read-only probes establish the facts. ## Follow-up: deployed historical route confirmed — 2026-09-27 After the maintainer asked whether the route could be checked through SSH, a temporary tunnel to the deployed Artemis listener on port 9016 and the app's existing HMAC signing code were used for read-only queries. The tunnel was closed afterward. `POST /artemis/api/aiapplication/v1/people/statisticsTotalNumByTime` returned `code: 0`, `msg: Success` on deployed OpenAPI **2.6.3.20250826**. A bounded request for one member camera of a people-counting group used `pageNo: 1`, `pageSize: 1`, `cameraIndexCodes: <one camera ID>`, `statisticsType: 0` (hour), and `startTime`/`endTime` covering 2026-08-31 12:00–13:00 in facility time. The response had `completeness: 1` and two aggregate rows with `time`, `cameraIndexCode`, `enterNum`, `exitNum`. The returned timestamps were exactly 12:00 and 13:00. No camera identity or counts are published here. This **confirms historical hourly people-counting aggregates exist before the app's 2026-09-03 recording start**, for at least one camera on 2026-08-31. It does not establish event-level history, all-camera coverage or retention depth. The two returned boundary rows despite requested `pageSize: 1` make endpoint inclusivity and pagination behavior explicit research tasks; do not infer a safe importer loop yet. Group-to-camera mapping and deduplication against existing local data remain unresolved. Hikvision's [official people-counting OpenAPI example](https://www.hikvisioneurope.com/eu/portal/portal/Technology%20Partner%20Program/03-How%20to/How%20to%20get%20people%20counting%20data%20from%20HCP%20via%20OpenAPI.pdf) documents the same request fields and aggregate shape. Continue research-first; retain `needs-triage` and the maintainer's importer-design deferral. ## Confirmed implementation scope — 2026-09-27 The maintainer completed the design interview and explicitly authorized implementation planning, decomposition and publication. **This resolution supersedes the earlier research-first deferral and `needs-triage` wording above.** The deployed Artemis 2.6.3 route was verified to return historical camera-hour IN/OUT aggregates on 2026-07-31 and 2026-08-31. Its boundary rows are inclusive at the tested hour; page 1 and 2 repeated rows despite `pageSize: 1`. Use bounded ranges and source-key deduplication, not page progression as a completion signal. Hikvision's [people-counting guide](https://www.hikvisioneurope.com/eu/portal/portal/Technology%20Partner%20Program/03-How%20to/How%20to%20get%20people%20counting%20data%20from%20HCP%20via%20OpenAPI.pdf) requires the report to be generated first. ### End-to-end behavior - One-time admin-triggered retrieval of a selected **completed facility-calendar month**, for all expected counting cameras. Default to the present count of 12; validate count/identity differences. Fetching stores durable source-attributed camera-hour aggregates and revisions in private staging, never in passage-event rows and never in statistics. - An admin curates the month over multiple sittings in a private web workspace: confirm historical weekly hours and dated exceptions, add holidays/events, review expected-camera coverage, resolve genuine zeros from HikCentral, and mark each open business day Complete, Partial or Missing. An absent report/row is not zero traffic. A fully missing local day may be recovered from valid HikCentral history. A month may be approved with unresolved Partial/Missing days when reasons are recorded and the gaps stay visible. - Approval publishes the reviewed monthly revision, counts and context **atomically**. Local observations own any camera-hour with local counts; source aggregates may fill wholly unobserved hours. Partly observed hours remain unresolved. Changed upstream counts create a reviewable revision; the current published month remains unchanged until reapproval. Preserve snapshots and actor/time/reason audit history. - Approved history uses a separate long-lived hourly aggregate table; durable local daily/hourly rollups preserve year-over-year totals after raw-event pruning. The deck shows counts and hourly flow with source and coverage markers. Complete imported days can supply count/hourly-flow usual-weekday baselines after independent quality checks and ordinary-day eligibility. Partial days show an Estimate marker. Do not invent peaks, dwell, dayparts or 15/30-minute timing from hourly totals. - **Any imported or mixed-source business day stays out of calibration**, both immediate nightly multiplier updates and subsequent EWMA/variance/sample history, even when its counts are usable for statistics. Historical corrections update statistics/baselines, preserve prior versions, and do not rewrite applied calibration. ### Child issues and order 1. [#162 — retrieve and stage historical HikCentral flow](https://git.gaboggamer.online/gabogg/hikcentral/issues/162). 2. [#163 — curate monthly historical flow drafts](https://git.gaboggamer.online/gabogg/hikcentral/issues/163), blocked by #162 and coordinated with #113/#114/#161. 3. [#164 — publish reviewed historical flow without calibrating](https://git.gaboggamer.online/gabogg/hikcentral/issues/164), blocked by #163. 4. [#165 — present approved historical flow and baselines](https://git.gaboggamer.online/gabogg/hikcentral/issues/165), blocked by #164. Related existing work: #113 (historical schedule), #114 (schedule activation), #161 (holiday/event context), #109 (Closed Day calibration), #58 (shared flow-query semantics), #62 (future multi-camera groups), #157 (statistics performance), and #73 (i18n identifiers). #62 remains trigger-based; the others are coordination points or explicit prerequisites as stated on the children. The four child issues are independently scoped and `ready-for-agent`; dependencies do not waive implementation gates. #160 is their ready-for-agent umbrella. The repository RFC and one draft PR carry the full operation specification. Draft implementation PR: [#166 — reviewed historical passenger-flow backfill](https://git.gaboggamer.online/gabogg/hikcentral/pulls/166). The PR currently contains the agreed RFC and glossary; implementation will follow on the same branch.

gabogg added the

needs-triage

label

2026-09-27 15:24:29 +00:00

gabogg referenced this issue

2026-09-27 23:29:26 +00:00

feat(occupancy): retrieve and stage historical HikCentral flow #162

gabogg referenced this issue

2026-09-27 23:29:46 +00:00

feat(occupancy): curate monthly historical flow drafts #163

gabogg referenced this issue

2026-09-27 23:30:05 +00:00

feat(occupancy): publish reviewed historical flow without calibrating #164

gabogg referenced this issue

2026-09-27 23:30:26 +00:00

feat(statistics): present approved historical flow and baselines #165

gabogg added

ready-for-agent

and removed

needs-triage

labels

2026-09-27 23:33:18 +00:00

gabogg changed title from ~~research: import historical passenger flow from HikCentral to backfill the database~~ to feat(occupancy): reviewed monthly historical passenger-flow backfill

2026-09-27 23:33:18 +00:00

~~gabogg referenced this issue 2026-09-27 23:39:54 +00:00~~

feat(occupancy): retrieve and stage historical HikCentral flow #166

~~gabogg referenced this issue 2026-09-28 00:03:59 +00:00~~

feat(occupancy): retrieve and stage historical HikCentral flow #166

gabogg referenced this issue

2026-09-28 09:01:37 +00:00

feat(occupancy): retrieve and stage historical HikCentral flow #166

gabogg referenced this issue

2026-09-28 09:02:10 +00:00

feat(occupancy): retrieve and stage historical HikCentral flow #166

gabogg commented

2026-09-28 09:02:11 +00:00

Author

Owner

Delivery split (2026-09-28): #166 now closes only #162 (retrieval and staging). #163, #164 and #165 continue as stacked draft PRs #170, #171 and #172. This umbrella stays open until #165 lands. Details: #166 (split comment).