118 lines
14 KiB
Markdown
118 lines
14 KiB
Markdown
|
# openrun — roadmap
|
|||
|
|
|
|||
|
|
Living backlog. Items are grouped by phase; within a phase, top to bottom is rough priority. Each item names its **test plan** alongside the implementation so they ship together.
|
|||
|
|
|
|||
|
|
This isn't a contract. If something turns out to be harder, smaller, or pointless once we're in the code, edit it here rather than rationalise away.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Phase 0 — test scaffolding (prerequisite, one-time)
|
|||
|
|
|
|||
|
|
The codebase has no `tests/` directory and no pytest in `pyproject.toml`. Everything below assumes this is in place first.
|
|||
|
|
|
|||
|
|
- **Add pytest + a minimal `tests/` layout.** Add `pytest` and `pytest-cov` to `[dependency-groups.dev]` in [pyproject.toml](pyproject.toml). Create `tests/{unit,integration,fixtures}/`. A `tests/conftest.py` should expose a shared `tmp_conn` fixture that builds an in-memory SQLite via `openrun.db.connect(":memory:")` (so the live `data/garmin.db` is never touched).
|
|||
|
|
- **Test plan:** N/A — this *is* the test plan.
|
|||
|
|
- **DoD:** `uv run pytest` exits 0 with one trivial passing test.
|
|||
|
|
|
|||
|
|
- **Pin a tiny fixture set.** One `.fit` (~5 MB workout), one Takeout `summarizedActivities` snippet (~3 activities), one Connect-format snippet, one of each daily wellness JSON. Anonymise; commit under `tests/fixtures/`. These back every test below.
|
|||
|
|
- **DoD:** Each fixture has a one-line `tests/fixtures/README.md` entry recording its provenance + what it exercises.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Phase 1 — code gaps (existing functionality)
|
|||
|
|
|
|||
|
|
### 1.1 Takeout export doesn't ingest splits — RETIRED (premise wrong for current account)
|
|||
|
|
~~[ingest/garmin_export.py](src/openrun/ingest/garmin_export.py) only handles `summarizedActivities`. Lap detail in Takeout lives in `<activity_id>_<name>.json` under `DI-Connect-Fitness/`~~
|
|||
|
|
|
|||
|
|
**Finding (2026-05-18):** the actual Takeout dump for this account contains zero per-activity lap-detail JSONs under `DI-Connect-Fitness/` — only `summarizedActivities`, `personalRecord`, `trainingPlan`, `workout`, `gear`. Lap data is only retrievable from the FIT files via `link_fit_files.py` (Path B sync also pulls it directly from `/activity-service/activity/{aid}/splits`). If a future Takeout shape *does* include `<activity_id>.json` lap files, reopen this item; until then the assumption that drove it doesn't hold.
|
|||
|
|
|
|||
|
|
### 1.2 Unhandled Takeout JSON categories
|
|||
|
|
[project_takeout_export_quirks.md](../.claude/projects/-Users-noise-Documents-obsidian-RunningLog-garmin/memory/project_takeout_export_quirks.md) lists files we currently `unrecognized`: `HydrationLog`, `TrainingReadinessDTO`, `EnduranceScore`, `HillScore`, `RunRacePredictions`. Each is a new SQLite table + dispatch entry. Skip `HydrationLog` (low value), prioritise `TrainingReadinessDTO` (Garmin's own readiness score — useful for comparing against derived TSB) and `RunRacePredictions` (a free baseline for the projected-PMC plan).
|
|||
|
|
- **Test plan:** per category, `test_garmin_export.py::test_handle_<category>` fixture-driven insert; one schema-roundtrip test (`SELECT *` returns the same values).
|
|||
|
|
|
|||
|
|
### 1.3 fit_linker can't disambiguate near-simultaneous activities — DONE
|
|||
|
|
Implemented warn-and-skip on collision: `link()` tracks `linked_aids: dict[int, Path]`; the second FIT to match an already-linked activity is logged on stderr and skipped (count surfaced in summary). Pure `_match_activity` helper extracted for unit-testable tolerance/closest-pick behaviour, and `link()` accepts a `fit_iter=` injection point so tests avoid the FIT-parsing path. See [tests/unit/test_fit_linker.py](tests/unit/test_fit_linker.py).
|
|||
|
|
|
|||
|
|
### 1.4 `handle_fit` silently drops FITs with no parseable activity-id chunk — DONE
|
|||
|
|
Main loop now counts FITs whose stem has no 8+-digit chunk and prints a one-line `n FITs skipped … run openrun-link-fit <root> for Takeout-style exports` hint. The activity-id extraction is now `_activity_id_from_filename`, with five unit tests covering classic/Takeout/year-prefix/empty cases, plus a subprocess end-to-end test of `main()`.
|
|||
|
|
|
|||
|
|
### 1.5 `_resolve_fit_path` is fragile (decided: absolute paths) — DONE
|
|||
|
|
Switched to absolute paths everywhere. `fit_linker.record_link` is a small shared helper used by both ingest paths and stores `str(p.resolve())`. `relink(conn, new_root)` walks a moved export by basename and rewrites the table; CLI flag is `openrun-link-fit <new_root> --relink`. `_resolve_fit_path` collapsed to a one-line existence check that raises with the relink hint on miss. The dead `fit_search_paths` config field (and matching `[ingest]` block in `openrun.toml`) was removed. Live DB migrated (349 paths rewritten, 0 unmatched).
|
|||
|
|
|
|||
|
|
### 1.6 Schema round-trip tests — DONE (wellness + activities; splits/fit_files/tiz open)
|
|||
|
|
[tests/integration/test_loaders.py](tests/integration/test_loaders.py) covers 8 ingest-handler → DB → loader round-trips:
|
|||
|
|
- `activities` (Takeout scaled-int units → `load_activities` derived columns)
|
|||
|
|
- 7 wellness tables (steps / sleep / stress / hrv / resting_hr / intensity_minutes / body_battery), each through `load_wellness`. Sleep additionally rides through `load_sleep_stages` to lock the deep+light+rem=1.0 invariant.
|
|||
|
|
**Caught a real bug in the test fixture:** my first draft used `averageSpeed: 27.78` thinking that was the SI value. Inspecting live Takeout data showed Garmin actually stores it as `m/s × 0.1` (e.g. a 3.247 m/s sprint stored as 0.3247), so the `× 10` conversion in the handler is correct — the README table can stay; the fixture was wrong.
|
|||
|
|
**Still open:** `activity_splits`, `activity_fit_files`, `activity_time_in_zone` — these aren't reached through a single Takeout JSON handler (splits are sync-only, FIT files are linker-driven, TIZ is precomputed), so each needs a different fixture/path. Pulled forward to a follow-up; the existing helpers' tests (test_fit_linker.py, test_weekly_tiz.py) cover most of what these round-trips would.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Phase 2 — analytical features hinted at in the README
|
|||
|
|
|
|||
|
|
### 2.1 DBSCAN route clustering at scale
|
|||
|
|
[README §3](README.md) explicitly: *"Adequate for a few hundred starts; switch to `sklearn.cluster.DBSCAN(metric='haversine')` for thousands."* Add `cluster_routes(..., method='dbscan')` alternate path. Don't replace the greedy version — it's a good correctness baseline.
|
|||
|
|
- **Test plan:**
|
|||
|
|
- `test_geo.py::test_haversine_known_pairs` — two cities, compare to a published value within 0.1 km.
|
|||
|
|
- `test_geo.py::test_cluster_routes_greedy_vs_dbscan_agree` — on a synthetic 200-point dataset (3 well-separated clusters + noise), both methods produce the same cluster assignment up to label permutation.
|
|||
|
|
|
|||
|
|
### 2.2 Off-watch volume integration
|
|||
|
|
[feedback_db_not_full_picture.md](../.claude/projects/-Users-noise-Documents-obsidian-RunningLog-garmin/memory/feedback_db_not_full_picture.md): recorded activity ≠ full training. Add a `manual_activities` table + a CSV/TOML loader so the user can log strength, hikes, unrecorded runs without faking Garmin uploads. `daily_training_load_series` learns an optional `include_manual=True`. Existing analyses pick this up via the same Banister pipeline.
|
|||
|
|
- **Test plan:**
|
|||
|
|
- `test_manual_activities.py::test_load_from_csv` — schema check.
|
|||
|
|
- `test_loaders.py::test_daily_training_load_series_with_manual` — assert `include_manual=True` increases the total when synthetic manual rows are present, and that the Banister output is monotonically affected.
|
|||
|
|
|
|||
|
|
### 2.3 PR detection by distance bucket — DONE
|
|||
|
|
`personal_records(activities, distance_bins_km=..., tolerance=0.05)` in [model.py](src/openrun/model.py); tested in [test_race_plan.py](tests/unit/test_race_plan.py) — fastest-in-band selection, tolerance enforcement, NaN-pace masking, empty input. On live data: PR table comes out 5K @ 4:49, 10K @ 5:38, half @ 6:18, 50 @ 8:04 (no marathon bin populated, correctly skipped).
|
|||
|
|
|
|||
|
|
### 2.4 Race-plan TL/km calibration from history — DONE
|
|||
|
|
`calibrate_tl_per_km(conn, *, activity_types=..., min_distance_km=2.0, lookback_days=365)` returns `{median, q1, q3, n}`. Tests cover the median/IQR math, the short-distance and other-activity-type exclusions, and the empty-DB → NaN path. On live data the median came in at 11.4 with IQR [9.2, 16.4] — confirms the README's "training runs are ~11" claim and shows the spread is wide enough that hardcoded constants are a real weakness.
|
|||
|
|
|
|||
|
|
### 2.5 Banister projection helper — DONE
|
|||
|
|
`banister_forecast(history, future, *, today=None)` lives in [model.py](src/openrun/model.py); the notebook's splice (`hist[hist.index <= TODAY]` ⨁ `forecast[forecast.index > TODAY]`) is now in one place. Tests in [test_banister.py](tests/unit/test_banister.py):
|
|||
|
|
- `test_banister_forecast_matches_full_history` — splice invariant: PMC frame equals `banister(full)` when full is split (history, future).
|
|||
|
|
- Plus a closed-form impulse-response check on the EWMA itself: with load=L on day 0 and zeros after, `CTL[i] = L·(1-decay)·decay^i` (the ROADMAP's original "CTL at day 42 ≈ L·(1-1/e)" was wrong — that's the step response after τ days, not impulse — corrected in the test).
|
|||
|
|
- `test_banister_steady_state_approaches_input` covers the step response (constant input L converges to L; at i=τ it's L·(1-1/e)).
|
|||
|
|
- Boundary-day overlap, default-today, and tail-extension tests round it out.
|
|||
|
|
|
|||
|
|
### 2.6 Sleep-stage breakdown surfacing — DONE
|
|||
|
|
`load_sleep_stages(conn)` in [model.py](src/openrun/model.py) returns deep/light/rem/awake (seconds + percentages) keyed by date. Percentages of present stages sum to 1.0 by construction; awake_pct uses total-time-in-bed (not asleep-only). Tests in [test_sleep_stages.py](tests/unit/test_sleep_stages.py) cover the invariant + the zero-sleep / NaN-stage edge cases (the latter exposed by real 2026-05-17 row where `rem_s` is NULL).
|
|||
|
|
|
|||
|
|
### 2.7 Per-second decoupling: drop-in plotting helper — DONE
|
|||
|
|
New `openrun.plots` submodule (matplotlib imported lazily inside the function so the rest of `openrun` stays import-light). `plot_fit_decoupling(records, *, segments=2, ax=None)` draws a per-segment bar chart with Friel's 5% / 10% reference lines. Tests in [test_plots.py](tests/unit/test_plots.py) cover bar count, caller-supplied Axes, and the "no usable records" sentinel path.
|
|||
|
|
|
|||
|
|
### 2.8 Time-in-zone weekly summary — DONE
|
|||
|
|
`weekly_time_in_zone(conn, *, start=None, end=None, activity_types=...)` in [model.py](src/openrun/model.py) pivots the cached `activity_time_in_zone` table by ISO week (Monday-anchored). Tests in [test_weekly_tiz.py](tests/unit/test_weekly_tiz.py): within-week summation, week splitting, activity-type filter, date window, empty-frame shape. On live data, recent weeks show a Z3-heavy profile that matches the README's "junk-miles middle" warning.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Phase 3 — aspirational (not on the critical path)
|
|||
|
|
|
|||
|
|
These are README-shaped or memory-shaped *maybes*. Don't take them on without a concrete reason; listed so they stop showing up as recurring "we should…" thoughts.
|
|||
|
|
|
|||
|
|
- **Heat & humidity adjustment of decoupling.** FIT messages carry `temperature` and sometimes `developer_data` from external sensors. We don't capture it. A weather-adjusted Pa:Hr would explain summer-vs-winter decoupling gaps. Real work: capture temp in `load_fit_records`, add a `decoupling_adjusted(...)` that conditions on it. Test plan: regression with vs without adjustment on a hot-run fixture should differ only when `temperature` is present.
|
|||
|
|
- **Multi-athlete support.** The config already accepts a per-user profile. Promoting it to `openrun.toml` having multiple `[athletes.<name>]` blocks + a `--athlete` CLI flag is straightforward but premature for a single-user dataset. Test plan: profile-resolution unit tests would multiply by N.
|
|||
|
|
- **Power & vertical-oscillation regression.** Lap raw has `averagePower`, `verticalOscillation`, `verticalRatio`. No analysis surfaces them. Would slot into [04_efficiency.ipynb](examples/notebooks/04_efficiency.ipynb). Worth a notebook section before becoming a `model.py` helper.
|
|||
|
|
- **Race-day projected vs actual diff tracker.** After a race, compare the [06_race_plan.ipynb](examples/notebooks/06_race_plan.ipynb) projection to what actually happened (CTL on race day, predicted vs realised pace from `RunRacePredictions`). One-shot retrospective; doesn't need to be reusable.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Test conventions
|
|||
|
|
|
|||
|
|
- **Layout:**
|
|||
|
|
- `tests/unit/` — pure-function metrics ([model.py](src/openrun/model.py) public surface). One file per concept: `test_banister.py`, `test_decoupling.py`, `test_geo.py`, `test_zones.py`.
|
|||
|
|
- `tests/integration/` — ingest pipelines + loaders. Use the in-memory SQLite fixture.
|
|||
|
|
- `tests/fixtures/` — pinned JSON / FIT / CSV samples. Anonymised.
|
|||
|
|
- **Granularity:** prefer one assertion per test (within reason). Friel's `< 5%` / `5–10%` thresholds make for natural assertions; the round-trip schema tests should be exact-equality on inserted rows.
|
|||
|
|
- **What we don't test:**
|
|||
|
|
- `garth.connectapi` — network-dependent, fragile, and garth is upstream-deprecated. Mock at the `_safe_call` boundary if we ever need to.
|
|||
|
|
- Notebook content. The build scripts ([examples/notebooks/_build_05.py](examples/notebooks/_build_05.py) etc.) are easier to read than the generated `.ipynb`; we test what they import from, not the notebook itself.
|
|||
|
|
- Plotting beyond smoke tests (artist counts, axis labels). Visual regression isn't worth the maintenance.
|
|||
|
|
- **Run:** `uv run pytest` (no markers needed at this scale). Add `pytest-cov` for an off-the-shelf coverage report; ignore the percentage as a target until we have meaningful coverage of the pure-function surface.
|
|||
|
|
|
|||
|
|
---
|
|||
|
|
|
|||
|
|
## Working agreement
|
|||
|
|
|
|||
|
|
When picking up an item: write the failing test first against the API the test plan describes, then make it pass. If the test plan turns out to be wrong (the function shouldn't behave that way after all), update this file in the same PR. Items removed because they turned out to be misguided are more valuable signal than items completed.
|