# cdr_6.pdhc — User manual cdr_6 is the durable longitudinal store in the synthetic-data pipeline: ``` sim.pdhc → cdr_6.pdhc → analyse.pdhc (future) CREATE STORE READ + analyse cohort data flat-row DB to be built later ``` `sim.pdhc` is the writer. `analyse.pdhc` (when it exists) will be the primary analytical reader. In the meantime the sim.web run viewer, human operators on an SSO login, notebooks, and ad-hoc psql sessions read from the same instance. cdr_6 is the **synthetic sink** — a distinct codebase from cdr1–5; it stores what sim generates and can promote that content into cdr1–5 on request. This manual is the operator entry point. For the architectural reference see [`technical.md`](technical.md); for the consumer-side integration contract see [`dashboard_handoff.md`](dashboard_handoff.md). --- ## 1. Who runs cdr_6 and how cdr_6 lives on miserver, loopback-only. There is **no public URL** — not `cdr_6.pdhc.se`, not anything else. Every interaction goes through: - the operator on miserver directly (`docker-compose`, `psql`), or - SSH port-forward from outside (`sim/tunnel.py` does this for sim; `ssh -L 9055:127.0.0.1:9055 miserver@192.168.1.154` does it by hand). Operator is responsible for: deploy, `.env`, backups, key rotation. Sim authors don't normally touch cdr_6 directly — they use `python -m sim run --target cdr_6 --tunnel` and sim handles the rest. The service does reach *out* to other PDHC services server-side: sso.pdhc (to validate operator bearer tokens on reads), ips.pdhc (for spärr blocks and analysis-consent), and cdr1–5 (for the transfer endpoint). Those calls happen over the miserver loopback / docker bridge — no tunnel needed. --- ## 2. Reaching cdr_6 — three patterns ### A. From miserver (operator only) ```bash ssh miserver@192.168.1.154 cd /usr/local/www/cdr_6.pdhc/current docker-compose ps # see container state docker exec -it cdr_6_app bash psql -h 127.0.0.1 -p 9056 -U cdr_user -d cdr_6_db ``` ### B. SSH port-forward (the everyone-else pattern) From your Mac or any other dev host: ```bash ssh -L 9055:127.0.0.1:9055 -L 9056:127.0.0.1:9056 miserver@192.168.1.154 # leave that terminal open; from another: curl http://127.0.0.1:9055/healthz psql -h 127.0.0.1 -p 9056 -U cdr_user -d cdr_6_db ``` This is what sim does automatically with `--tunnel`. It's what analyse.pdhc will use too, and what a notebook would use. ### C. Direct Postgres reads (analytical) For analyse.pdhc, Jupyter, or any aggregate query: tunnel 9056 and talk SQL. cdr_6's HTTP surface is narrow on purpose — joins, window functions, and cohort-wide aggregations belong in SQL. Note the two tables: `cdr_6_observations` (the data) and `cdr_6_read_audit` (the read audit trail). --- ## 3. The `.env` file (operator) On miserver: `/usr/local/www/cdr_6.pdhc/current/cdr_6_app/.env`, mode 600. | Variable | Purpose | |---|---| | `COMPOSE_PROJECT_NAME` | Pins compose project name (`cdr_6_pdhc`) — load-bearing, don't remove | | `CDR_INSTANCE` | Self-name (`cdr_6`); used by the transfer self-target check | | `APP_PORT` | Host port for gunicorn (default 9055) | | `DB_PORT` | Host port for Postgres (default 9056) | | `DB_VOLUME` | Docker volume name for pgdata (`cdr_6_pgdata`) | | `POSTGRES_USER` | DB role (default `cdr_user`) | | `POSTGRES_PASSWORD` | Set on first deploy; only enforced from outside the container | | `POSTGRES_DB` | DB name (default `cdr_6_db`) | | `SIM_PDHC_SERVICE_KEY` | The write/strict-path token (ingest, DELETE, export, transfer) | | `IPS_BASE_URL` | ips.pdhc base — spärr blocks + analysis-consent (#207/#422) | | `CDR_PDHC_SERVICE_KEY` | cdr_6's **own** key for its outbound ips.pdhc calls | | `SSO_BASE_URL` / `SSO_CLIENT_ID` / `SSO_CLIENT_SECRET` | validate operator bearer tokens on reads (#217) | | `CDRn_BASE_URL` (n=1..5) | destination base URLs for the transfer endpoint | | `CDRn_SERVICE_KEY` (n=1..5) | optional per-target key; defaults to `SIM_PDHC_SERVICE_KEY` | | `CDR_TRANSFER_SOURCE_SERVICE` | outbound source header (default `sim.pdhc`) | | `CDR_TRANSFER_TIMEOUT` | outbound push timeout, seconds (default 30) | Never commit `.env`. Never paste `SIM_PDHC_SERVICE_KEY` (or the SSO secret) into a ticket, log, or message. Memory `infra_ticket_api_respond_auto_closes` applies — be careful with what ends up in any system message. Leaving the `SSO_*` / `IPS_BASE_URL` blocks blank simply disables those integrations: bearer reads won't validate, and the spärr/consent joins short-circuit. Leave the `CDRn_*` block blank until a transfer target is actually wired — an unconfigured target is rejected at call time. --- ## 4. Day-to-day operator commands ### Health probe ```bash curl -s http://127.0.0.1:9055/healthz | jq . # expected: {"status":"ok","service":"cdr_6.pdhc","database":"connected"} ``` 503 with `"database":"unavailable"` → either Postgres is down or the container has stale env (the memory `infra_docker_restart_env` applies). ### See what's in the store ```bash SK=$(grep '^SIM_PDHC_SERVICE_KEY=' /usr/local/www/cdr_6.pdhc/current/cdr_6_app/.env | cut -d= -f2-) curl -s -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ http://127.0.0.1:9055/api/v1/runs | jq . curl -s -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ http://127.0.0.1:9055/api/v1/stats | jq . ``` Each run carries `cohort_name`, `cohort_description`, observation / patient / concept counts, and the effective-time window. That's the shape analyse.pdhc will browse. The read endpoints (`/runs`, `/runs//observations`, `/stats`) also accept an **SSO bearer token** instead of the service key (#217): ```bash curl -s -H "Authorization: Bearer $TOKEN" \ http://127.0.0.1:9055/api/v1/stats | jq . ``` A bearer caller is scoped to their own organisation's rows (or sees everything if they're an SU admin); the service key remains unfiltered. Every read writes a row into `cdr_6_read_audit`. ### Inspect one run's observations ```bash curl -s -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ "http://127.0.0.1:9055/api/v1/runs//observations?limit=50" | jq . ``` This is the sim.web viewer path. It is the one read endpoint that applies **spärr** (ips.pdhc PatientBlock) and **analysis-consent** filtering before returning rows — an operator (bearer) read of a blocked or non-consenting patient will silently drop those rows, and if ips.pdhc can't answer the consent question the request fails **closed** with 503. ### Purge a run ```bash curl -s -X DELETE -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ http://127.0.0.1:9055/api/v1/runs/ | jq . ``` 200 with `{"sim_run_id":"…","deleted":N}` if rows were removed; 404 if no rows matched. There is no soft-delete. ### Rotate the service key ```bash NEW=$(python3 -c 'import secrets; print(secrets.token_urlsafe(32))') ssh miserver@192.168.1.154 cd /usr/local/www/cdr_6.pdhc/current/cdr_6_app sudo sed -i.bak "s|^SIM_PDHC_SERVICE_KEY=.*$|SIM_PDHC_SERVICE_KEY=$NEW|" .env sudo docker-compose up -d --force-recreate app ``` `--force-recreate` is the load-bearing flag — `docker restart` alone keeps the create-time env (memory `infra_docker_restart_env`). After rotation, update every consumer's `.env` to match (sim.pdhc on every Mac that pushes). Because cdr1–5 accept `X-Source-Service: sim.pdhc` with the shared sim key, a rotation may also need propagating to the transfer targets (or set a per-target `CDRn_SERVICE_KEY`). ### Restart cdr_6 ```bash cd /usr/local/www/cdr_6.pdhc/current sudo ./start.sh ``` That's the single entry point per `top_rules.md` rule 8. It only touches ports 9055 and 9056 — sibling services on the shared Colima VM are unaffected. --- ## 5. Backups ### What's running - `~/pg_dump_cdr_6.sh` on miserver runs nightly at 03:00 via the system LaunchDaemon `se.pdhc.pgdump-cdr6`, installed at `/Library/LaunchDaemons/se.pdhc.pgdump-cdr6.plist`. Source-of-truth plist lives in `miserver-ops/launchd/se.pdhc.pgdump-cdr6.plist`. - Migrated from cron in ticket #321 (2026-06-29) because the macOS 26.5 TCC trap made cron-triggered T9 writes EPERM (memory `infra_t9_spindown_trap`). LaunchDaemons run in the system domain with FDA granted directly, avoiding cron's TCC inheritance issue. - Output: `/Volumes/T9/cdr_6_dumps/cdr_6-YYYYMMDD-HHMMSS.pgdump` - Rotation: 30 days - Log: `/Users/miserver/logs/pg_dump_cdr_6.log` - Lock-aware: if `/Volumes/T9/.sim_run_lock` exists, the wrapper skips that night and logs `sim_run_lock present; skipping` (so it doesn't fight a long sim push for T9 throughput). The lock file sits at the T9 mount root, not inside `cdr_6_dumps/`. ### Day-to-day checks ```bash ssh miserver@192.168.1.154 'tail -40 /Users/miserver/logs/pg_dump_cdr_6.log; \ ls -lh /Volumes/T9/cdr_6_dumps/ | tail -5; df -h /Volumes/T9' ``` Look for: - A `done` line per day - New dump file per day, size growing with the DB - T9 free space stable (or shrinking predictably) - No long string of `lock present, skipping` unless you ran sim at 03:00 ### Restoring ```bash # from miserver, with the app shut down: sudo docker-compose stop app sudo docker exec -i cdr_6_db \ pg_restore -U cdr_user -d cdr_6_db --clean --if-exists \ < /Volumes/T9/cdr_6_dumps/cdr_6-YYYYMMDD-HHMMSS.pgdump sudo docker-compose start app ``` Test against `/healthz` and `/api/v1/runs` after. --- ## 6. Writing to cdr_6 (sim only) You don't normally write to cdr_6 by hand. sim.pdhc does it for you: ```bash # on your Mac cd ~/T7_sidewinder/sim.pdhc .venv/bin/python -m sim run \ --profile profiles/smoke_v2.yaml \ --count 100 --seed 42 \ --target cdr_6 --tunnel ``` `SIM_PDHC_SERVICE_KEY` must match the value in cdr_6's `.env`. sim opens the SSH tunnel, pushes flat-row NDJSON in batches of ≤ 1000 (HTTP 413 above that), and prints per-batch results. The cdr_6 side dedups on `(patient_guid, transaction_guid, effective_at)` — re-running the same `(profile, count, seed)` produces 100% duplicates and 0% new rows (deterministic). Two ingest field names are **deprecated aliases**, silently remapped and then left NULL on the stored row (#294/#304): - `activity_guid` → `transaction_guid` - `author_org_guid` → `provider_org_guid` New callers should emit the canonical names (`transaction_guid`, `provider_org_guid`, plus `requesting_org_guid` / `requester_user_guid`). If you really need to write a single row by hand: ```bash curl -X POST http://127.0.0.1:9055/api/v1/ingest \ -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ -H "content-type: application/json" \ -d '{ "concept_guid":"22222222-2222-2222-2222-000000000001", "patient_guid":"33333333-3333-3333-3333-000000000001", "value":"42.0", "sim_run_id":"manual-smoke-1", "effective_at":"2026-05-30T12:00:00Z" }' | jq . # 202 accepted with the new row's guid ``` But that bypasses every sim contract — no canonical provenance, no run manifest. Use it only for plumbing checks. (Note: the dedup key is only computed when `effective_at` is present.) --- ## 7. Reading from cdr_6 (consumers) For details on patterns + auth, see [`dashboard_handoff.md`](dashboard_handoff.md). Short version — a machine reader uses the service key: ```python import os, requests HEADERS = {"X-Source-Service": "sim.pdhc", "X-Service-Key": os.environ["SIM_PDHC_SERVICE_KEY"]} runs = requests.get("http://127.0.0.1:9055/api/v1/runs", headers=HEADERS).json() for r in runs: print(r["sim_run_id"], r["cohort_name"], r["observations"]) ``` A human operator instead sends `Authorization: Bearer ` and sees only their organisation's rows (#217). For analytical queries (joins across runs, window functions, percentiles) go straight to Postgres on 9056. The data table is `cdr_6_observations`; key columns for analytics are `sim_run_id`, `patient_guid`, `concept_guid`, `effective_at`, `value`. ```sql -- mean value per concept, per run SELECT sim_run_id, concept_guid, AVG(value::numeric) FROM cdr_6_observations WHERE effective_at >= '2026-01-01' GROUP BY sim_run_id, concept_guid; ``` When analyse.pdhc is built, it will own these queries; until then, notebooks are the supported substitute. --- ## 8. Moving content into cdr1–5 (export + transfer) cdr_6 can promote its own synthetic content into a working CDR. Two service-key-only surfaces support this: - **`GET /api/v1/export`** (#399) — a complete, cursor-paged (`after_id` / `limit` ≤ 5000 / `sim_run_id`), **deliberately unfiltered** raw dump (no spärr, no consent, no org scope). It is the faithful read source for a CDR-to-CDR transfer. - **`POST /api/v1/transfer`** (#529/#530) — the server-side self-promote. cdr_6 reads its own rows and pushes them into a destination's `/ingest/batch`. Because the source is always cdr_6 itself, no real patient data can be moved. ```bash # dry-run first: count what would move curl -s -X POST -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ -H "content-type: application/json" \ -d '{"to":"cdr1","dry_run":true}' \ http://127.0.0.1:9055/api/v1/transfer | jq . # then transfer (copy). Add "purge_source":true to turn copy into move. curl -s -X POST -H "X-Source-Service: sim.pdhc" -H "X-Service-Key: $SK" \ -H "content-type: application/json" \ -d '{"to":"cdr1","sim_run_id":"","batch_size":100}' \ http://127.0.0.1:9055/api/v1/transfer | jq . ``` `purge_source` deletes the transferred run(s) from cdr_6 **only after a verified-complete copy** (every batch 2xx, nothing rejected, all rows accounted for) — a partial or failed transfer never loses data. `batch_size` is clamped to ≤100. On transfer, cdr_6 **synthesises a FHIR R5 Observation** per row (#531) so the destination materialises a queryable observation, not just a staging row. `plan.pdhc`'s trigger UI (#530) calls this same endpoint. --- ## 9. Troubleshooting | Symptom | Likely cause | Fix | |---|---|---| | `services.html` shows cdr_6 green but `/healthz` returns 503 | services.html uses `mode: 'no-cors'` and always shows green when nginx is up — don't trust it | curl `/healthz` directly | | `403 missing X-Source-Service` | Both auth headers absent on a strict endpoint | Set `X-Source-Service: sim.pdhc` AND `X-Service-Key: ` | | `403 unknown source service: ''` | Wrong `X-Source-Service` | Must be exactly `sim.pdhc` | | `401 missing X-Service-Key` | Source ok but no key | Add the `X-Service-Key` header | | `403 invalid service key` | Key doesn't match `.env` | Check the key / rotation drift | | `401 invalid bearer token` | SSO token rejected or `SSO_*` unset | Verify the token, and that `SSO_BASE_URL/CLIENT_ID/CLIENT_SECRET` are configured | | `503` on `/runs//observations` for an operator | analysis-consent fails closed — ips.pdhc unreachable | Check `IPS_BASE_URL` + ips.pdhc health | | App container `(unhealthy)`, db `(healthy)` | Postgres credential drift (CLAUDE.md §9) | `docker exec cdr_6_db psql -U cdr_user -d cdr_6_db -c "ALTER USER cdr_user WITH PASSWORD '';"` then restart app | | `Connection refused` | Forgot `--tunnel` (sim) or no SSH forward open (curl) | Open the tunnel | | `duplicate` on every row of a batch | Same `(profile, count, seed)` already ingested | Pick a new seed or purge the prior run | | transfer returns `verified:false`, nothing purged | A destination batch failed (see `http_errors`) | Fix the target (base URL / key / health); source is left intact | | Backups don't appear on T9 | Disk-full, T9 not mounted, stuck `.sim_run_lock`, or the LaunchDaemon lacks Full Disk Access | Check `df -h /Volumes/T9`, `ls /Volumes/T9/.sim_run_lock`, `tail` the log, re-grant FDA (add `/bin/bash`), then `sudo launchctl kickstart -k system/se.pdhc.pgdump-cdr6` | | `colima ssh -- docker info` works but host `docker ps` doesn't | Colima docker-socket forward died silently (memory `infra_colima_default_socket_forward_dies`) | Restart Colima | --- ## 10. What cdr_6 does and doesn't do **Does** (contrary to older docs): - Accepts SSO bearer tokens on reads, org-scoped per the SSO blob (#217). - Calls ips.pdhc for spärr blocks and analysis-consent on `/runs//observations` (#207/#422), failing closed on consent. - Synthesises FHIR R5 Observations when transferring into cdr1–5 (#531). - Keeps a read-audit trail in `cdr_6_read_audit`. **Doesn't:** - Validate FHIR on ingest — sim does that upstream; the write side trusts sim's emissions verbatim. - Authenticate any *write/strict-path* source other than `sim.pdhc`. - Expose itself publicly. No DNS, no nginx, no SSL. - Soft-delete. `DELETE /runs/` (and verified `/transfer purge`) are permanent. - Render dashboards. Consumers (sim.web, analyse.pdhc, notebooks) render. --- ## 11. Where to look next - [`technical.md`](technical.md) — architecture, schema, endpoints, auth - [`dashboard_handoff.md`](dashboard_handoff.md) — consumer-side integration contract - `progress.md` — current deployment state - `sim.pdhc/docs/phase_3_operator_runbook.md` — T9 + miserver prep - `sim.pdhc/docs/user_manual.md` — the writer's manual