Technical manual
cdr_6 is the durable longitudinal store in the synthetic-data
pipeline:
sim.pdhc → cdr_6.pdhc → analyse.pdhc (future)
CREATE STORE READ + analyse
longitudinal flat-row cohort analysis
cohort data two-table service to be built
from plandef schema later
Terms
sim.pdhc generates longitudinal time-series for each cohort patient
based on plan.pdhc PlanDefinitions and their leaf Terms (concepts).
cdr_6 is the durable store. The primary planned consumer is the
future analyse.pdhc service. Existing readers — the sim.web run
viewer, human operators on an SSO bearer token, notebooks, and direct
psql — work against the same instance.
Flat-row ingest (sim trusts its own emissions), but the read and
transfer surfaces are not unauthenticated or isolated any more:
reads accept SSO bearer tokens with per-org scoping (#217/#412), the
/runs/<id>/observations path enforces spärr (#207) and analysis
consent (#422), and /transfer synthesises FHIR R5 to promote content
into cdr1–5 (#529/#530/#531). Built for ticket
#178 Phase 3 (rescoped
2026-05-29) and extended since.
This document is the architectural overview for someone inheriting
the service. It is generated to match the code under cdr_6_app/ — if
it disagrees with the code, trust the code and fix this file.
Is:
- A Flask app + Postgres pair running in Docker on miserver, both
bound to 127.0.0.1.
- The durable longitudinal store between sim.pdhc (the producer) and
analyse.pdhc (the future analytical consumer). Multiple sim runs
coexist in one DB, keyed by sim_run_id.
- The synthetic sink — a distinct codebase from cdr1–5. It stores
a flat schema and trusts sim's emissions verbatim on the write side
(no FHIR validation, no xlate, no plan.pdhc canonicalisation on
ingest).
- Integrated with the rest of the platform on the read/transfer side:
it validates SSO bearer tokens against sso.pdhc, calls ips.pdhc for
spärr blocks and analysis-consent verdicts, and synthesises FHIR R5
Observations when promoting rows into cdr1–5.
Isn't:
- A general-purpose CDR. The only valid X-Source-Service for the
write/strict paths is sim.pdhc (top_rules.md rule 2). Other
producers get their own sibling instance.
- Publicly reachable. No DNS, no nginx vhost. The only way in from
outside miserver is an SSH port-forward (sim/tunnel.py).
- A FHIR server for ingest. The stored schema is flat; FHIR bundles are
not the ingest wire format (cdr1–5 cover that case). cdr_6 does
synthesise FHIR on the way out during /transfer (§8).
cdr_6.pdhc/
├── cdr_6_app/ # everything the container needs
│ ├── app/
│ │ ├── __init__.py # create_app() — Flask factory + config wiring
│ │ ├── extensions.py # SQLAlchemy + Migrate handles
│ │ ├── api/
│ │ │ ├── auth.py # require_sim_service_key + require_read_auth
│ │ │ ├── health.py # GET /healthz (no auth)
│ │ │ ├── ingest.py # POST /api/v1/ingest[/batch]
│ │ │ ├── stats.py # GET /api/v1/stats
│ │ │ ├── runs.py # /runs, /runs/<id>/observations, /export, DELETE
│ │ │ └── transfer.py # POST /api/v1/transfer (self-promote)
│ │ ├── models/
│ │ │ ├── observation.py # cdr_6_observations
│ │ │ └── read_audit.py # cdr_6_read_audit
│ │ └── services/
│ │ ├── sso_client.py # bearer-token validation (#217)
│ │ ├── ips_client.py # spärr / PatientBlock fetch (#207)
│ │ ├── analysis_consent.py # analysis-consent filter (#422)
│ │ ├── audit.py # record_read_audit()
│ │ └── transfer.py # run_transfer() + FHIR synth (#529/#531)
│ ├── migrations/ # Alembic; head = 0005_read_audit_reform_tuple
│ ├── Dockerfile
│ ├── docker-compose.yml # db (postgres:16) + app (gunicorn)
│ ├── requirements.txt
│ ├── .env.example
│ └── wsgi.py
├── docs/
│ ├── dashboard_handoff.md # consumer contract (cross-repo)
│ ├── technical.md # ← this file
│ └── user_manual.md
├── start.sh # compose up + flask db upgrade + /healthz probe
├── top_rules.md
├── readme.md
├── progress.md
└── changed_files.md
| Port | Service | Binds to | Reachable from |
|---|---|---|---|
| 9055 | gunicorn (Flask app) | 127.0.0.1 |
miserver loopback only |
| 9056 | postgres:16 | 127.0.0.1 |
miserver loopback only |
| 9057–9 | reserved | — | — |
| 9046 | (internal) gunicorn-in-container | container network | docker bridge only |
| 5432 | (internal) postgres-in-container | container network | docker bridge only |
The compose file maps host 127.0.0.1:9055 → container :9046 for the
app, and 127.0.0.1:9056 → container :5432 for Postgres. Container
names are cdr_6_app and cdr_6_db.
There is no nginx vhost. There is no public DNS. There is no *.pdhc.se
subdomain. The only way in from outside miserver is an SSH tunnel:
ssh -L 9055:127.0.0.1:9055 miserver@192.168.1.154
ssh -L 9056:127.0.0.1:9056 miserver@192.168.1.154 # for direct psql
sim/tunnel.py does this automatically for sim's --target cdr_6 --tunnel.
For the outbound /transfer path, cdr_6 reaches cdr1–5 (and ips.pdhc /
sso.pdhc) over the miserver loopback / docker bridge — no tunnel; those
are server-side calls made by the running container.
Two decorators, both in cdr_6_app/app/api/auth.py:
require_sim_service_key — strict, write/transfer pathsUsed by ingest, DELETE /runs/<id>, /export, and /transfer.
Requires:
X-Source-Service: sim.pdhc
X-Service-Key: <value of SIM_PDHC_SERVICE_KEY>
Error codes are not all 401 — they distinguish the failure:
| Condition | HTTP | Body |
|---|---|---|
No X-Source-Service and no X-Service-Key |
403 | missing X-Source-Service |
X-Source-Service ≠ sim.pdhc |
403 | unknown source service: '<value>' |
Source ok but X-Service-Key missing |
401 | missing X-Service-Key |
| Key present but wrong (or none configured) | 403 | invalid service key |
Service-key callers are treated as is_admin and unfiltered —
g.user_org_guids = None signals "no row filter".
require_read_auth — service-key OR SSO bearer, read paths (#217)Used by /runs, /runs/<id>/observations, and /stats. It tries the
service-key first (only when both service headers are present); if the
caller instead sends Authorization: Bearer <token>, the token is
validated against sso.pdhc.
services/sso_client.py): callsGET {SSO_BASE_URL}/api/auth/me/service with cdr_6's ownX-SSO-Client-Id / X-SSO-Client-Secret (registered in sso.pdhc ascdr_6 consumer) plus the caller's bearer. No caching — everyNone → 401 invalid bearer
token.affiliations[].care_unit_guid (Zone 1),organization_ids for pre-reformauthor_org_guid is in_apply_org_scope / _scope_clause). A bearer caller within_([]) → 1=0). is_su_adminrole_guid (role of the activepurpose (constant "research" here), andaccess_basis (su_admin / same_unit) are set on g for the auditmissing auth: send X-Service-Key or
Authorization: Bearer.Scope caveat worth knowing: org filtering matches on
author_org_guid, which is the deprecated column left NULL on new
rows (§5.3). Bearer non-admin scoping therefore only matches
historical rows that still carryauthor_org_guid; SU-admin and
service-key reads are the practical read paths for current data.
All /api/v1/* paths are registered under that prefix; /healthz is at
the root.
| Method | Path | Auth | Purpose |
|---|---|---|---|
| GET | /healthz |
none | {status, service, database} — 200 ok, 503 degraded |
| POST | /api/v1/ingest |
service-key | Insert one observation row |
| POST | /api/v1/ingest/batch |
service-key | Insert ≤ 1000 rows; per-item status |
| GET | /api/v1/stats[?run_id] |
read-auth | Counts of observations / patients / concepts |
| GET | /api/v1/runs |
read-auth | Per-run summary (counts, period, cohort metadata) |
| GET | /api/v1/runs/<id>/observations |
read-auth | Capped, ordered sample of a run's rows (spärr + consent filtered) |
| GET | /api/v1/export |
service-key | Unfiltered, cursor-paged raw export (CDR-to-CDR source) |
| POST | /api/v1/transfer |
service-key | Self-promote rows into cdr1–5 (+FHIR synth) |
| DELETE | /api/v1/runs/<id> |
service-key | Purge every row with sim_run_id == id |
/healthz matches CLAUDE.md §10:
{"status":"ok|degraded","service":"cdr_6.pdhc","database":"connected|unavailable"}
200 for ok, 503 for degraded. The DB probe is SELECT 1. The
status page at services.html doesn't render this correctly (no-cors
opaque response) — don't trust it; curl /healthz directly.
{
// Required
"concept_guid": "<uuid>",
"patient_guid": "<uuid>",
"value": "<any> — coerced to string in storage",
// Canonical clinical-context (all optional)
"service_request_guid": "<uuid>",
"transaction_guid": "<uuid>",
"provider_org_guid": "<uuid>",
"contract_guid": "<uuid>",
"requesting_org_guid": "<uuid>", // canonical (added #294/#304 phase 4)
"requester_user_guid": "<uuid>", // canonical (added #294/#304 phase 4)
"plan_definition_guid": "<uuid>",
"care_plan_guid": "<uuid>",
// Sim extras / cohort provenance
"grant_token": "<token>",
"sim_run_id": "<sim run id>",
"cohort_name": "<from profile.name>",
"cohort_description": "<from profile.description>",
"effective_at": "<ISO 8601>",
"response_type": "<string>",
"fhir_observation_json":{ },
"payload_hash": "<sha256>",
"is_late": false,
// Legacy aliases — accepted, silently remapped, then dropped (§5.3)
"activity_guid": "<uuid>", // → transaction_guid
"author_org_guid": "<uuid>" // → provider_org_guid
}
Batch endpoint accepts either a bare array or {"items": [...]}. Hard
cap is 1000 (HTTP 413 above that). The batch response is
{total, accepted, duplicate, rejected, entries[]} with a per-item
{index, status, guid|error}.
dedup_key = sha256(patient_guid | transaction_guid | effective_at),
computed in _build_row only when effective_at is present. A
DB-level collision (unique constraint) rolls the single row back and
returns {"status": "duplicate"} (HTTP 200); in a batch the item is
counted under duplicate. Sim is responsible for making rows unique
enough on its side; this is a backstop, not the primary contract.
Two legacy aliases are accepted at the API boundary and silently
remapped onto canonical columns, with a deprecation warning logged:
| Legacy alias | Canonical column | Behaviour |
|---|---|---|
activity_guid |
transaction_guid |
#294 RFC G2 — canonical wins if both sent |
author_org_guid |
provider_org_guid |
#294/#304 phase 4 — canonical wins if both sent |
The legacy columns (activity_guid, author_org_guid) still exist for
back-compat reads on historical rows, but new rows leave them NULL —
ingest never populates them. to_dict() still emits both alias keys
during the deprecation window.
/runs/<id>/observations — filtered readReturns a deterministically-ordered (patient_guid, concept_guid,
effective_at) capped sample of one run. Optional ?concept_guid=,
?patient_guid=, ?limit= (default 200, hard max 5000). This is the
one read path where the spärr and analysis-consent filters fire
(§6), after org scoping. Used by the sim.web viewer.
/export — unfiltered raw export (#399)Service-key only. Cursor-paginated by integer PK for a stable,
exhaustive sweep:
?after_id=<int> — rows with id > after_id (default 0)?limit=<n> — page size (default 1000, hard max 5000)?sim_run_id=<id> — restrict to one run (default: all rows)Response: {row_count, limit, sim_run_id, next_after_id, rows[...]}.
next_after_id is the last row's id when the page was full, or null
when exhausted; the caller pages until it is null.
Deliberately unfiltered — no spärr, no consent, no org scope — so a
CDR-to-CDR transfer moves the whole content, not a filtered sample.
This is the read source behind the sim cdr-transfer operator path and
a faithful mirror of /transfer's in-process reader.
/transfer — self-promote into cdr1–5 (#529/#530/#531)Service-key only. cdr_6 reads its own rows and pushes them to a
destination CDR's /api/v1/ingest/batch. The source is always cdr_6
itself — the endpoint takes no external source, so no real
spärr/consent-protected patient data can be moved. See §8.
Body (JSON object):
| Field | Req? | Meaning |
|---|---|---|
to |
yes | destination CDR name (cdr1..cdr5) |
sim_run_id |
no | one run; default = all rows in the sink |
dry_run |
no | count what would move, no writes |
purge_source |
no | after a verified-complete copy, delete the run(s) from cdr_6 (copy → move) |
batch_size |
no | push chunk size, clamped to 1..100 (default 100) |
Returns the run_transfer() summary. Caller errors (unknown/
unconfigured target, target == self) → 4xx TransferError; push/HTTP
failures are reported inside the summary (http_errors), never raised.
GET /api/v1/runs/<id>/observations applies three filters in order:
_apply_org_scope) — bearer non-admin callers seeauthor_org_guid is in their blob org set (§4.2).services/ips_client.py, #207 / PDL Ch 4GET {IPS_BASE_URL}/api/v1/patients/<pid>/blocks?active=true usingCDR_PDHC_SERVICE_KEY (spärr is patient-scoped, notauthor_org_guid, fallingprovider_org_guid) is actively blocked are dropped, unlessindispensable_care lift matches by concept + date range. A 30 sinvalidate() is the webhook hook forservices/analysis_consent.py, #422) — for aPOST {IPS_BASE_URL}/api/v1/patients/analysis-filter with theresearch / quality_registry / statistics) and/stats and /runs apply org scope only (aggregate counts, no
per-patient spärr/consent join). /export applies none of the three.
Two tables.
cdr_6_observations (models/observation.py)| Group | Columns |
|---|---|
| Identity | id (pk), guid (unique uuid4) |
| Canonical clinical-context | service_request_guid, transaction_guid, concept_guid, patient_guid, provider_org_guid, contract_guid, requesting_org_guid, requester_user_guid, grant_token |
| Payload | fhir_observation_json (JSON, optional), value (text), response_type, payload_hash, dedup_key |
| Statuses | resolution_status = "synthetic", validation_status = "synthetic", is_late |
| Timestamps | received_at, created_at, effective_at |
| Characterisation (#178) | plan_definition_guid, care_plan_guid |
| Deprecated aliases (#294/#304) | activity_guid, author_org_guid — NULL on new rows (§5.3) |
| Cohort metadata (#178) | sim_run_id, cohort_name, cohort_description |
Indexed: the guid FKs, dedup_key, is_late, effective_at,
sim_run_id, cohort_name, plus the canonical/characterisation
columns. Per top_rules.md rule 5, all schema changes go through
Alembic — no ALTER TABLE outside a migration.
cdr_6_read_audit (models/read_audit.py)One row per read-endpoint response (services/audit.record_read_audit,
called by /runs, /runs/<id>/observations, /stats). PDL Ch 4 + Lag
(2022:913) chain-of-custody: who read, what they got, how filtered.
Columns: id, timestamp, caller_service (sim.pdhc |
bearer), caller_user_guid, caller_org_guids (JSON), route,
sim_run_id, patient_guid, concept_guid_filter, n_rows_returned,
response_status, session_id, and the X1 tuple role_guid /
purpose / access_basis. Bearer identity comes from the validated SSO
blob; service-key callers may forward X-Caller-User-Guid /
X-Caller-Org-Guids / X-SSO-Session-Id. The write is best-effort —
an audit-write failure is logged and swallowed, never blocking the data
response.
Chain: 0001_initial → 0002_cohort_metadata → 0003_read_audit →
0004_canonical_context → 0005_read_audit_reform_tuple.
Current head: 0005_read_audit_reform_tuple. (0003 adds the read-audit
table; 0004 adds the canonical context fields requesting_org_guid /
requester_user_guid; 0005 adds the read-audit X1 tuple columns.)
/transfer — self-promote engine (services/transfer.py)The engine that used to live in sim's CLI (sim cdr-transfer) now runs
server-side in the data plane. plan.pdhc's trigger UI (#530) just proxies
the route, presenting X-Source-Service: sim.pdhc with the sim key held
as its CDR6 key, so cdr_6 stays strictly sim-only.
Flow of run_transfer():
to == self (400) and unknown/unconfigured targets (400,TransferError).SELECT count +dry_run returns {mode, to, sim_run_id, rows,
runs} here.READ_PAGE = 1000),expunge_all() after each page so peak memory stays ~one page + one_TRANSFER_KEYS) and, crucially, given a synthesisedfhir_resource (below). Batches of batch_size (≤100) are POSTed to{base_url}/api/v1/ingest/batch with headers X-Source-Service,X-Service-Key, X-Sim-Run-Id, X-Request-Id.verified is True only when every batch returned 2xx,accepted + duplicate == rows read.purge_source): only on a verified copy are the run(s)sim_run_id are never purged.Summary: {mode, to, sim_run_id, rows, runs, accepted, duplicate,
rejected, http_errors, verified, purged}.
_build_fhir_observation() synthesises a minimal FHIR R5 Observation
per row so the destination cdr materialises a queryable Live
Observation, not just an ingest_raw staging row (cdr.pdhc builds the
Live Observation from body["fhir_resource"]). The shape matches
cdr_app.ingest_pipeline.build_live_observation_row:
id = row guidcode.coding[0] = {system: "urn:pdhc:concept", code: <concept_guid>}urn:pdhc:concept/<guid> — see project_cdr_code_canonical_format)subject.reference = Patient/<patient_guid>effectiveDateTime = effective_atperformer[].identifier.value = provider_org_guid (fallbackauthor_org_guid)response_type (quantity/numeric/slider/…) →valueQuantity; everything else → valueString. Unit isfeedback_unit_lives_on_concept).create_app() reads all config from the environment (no config.py).
| Variable | Purpose |
|---|---|
DATABASE_URL |
SQLAlchemy URI (compose builds it from the POSTGRES_* pair) |
COMPOSE_PROJECT_NAME |
Pins compose project (cdr_6_pdhc) — load-bearing |
CDR_INSTANCE |
Self-name for transfer self-target check (default cdr_6) |
APP_PORT / DB_PORT / DB_VOLUME |
Host ports + pgdata volume name |
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB |
DB creds/name |
SIM_PDHC_SERVICE_KEY |
The write/strict-path service key |
IPS_BASE_URL |
ips.pdhc base for spärr + analysis-consent (#207/#422) |
CDR_PDHC_SERVICE_KEY |
cdr_6's own key for its outbound ips calls |
SSO_BASE_URL / SSO_CLIENT_ID / SSO_CLIENT_SECRET |
bearer-token validation against sso.pdhc (#217) |
CDRn_BASE_URL (n=1..5) |
destination CDR base URLs for /transfer |
CDRn_SERVICE_KEY (n=1..5) |
optional per-target key; defaults to SIM_PDHC_SERVICE_KEY |
CDR_TRANSFER_SOURCE_SERVICE |
outbound X-Source-Service (default sim.pdhc) |
CDR_TRANSFER_TIMEOUT |
outbound push timeout seconds (default 30) |
CDR_TRANSFER_TARGETS is assembled at boot from CDRn_BASE_URL /
CDRn_SERVICE_KEY; an unconfigured target (no base_url) is rejected at
call time. Leaving SSO_* / IPS_BASE_URL blank simply disables those
integrations (bearer validation returns None; ips calls short-circuit).
Local dev (rare; cdr_6 is primarily a deployed service):
cd cdr_6_app
docker-compose up -d db
.venv/bin/flask db upgrade
.venv/bin/flask run # never in prod — daemonised gunicorn instead
Production lives on miserver under the release-symlink layout from
CLAUDE.md §7:
/usr/local/www/cdr_6.pdhc/
├── current → releases/<ISO-UTC-timestamp>
├── releases/<ISO-UTC-timestamp>/
│ ├── cdr_6_app/
│ └── start.sh
└── shared/
start.sh is the single deploy entry point per CLAUDE.md §6:
.env.colima stop / colima delete.docker-compose -p $COMPOSE_PROJECT_NAME up -d db (pinned projectproject_termbank_location).--force-recreate so a fresh .envinfra_docker_restart_env).flask db upgrade./healthz with bounded wait; on failure print the last 40start.sh may only touch 9055/9056. Sibling services share the Colima
VM (CLAUDE.md §8 #4) — never run colima stop from here.
cdr_6 holds all ingested rows forever (until explicitly purged).
There is no TTL, no rotation, no archival. The unit of identity is
sim_run_id:
(profile, count, seed) emits the samesim_run_id and rows — re-ingesting produces 100% duplicates and 0%effective_at is present).DELETE /api/v1/runs/<id> removes every row for a run, full stop./transfer with purge_source is the othersim_run_id; nothing else partitions the(patient_guid, concept_guid) sit/stats without run_id= sums them.cdr_6_pgdata on the miserver SSD (D3' fallback per/Users/miserver/pg_dump_cdr_6.sh runs nightly at 03:00se.pdhc.pgdump-cdr6/Library/LaunchDaemons/se.pdhc.pgdump-cdr6.plist). Output:/Volumes/T9/cdr_6_dumps/cdr_6-YYYYMMDD-HHMMSS.pgdump, 30-daycron in ticket #321 (2026-06-29) after theinfra_t9_spindown_trap). Source-of-truth plist lives inmiserver-ops/launchd/se.pdhc.pgdump-cdr6.plist./Volumes/T9/.sim_run_lock file (at the mount root, notcdr_6_dumps/) is touched during a sim push and cleared onLives upstream in
sim.pdhc/docs/phase_3_operator_runbook.md.
Four mitigations: pmset -a disksleep 0 (keep T9 awake), .alive
keepalive, .sim_run_lock (§12), and a 70% T9 fill alert (fail-closed
before the disk does). All four are operator-side, on miserver.
cdr_6_app/tests/ — pytest, runs against a sqlite-in-memory DB by
default and Postgres when DATABASE_URL points at one.
cd cdr_6_app
.venv/bin/python -m pytest -x -q
Coverage spans the auth decorators (service-key + bearer), ingest single
+ batch, dedup, stats/runs, delete-run, and the transfer/export paths.
The migration head is exercised by a fresh flask db upgrade. Note the
PDHC test-harness gotchas (AUTH_MODE env, sqlite StaticPool, service-key
write headers, SSO revalidation mock) in memory
infra_pdhc_test_harness_gotchas.
analyse.pdhc is the planned cohort-analysis service that will read
cdr_6 longitudinally — not yet built. It will use the existing surface:
GET /api/v1/runs to discover cohorts, GET /api/v1/stats?run_id=… for
gross counts, GET /api/v1/runs/<id>/observations for filtered samples,
and direct SQL via psql on 9056 for analytical queries that don't fit
the HTTP shape. Nothing about cdr_6 needs to change before it is built.
All ports bind to 127.0.0.1 (loopback only); there is no *.pdhc.se
vhost — reach it via an SSH tunnel (detailed in §3 above).
| Port | Service |
|---|---|
| 9055 | Flask application (Gunicorn) |
| 9056 | PostgreSQL database |
| 9057–9059 | Reserved |