All documentation Download (Markdown) Technical

CDR 6 · cdr_6 (loopback :9055)

Technical manual


cdr_6.pdhc — Technical reference

cdr_6 is the durable longitudinal store in the synthetic-data
pipeline:

sim.pdhc   →   cdr_6.pdhc   →   analyse.pdhc (future)
 CREATE        STORE             READ + analyse
 longitudinal  flat-row          cohort analysis
 cohort data   two-table         service to be built
 from plandef  schema            later
 Terms

sim.pdhc generates longitudinal time-series for each cohort patient
based on plan.pdhc PlanDefinitions and their leaf Terms (concepts).
cdr_6 is the durable store. The primary planned consumer is the
future analyse.pdhc service. Existing readers — the sim.web run
viewer, human operators on an SSO bearer token, notebooks, and direct
psql — work against the same instance.

Flat-row ingest (sim trusts its own emissions), but the read and
transfer surfaces are not unauthenticated or isolated any more:
reads accept SSO bearer tokens with per-org scoping (#217/#412), the
/runs/<id>/observations path enforces spärr (#207) and analysis
consent (#422), and /transfer synthesises FHIR R5 to promote content
into cdr1–5 (#529/#530/#531). Built for ticket
#178 Phase 3 (rescoped
2026-05-29) and extended since.

This document is the architectural overview for someone inheriting
the service. It is generated to match the code under cdr_6_app/ — if
it disagrees with the code, trust the code and fix this file.


1. What cdr_6 is and isn't

Is:
- A Flask app + Postgres pair running in Docker on miserver, both
bound to 127.0.0.1.
- The durable longitudinal store between sim.pdhc (the producer) and
analyse.pdhc (the future analytical consumer). Multiple sim runs
coexist in one DB, keyed by sim_run_id.
- The synthetic sink — a distinct codebase from cdr1–5. It stores
a flat schema and trusts sim's emissions verbatim on the write side
(no FHIR validation, no xlate, no plan.pdhc canonicalisation on
ingest).
- Integrated with the rest of the platform on the read/transfer side:
it validates SSO bearer tokens against sso.pdhc, calls ips.pdhc for
spärr blocks and analysis-consent verdicts, and synthesises FHIR R5
Observations when promoting rows into cdr1–5.

Isn't:
- A general-purpose CDR. The only valid X-Source-Service for the
write/strict paths is sim.pdhc (top_rules.md rule 2). Other
producers get their own sibling instance.
- Publicly reachable. No DNS, no nginx vhost. The only way in from
outside miserver is an SSH port-forward (sim/tunnel.py).
- A FHIR server for ingest. The stored schema is flat; FHIR bundles are
not the ingest wire format (cdr1–5 cover that case). cdr_6 does
synthesise FHIR on the way out during /transfer (§8).


2. Service layout

cdr_6.pdhc/
├── cdr_6_app/                # everything the container needs
│   ├── app/
│   │   ├── __init__.py       # create_app() — Flask factory + config wiring
│   │   ├── extensions.py     # SQLAlchemy + Migrate handles
│   │   ├── api/
│   │   │   ├── auth.py       # require_sim_service_key + require_read_auth
│   │   │   ├── health.py     # GET /healthz (no auth)
│   │   │   ├── ingest.py     # POST /api/v1/ingest[/batch]
│   │   │   ├── stats.py      # GET /api/v1/stats
│   │   │   ├── runs.py       # /runs, /runs/<id>/observations, /export, DELETE
│   │   │   └── transfer.py   # POST /api/v1/transfer (self-promote)
│   │   ├── models/
│   │   │   ├── observation.py  # cdr_6_observations
│   │   │   └── read_audit.py   # cdr_6_read_audit
│   │   └── services/
│   │       ├── sso_client.py       # bearer-token validation (#217)
│   │       ├── ips_client.py       # spärr / PatientBlock fetch (#207)
│   │       ├── analysis_consent.py # analysis-consent filter (#422)
│   │       ├── audit.py            # record_read_audit()
│   │       └── transfer.py         # run_transfer() + FHIR synth (#529/#531)
│   ├── migrations/           # Alembic; head = 0005_read_audit_reform_tuple
│   ├── Dockerfile
│   ├── docker-compose.yml    # db (postgres:16) + app (gunicorn)
│   ├── requirements.txt
│   ├── .env.example
│   └── wsgi.py
├── docs/
│   ├── dashboard_handoff.md  # consumer contract (cross-repo)
│   ├── technical.md          # ← this file
│   └── user_manual.md
├── start.sh                  # compose up + flask db upgrade + /healthz probe
├── top_rules.md
├── readme.md
├── progress.md
└── changed_files.md

3. Ports and network surface

Port Service Binds to Reachable from
9055 gunicorn (Flask app) 127.0.0.1 miserver loopback only
9056 postgres:16 127.0.0.1 miserver loopback only
9057–9 reserved
9046 (internal) gunicorn-in-container container network docker bridge only
5432 (internal) postgres-in-container container network docker bridge only

The compose file maps host 127.0.0.1:9055 → container :9046 for the
app, and 127.0.0.1:9056 → container :5432 for Postgres. Container
names are cdr_6_app and cdr_6_db.

There is no nginx vhost. There is no public DNS. There is no *.pdhc.se
subdomain. The only way in from outside miserver is an SSH tunnel:

ssh -L 9055:127.0.0.1:9055 miserver@192.168.1.154
ssh -L 9056:127.0.0.1:9056 miserver@192.168.1.154   # for direct psql

sim/tunnel.py does this automatically for sim's --target cdr_6 --tunnel.

For the outbound /transfer path, cdr_6 reaches cdr1–5 (and ips.pdhc /
sso.pdhc) over the miserver loopback / docker bridge — no tunnel; those
are server-side calls made by the running container.


4. Auth

Two decorators, both in cdr_6_app/app/api/auth.py:

4.1 require_sim_service_key — strict, write/transfer paths

Used by ingest, DELETE /runs/<id>, /export, and /transfer.
Requires:

X-Source-Service: sim.pdhc
X-Service-Key:    <value of SIM_PDHC_SERVICE_KEY>

Error codes are not all 401 — they distinguish the failure:

Condition HTTP Body
No X-Source-Service and no X-Service-Key 403 missing X-Source-Service
X-Source-Servicesim.pdhc 403 unknown source service: '<value>'
Source ok but X-Service-Key missing 401 missing X-Service-Key
Key present but wrong (or none configured) 403 invalid service key

Service-key callers are treated as is_admin and unfiltered
g.user_org_guids = None signals "no row filter".

4.2 require_read_auth — service-key OR SSO bearer, read paths (#217)

Used by /runs, /runs/<id>/observations, and /stats. It tries the
service-key first (only when both service headers are present); if the
caller instead sends Authorization: Bearer <token>, the token is
validated against sso.pdhc.

Scope caveat worth knowing: org filtering matches on
author_org_guid, which is the deprecated column left NULL on new
rows (§5.3). Bearer non-admin scoping therefore only matches
historical rows that still carry author_org_guid; SU-admin and
service-key reads are the practical read paths for current data.


5. Endpoints

All /api/v1/* paths are registered under that prefix; /healthz is at
the root.

Method Path Auth Purpose
GET /healthz none {status, service, database} — 200 ok, 503 degraded
POST /api/v1/ingest service-key Insert one observation row
POST /api/v1/ingest/batch service-key Insert ≤ 1000 rows; per-item status
GET /api/v1/stats[?run_id] read-auth Counts of observations / patients / concepts
GET /api/v1/runs read-auth Per-run summary (counts, period, cohort metadata)
GET /api/v1/runs/<id>/observations read-auth Capped, ordered sample of a run's rows (spärr + consent filtered)
GET /api/v1/export service-key Unfiltered, cursor-paged raw export (CDR-to-CDR source)
POST /api/v1/transfer service-key Self-promote rows into cdr1–5 (+FHIR synth)
DELETE /api/v1/runs/<id> service-key Purge every row with sim_run_id == id

/healthz matches CLAUDE.md §10:

{"status":"ok|degraded","service":"cdr_6.pdhc","database":"connected|unavailable"}

200 for ok, 503 for degraded. The DB probe is SELECT 1. The
status page at services.html doesn't render this correctly (no-cors
opaque response) — don't trust it; curl /healthz directly.

5.1 Ingest body schema

{
  // Required
  "concept_guid":         "<uuid>",
  "patient_guid":         "<uuid>",
  "value":                "<any> — coerced to string in storage",

  // Canonical clinical-context (all optional)
  "service_request_guid": "<uuid>",
  "transaction_guid":     "<uuid>",
  "provider_org_guid":    "<uuid>",
  "contract_guid":        "<uuid>",
  "requesting_org_guid":  "<uuid>",   // canonical (added #294/#304 phase 4)
  "requester_user_guid":  "<uuid>",   // canonical (added #294/#304 phase 4)
  "plan_definition_guid": "<uuid>",
  "care_plan_guid":       "<uuid>",

  // Sim extras / cohort provenance
  "grant_token":          "<token>",
  "sim_run_id":           "<sim run id>",
  "cohort_name":          "<from profile.name>",
  "cohort_description":   "<from profile.description>",
  "effective_at":         "<ISO 8601>",
  "response_type":        "<string>",
  "fhir_observation_json":{ },
  "payload_hash":         "<sha256>",
  "is_late":              false,

  // Legacy aliases — accepted, silently remapped, then dropped (§5.3)
  "activity_guid":        "<uuid>",   // → transaction_guid
  "author_org_guid":      "<uuid>"    // → provider_org_guid
}

Batch endpoint accepts either a bare array or {"items": [...]}. Hard
cap is 1000 (HTTP 413 above that). The batch response is
{total, accepted, duplicate, rejected, entries[]} with a per-item
{index, status, guid|error}.

5.2 Dedup

dedup_key = sha256(patient_guid | transaction_guid | effective_at),
computed in _build_row only when effective_at is present. A
DB-level collision (unique constraint) rolls the single row back and
returns {"status": "duplicate"} (HTTP 200); in a batch the item is
counted under duplicate. Sim is responsible for making rows unique
enough on its side; this is a backstop, not the primary contract.

5.3 Ingest field deprecations (#294 / #304)

Two legacy aliases are accepted at the API boundary and silently
remapped onto canonical columns, with a deprecation warning logged:

Legacy alias Canonical column Behaviour
activity_guid transaction_guid #294 RFC G2 — canonical wins if both sent
author_org_guid provider_org_guid #294/#304 phase 4 — canonical wins if both sent

The legacy columns (activity_guid, author_org_guid) still exist for
back-compat reads on historical rows, but new rows leave them NULL
ingest never populates them. to_dict() still emits both alias keys
during the deprecation window.

5.4 /runs/<id>/observations — filtered read

Returns a deterministically-ordered (patient_guid, concept_guid, effective_at) capped sample of one run. Optional ?concept_guid=,
?patient_guid=, ?limit= (default 200, hard max 5000). This is the
one read path where the spärr and analysis-consent filters fire
(§6), after org scoping. Used by the sim.web viewer.

5.5 /export — unfiltered raw export (#399)

Service-key only. Cursor-paginated by integer PK for a stable,
exhaustive sweep:

Response: {row_count, limit, sim_run_id, next_after_id, rows[...]}.
next_after_id is the last row's id when the page was full, or null
when exhausted; the caller pages until it is null.

Deliberately unfiltered — no spärr, no consent, no org scope — so a
CDR-to-CDR transfer moves the whole content, not a filtered sample.
This is the read source behind the sim cdr-transfer operator path and
a faithful mirror of /transfer's in-process reader.

5.6 /transfer — self-promote into cdr1–5 (#529/#530/#531)

Service-key only. cdr_6 reads its own rows and pushes them to a
destination CDR's /api/v1/ingest/batch. The source is always cdr_6
itself — the endpoint takes no external source, so no real
spärr/consent-protected patient data can be moved. See §8.

Body (JSON object):

Field Req? Meaning
to yes destination CDR name (cdr1..cdr5)
sim_run_id no one run; default = all rows in the sink
dry_run no count what would move, no writes
purge_source no after a verified-complete copy, delete the run(s) from cdr_6 (copy → move)
batch_size no push chunk size, clamped to 1..100 (default 100)

Returns the run_transfer() summary. Caller errors (unknown/
unconfigured target, target == self) → 4xx TransferError; push/HTTP
failures are reported inside the summary (http_errors), never raised.


GET /api/v1/runs/<id>/observations applies three filters in order:

  1. Org scope (_apply_org_scope) — bearer non-admin callers see
    only rows whose author_org_guid is in their blob org set (§4.2).
  2. Spärr / PatientBlock (services/ips_client.py, #207 / PDL Ch 4
    §4) — for the unique patient set in the window, cdr_6 calls
    GET {IPS_BASE_URL}/api/v1/patients/<pid>/blocks?active=true using
    cdr_6's own CDR_PDHC_SERVICE_KEY (spärr is patient-scoped, not
    caller-scoped). Rows whose source clinic (author_org_guid, falling
    back to provider_org_guid) is actively blocked are dropped, unless
    an indispensable_care lift matches by concept + date range. A 30 s
    TTL cache bounds staleness; invalidate() is the webhook hook for
    IPS Renov 6 / #202.
  3. Analysis consent (services/analysis_consent.py, #422) — for a
    human operator read (bearer path with an affiliation), cdr_6
    calls POST {IPS_BASE_URL}/api/v1/patients/analysis-filter with the
    derived purpose (research / quality_registry / statistics) and
    any research-project guids, forwarding the operator bearer so ips
    audits the human. Rows for non-consenting patients are dropped.
    Fail-closed: if ips can't answer, the read aborts 503, no
    data. Machine (service-key) reads and SU-admin-without-affiliation
    reads skip this join.

/stats and /runs apply org scope only (aggregate counts, no
per-patient spärr/consent join). /export applies none of the three.


7. Schema

Two tables.

7.1 cdr_6_observations (models/observation.py)

Group Columns
Identity id (pk), guid (unique uuid4)
Canonical clinical-context service_request_guid, transaction_guid, concept_guid, patient_guid, provider_org_guid, contract_guid, requesting_org_guid, requester_user_guid, grant_token
Payload fhir_observation_json (JSON, optional), value (text), response_type, payload_hash, dedup_key
Statuses resolution_status = "synthetic", validation_status = "synthetic", is_late
Timestamps received_at, created_at, effective_at
Characterisation (#178) plan_definition_guid, care_plan_guid
Deprecated aliases (#294/#304) activity_guid, author_org_guid — NULL on new rows (§5.3)
Cohort metadata (#178) sim_run_id, cohort_name, cohort_description

Indexed: the guid FKs, dedup_key, is_late, effective_at,
sim_run_id, cohort_name, plus the canonical/characterisation
columns. Per top_rules.md rule 5, all schema changes go through
Alembic — no ALTER TABLE outside a migration.

7.2 cdr_6_read_audit (models/read_audit.py)

One row per read-endpoint response (services/audit.record_read_audit,
called by /runs, /runs/<id>/observations, /stats). PDL Ch 4 + Lag
(2022:913) chain-of-custody: who read, what they got, how filtered.

Columns: id, timestamp, caller_service (sim.pdhc |
bearer), caller_user_guid, caller_org_guids (JSON), route,
sim_run_id, patient_guid, concept_guid_filter, n_rows_returned,
response_status, session_id, and the X1 tuple role_guid /
purpose / access_basis. Bearer identity comes from the validated SSO
blob; service-key callers may forward X-Caller-User-Guid /
X-Caller-Org-Guids / X-SSO-Session-Id. The write is best-effort —
an audit-write failure is logged and swallowed, never blocking the data
response.

7.3 Migrations

Chain: 0001_initial → 0002_cohort_metadata → 0003_read_audit → 0004_canonical_context → 0005_read_audit_reform_tuple.

Current head: 0005_read_audit_reform_tuple. (0003 adds the read-audit
table; 0004 adds the canonical context fields requesting_org_guid /
requester_user_guid; 0005 adds the read-audit X1 tuple columns.)


8. /transfer — self-promote engine (services/transfer.py)

The engine that used to live in sim's CLI (sim cdr-transfer) now runs
server-side in the data plane. plan.pdhc's trigger UI (#530) just proxies
the route, presenting X-Source-Service: sim.pdhc with the sim key held
as its CDR6 key, so cdr_6 stays strictly sim-only.

Flow of run_transfer():

  1. Reject to == self (400) and unknown/unconfigured targets (400,
    TransferError).
  2. Size the source without materialising it (SELECT count +
    distinct run ids) — a full load would OOM the worker on a
    million-row sink. dry_run returns {mode, to, sim_run_id, rows, runs} here.
  3. Stream the source PK-cursor-paged (READ_PAGE = 1000),
    expunge_all() after each page so peak memory stays ~one page + one
    batch. Each row is mapped to the flat-ingest item shape
    (_TRANSFER_KEYS) and, crucially, given a synthesised
    fhir_resource (below). Batches of batch_size (≤100) are POSTed to
    {base_url}/api/v1/ingest/batch with headers X-Source-Service,
    X-Service-Key, X-Sim-Run-Id, X-Request-Id.
  4. Verify: verified is True only when every batch returned 2xx,
    nothing was rejected, and accepted + duplicate == rows read.
  5. Move (purge_source): only on a verified copy are the run(s)
    deleted from cdr_6; an unverified transfer leaves the source intact
    and logs an error. Rows with a NULL sim_run_id are never purged.

Summary: {mode, to, sim_run_id, rows, runs, accepted, duplicate, rejected, http_errors, verified, purged}.

FHIR R5 synthesis (#531)

_build_fhir_observation() synthesises a minimal FHIR R5 Observation
per row so the destination cdr materialises a queryable Live
Observation, not just an ingest_raw staging row (cdr.pdhc builds the
Live Observation from body["fhir_resource"]). The shape matches
cdr_app.ingest_pipeline.build_live_observation_row:


9. Configuration

create_app() reads all config from the environment (no config.py).

Variable Purpose
DATABASE_URL SQLAlchemy URI (compose builds it from the POSTGRES_* pair)
COMPOSE_PROJECT_NAME Pins compose project (cdr_6_pdhc) — load-bearing
CDR_INSTANCE Self-name for transfer self-target check (default cdr_6)
APP_PORT / DB_PORT / DB_VOLUME Host ports + pgdata volume name
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB DB creds/name
SIM_PDHC_SERVICE_KEY The write/strict-path service key
IPS_BASE_URL ips.pdhc base for spärr + analysis-consent (#207/#422)
CDR_PDHC_SERVICE_KEY cdr_6's own key for its outbound ips calls
SSO_BASE_URL / SSO_CLIENT_ID / SSO_CLIENT_SECRET bearer-token validation against sso.pdhc (#217)
CDRn_BASE_URL (n=1..5) destination CDR base URLs for /transfer
CDRn_SERVICE_KEY (n=1..5) optional per-target key; defaults to SIM_PDHC_SERVICE_KEY
CDR_TRANSFER_SOURCE_SERVICE outbound X-Source-Service (default sim.pdhc)
CDR_TRANSFER_TIMEOUT outbound push timeout seconds (default 30)

CDR_TRANSFER_TARGETS is assembled at boot from CDRn_BASE_URL /
CDRn_SERVICE_KEY; an unconfigured target (no base_url) is rejected at
call time. Leaving SSO_* / IPS_BASE_URL blank simply disables those
integrations (bearer validation returns None; ips calls short-circuit).


10. Process model and deployment

Local dev (rare; cdr_6 is primarily a deployed service):

cd cdr_6_app
docker-compose up -d db
.venv/bin/flask db upgrade
.venv/bin/flask run            # never in prod — daemonised gunicorn instead

Production lives on miserver under the release-symlink layout from
CLAUDE.md §7:

/usr/local/www/cdr_6.pdhc/
├── current → releases/<ISO-UTC-timestamp>
├── releases/<ISO-UTC-timestamp>/
│   ├── cdr_6_app/
│   └── start.sh
└── shared/

start.sh is the single deploy entry point per CLAUDE.md §6:

  1. Load .env.
  2. Ensure Colima is up — never colima stop / colima delete.
  3. docker-compose -p $COMPOSE_PROJECT_NAME up -d db (pinned project
    name is load-bearing; see memory project_termbank_location).
  4. Stop the previous app container gracefully.
  5. Bring up the app container with --force-recreate so a fresh .env
    (especially a rotated key or a new SSO/IPS/transfer var) is picked up
    (memory infra_docker_restart_env).
  6. Run flask db upgrade.
  7. Probe /healthz with bounded wait; on failure print the last 40
    lines of the error log and exit non-zero.

start.sh may only touch 9055/9056. Sibling services share the Colima
VM (CLAUDE.md §8 #4) — never run colima stop from here.


11. Determinism and isolation

cdr_6 holds all ingested rows forever (until explicitly purged).
There is no TTL, no rotation, no archival. The unit of identity is
sim_run_id:


12. Storage and backups


13. Operational pre-reqs (Phase 3 runbook)

Lives upstream in
sim.pdhc/docs/phase_3_operator_runbook.md.
Four mitigations: pmset -a disksleep 0 (keep T9 awake), .alive
keepalive, .sim_run_lock (§12), and a 70% T9 fill alert (fail-closed
before the disk does). All four are operator-side, on miserver.


14. Testing

cdr_6_app/tests/ — pytest, runs against a sqlite-in-memory DB by
default and Postgres when DATABASE_URL points at one.

cd cdr_6_app
.venv/bin/python -m pytest -x -q

Coverage spans the auth decorators (service-key + bearer), ingest single
+ batch, dedup, stats/runs, delete-run, and the transfer/export paths.
The migration head is exercised by a fresh flask db upgrade. Note the
PDHC test-harness gotchas (AUTH_MODE env, sqlite StaticPool, service-key
write headers, SSO revalidation mock) in memory
infra_pdhc_test_harness_gotchas.


15. Future consumer: analyse.pdhc

analyse.pdhc is the planned cohort-analysis service that will read
cdr_6 longitudinally — not yet built. It will use the existing surface:
GET /api/v1/runs to discover cohorts, GET /api/v1/stats?run_id=… for
gross counts, GET /api/v1/runs/<id>/observations for filtered samples,
and direct SQL via psql on 9056 for analytical queries that don't fit
the HTTP shape. Nothing about cdr_6 needs to change before it is built.

Port Allocation

All ports bind to 127.0.0.1 (loopback only); there is no *.pdhc.se
vhost — reach it via an SSH tunnel (detailed in §3 above).

Port Service
9055 Flask application (Gunicorn)
9056 PostgreSQL database
9057–9059 Reserved