Understanding your data¶

This page explains how data is delivered to your workspace and what you can expect to find.

What is a cohort?¶

All data delivered through the TVS SDE is based on an approved cohort — a defined set of patients that meets the criteria specified in your research protocol and approved by the governance process. Your cohort is the foundation of your dataset: every table you receive is filtered to only include records for patients within your cohort.

This means:

You will only ever see data for patients in your approved cohort
Any new data request (additional datasets, extended cohort criteria) must go through the approval process again
The cohort is defined and managed by the TVS SDE team — you cannot modify it yourself

How data is delivered¶

Data is delivered as a set of CSV files, one file per dataset, organised into folders by dataset name. Your delivery will always include a demographics table, plus whichever clinical datasets were approved for your project.

Folder structure¶

your-delivery/
├── demographics/
│   └── demographics.csv
├── diagnosis/
│   └── diagnosis.csv
├── cerner_biochemistry/
│   └── cerner_biochemistry.csv
└── ... (other approved datasets)

Each folder contains one CSV file. You may also see some additional files alongside the CSV — these are internal Spark metadata files generated during the export process and can be ignored:

demographics/
├── _SUCCESS                          ← ignore
├── _committed_635578353...           ← ignore
├── _started_635578353...             ← ignore
└── demographics.csv                  ← this is your data

Tip

Only open files with a .csv extension. The other files contain no data and are safe to ignore.

The demographics table¶

Every delivery includes a demographics table. This is the core table that links your cohort to all other datasets. It contains one row per patient and includes:

Column	Description
`salted_warehouse_identifier`	Anonymised patient identifier — use this to join to other tables
`ETHNIC_GRP_CD`	Ethnic group code
`ETHNIC_GROUP`	Ethnic group description
`SEX_CD`	Sex code
`SEX`	Sex description
`BIRTH_DT_TM`	Date of birth
`DECEASED_DT_TM`	Date of death (where applicable)
`BEG_EFFECTIVE_DT_TM`	Record effective date
`INDEX_MULTIPLE_DEPRIVATION_DECILE_2015`	IMD decile (2015)

Patient identifier

The salted_warehouse_identifier is an anonymised identifier specific to your project. It is not the same as an NHS number or hospital number and cannot be used to identify patients outside the TRE.

How to join datasets¶

All clinical tables link back to demographics via the salted_warehouse_identifier column. Use this as your join key across all datasets.

demographics.salted_warehouse_identifier
    = diagnosis.salted_warehouse_identifier
    = cerner_biochemistry.salted_warehouse_identifier
    = ... (all other tables)

How to access your data¶

Data arrives in your workspace via one of three routes depending on how your project is set up:

Route	When to use	Where data appears
Airlock	Data delivered as CSV files by the TVS SDE team	Downloaded via Storage Explorer to your VM
Shared storage (Z: drive)	Data placed in shared storage by your workspace manager	`Z:\` on your VM
Databricks	Direct access to the data mart via Unity Catalog	Databricks workspace

For most projects receiving CSV deliveries, data arrives via the airlock. Once your import request is approved, open the request in the Airlock tab, copy the SAS URL, connect via Azure Storage Explorer, and download the files to your VM. See Airlock requests for step-by-step instructions.

Available datasets¶

The datasets available to your project depend on your approved protocol. Common datasets include:

Dataset	Description
`demographics`	Core patient demographics — always included
`diagnosis`	ICD-10 coded diagnoses
`cerner_biochemistry`	Biochemistry results from Cerner
`cerner_microbiology`	Microbiology results from Cerner
`cerner_winpath_biochemistry`	Biochemistry results from WinPath
`cerner_winpath_microbiology`	Microbiology results from WinPath
`cerner_winpath_microbiology_antibiotics`	Antibiotic sensitivity results
`cerner_winpath_microbiology_isolates`	Microbiology isolates
`cerner_winpath_microbiology_reports`	Microbiology free-text reports
`epr_medications_administered`	Medications administered from EPR
`epr_orders_meds`	Medication orders from EPR
`epr_orders_meds_details`	Medication order details
`epr_orders_pharmacy`	Pharmacy orders
`epr_orders_pharmacy_details`	Pharmacy order details
`imaging_reports`	Radiology imaging reports
`pathology_cytology_reports`	Cytology pathology reports
`pathology_neuro_reports`	Neuropathology reports
`pathology_noc_reports`	NOC pathology reports
`pathology_surgical_reports`	Surgical pathology reports
`starlims_diagnoses`	Diagnoses from StarLIMS

Dataset descriptions

Detailed descriptions of each dataset, including column-level data dictionaries, will be added to this documentation in a future update. If you need information about a specific dataset in the meantime, contact the TVS SDE team.