Working with Databricks¶
This section covers how to use Databricks to access and analyse data in the TVS SDE data mart.
What is Databricks?¶
Databricks is an analytics platform built on Apache Spark. In the TVS SDE, it gives you direct query access to the data mart — structured tables that you can query, filter, join, and analyse on demand using Python, SQL, or R.
How this differs from CSV delivery¶
Previously, data was delivered as CSV files via the airlock process. Each dataset had to be specifically requested, exported, reviewed, and approved before it reached you. This worked, but it meant waiting for each new extract, working with static snapshots, and repeating the process for updated or additional data.
With Databricks you have direct query access to the data mart. The data lives in structured tables that are refreshed by the data TVS SDE team — no more waiting for exports.
CSV delivery still available
Direct Databricks access is an additional route, not a replacement. Projects that received data via CSV and airlock can continue to work that way. Your existing pandas or R scripts will still work — you can also load data from the mart directly and use the same code downstream.
The two-catalog model¶
Your Databricks workspace has access to two catalogs:
| Catalog | Purpose | Access |
|---|---|---|
<mart_catalog_name> |
The source data mart — all project datasets live here | Read-only |
<workspace_catalog_name> |
Your project's own workspace — for derived tables and outputs | Read/write |
Your workspace manager will provide you with the correct catalog names for your project.
You cannot write to the mart catalog
The mart catalog is read-only. All outputs and derived tables must be saved to your workspace catalog.
Compute options¶
Databricks offers two types of compute:
| Type | Supports | Notes |
|---|---|---|
| Serverless | Python, SQL | No cluster management needed — just run. Packages installed from a Unity Catalog volume. |
| Classic cluster | Python, SQL, R | Shared across the workspace. Required for R. Routes through Nexus for package installation. |
For most Python and SQL work, use serverless. For R, attach your notebook to the classic cluster.
Strict sandbox mode¶
You may see a prompt about "strict sandbox" mode when running notebooks. This is expected in the TRE — Databricks normally renders rich output (charts, HTML) via an external domain that the TRE blocks by design. Click "Run in strict sandbox" to render output locally instead. Charts and visualisations still work normally.
Getting started¶
Work through the notebooks in order:
- Exploring the data mart — browse catalogs, schemas, and tables; run your first queries
- Analysis template — a starting point for a new analysis
- Saving and sharing results — save outputs to your workspace catalog
- Installing packages — add Python packages on serverless compute
- Using R — R support via the classic cluster