Analysis template

Download notebook

---

Using this notebook in Databricks

Download the notebook using the button above, then import it into your Databricks workspace:

1. In the Databricks UI, go to Workspace → Shared
2. Click the ⋮ menu and choose Import
3. Select File and upload the downloaded .py file
4. The notebook will appear in the Shared space and be accessible to everyone in your workspace

Adding it to Shared means all workspace members can access and run it. If you want a personal copy to edit, clone it into your own folder first (File → Clone).

Use this as a starting point for a new analysis. Copy it into your own folder before editing — go to File → Clone, then move the clone to your personal or project folder.

---

Setup

Configure your catalog, schema, and table names here. The rest of the notebook references these variables so you only need to change them in one place.

# -- Configuration --
CATALOG = "<mart_catalog_name>"       # your project's mart catalog
SCHEMA = "<schema_name>"              # edit this
TABLE = "<table_name>"                # edit this

WORKSPACE_CATALOG = "<workspace_catalog_name>"  # your project's workspace catalog
OUTPUT_SCHEMA = "my_project"          # edit this — your output schema name

full_table = f"{CATALOG}.{SCHEMA}.{TABLE}"

---

1. Load data

df = spark.table(full_table)

print(f"Table: {full_table}")
print(f"Rows: {df.count():,}")
print(f"Columns: {len(df.columns)}")
df.printSchema()

# Preview
df.show(10, truncate=False)

---

2. Data quality checks

from pyspark.sql.functions import col, count, when, isnan

# Null counts per column
df.select([
    count(when(col(c).isNull(), c)).alias(c)
    for c in df.columns
]).show()

# Row count
print(f"Total rows: {df.count():,}")

# Distinct values in a key column
print(f"Distinct patients: {df.select('<patient_id_column>').distinct().count():,}")

---

3. Analysis

# Filter
filtered = df.filter(
    (col("<column_a>") == "value") &
    (col("<column_b>").isNotNull())
)

# Aggregate
from pyspark.sql.functions import count, avg, min, max

summary = filtered.groupBy("<group_column>").agg(
    count("*").alias("n"),
    avg("<numeric_column>").alias("mean"),
)
summary.show()

SQL alternative:

SELECT <group_column>,
       COUNT(*) AS n,
       AVG(<numeric_column>) AS mean
FROM <mart_catalog_name>.<schema_name>.<table_name>
WHERE <column_a> = 'value'
  AND <column_b> IS NOT NULL
GROUP BY <group_column>
ORDER BY n DESC

Convert to pandas for further analysis or plotting:

pdf = summary.toPandas()

---

4. Visualisation

import matplotlib.pyplot as plt

# Edit to match your data
fig, ax = plt.subplots(figsize=(10, 6))
pdf['numeric_column'].hist(bins=30, ax=ax)
ax.set_title('Distribution of numeric_column')
ax.set_xlabel('Value')
ax.set_ylabel('Count')
plt.tight_layout()
plt.show()

---

5. Save results

# Create your output schema (run once)
spark.sql(f"CREATE SCHEMA IF NOT EXISTS {WORKSPACE_CATALOG}.{OUTPUT_SCHEMA}")

# Save a result table
output_table = f"{WORKSPACE_CATALOG}.{OUTPUT_SCHEMA}.my_result_table"
result.write.mode("overwrite").saveAsTable(output_table)
print(f"Saved to {output_table}")

See Saving and sharing results for write mode options.

---

Notes

Add any notes, caveats, or follow-up items here.