Exploring the data mart

Download notebook

---

Using this notebook in Databricks

Download the notebook using the button above, then import it into your Databricks workspace:

1. In the Databricks UI, go to Workspace → Shared
2. Click the ⋮ menu and choose Import
3. Select File and upload the downloaded .py file
4. The notebook will appear in the Shared space and be accessible to everyone in your workspace

Adding it to Shared means all workspace members can access and run it. If you want a personal copy to edit, clone it into your own folder first (File → Clone).

---

Exploring the data mart

This page walks you through browsing the data mart catalog, inspecting tables, and running your first queries. All examples use <mart_catalog_name> — replace this with your project's mart catalog name.

---

Browsing the catalog

Using the Catalog Explorer

The easiest way to explore the mart is via the Catalog tab in the left sidebar of the Databricks UI. Expand your mart catalog to browse schemas and tables, view column names and types, and preview sample data — all without writing any code.

Using SQL

-- List all available catalogs
SHOW CATALOGS;

-- List schemas in the mart
SHOW SCHEMAS IN <mart_catalog_name>;

-- List tables in a schema
SHOW TABLES IN <mart_catalog_name>.<schema_name>;

Using Python

# List catalogs
spark.catalog.listCatalogs()

# List schemas
spark.catalog.listDatabases()

# List tables in a schema
spark.catalog.listTables("<schema_name>")

---

Inspecting a table

-- View column names and types
DESCRIBE <mart_catalog_name>.<schema_name>.<table_name>;

-- Preview rows
SELECT * FROM <mart_catalog_name>.<schema_name>.<table_name> LIMIT 10;

-- Row count
SELECT COUNT(*) FROM <mart_catalog_name>.<schema_name>.<table_name>;

# Load a table as a Spark DataFrame
df = spark.table("<mart_catalog_name>.<schema_name>.<table_name>")

print(f"Rows: {df.count():,}")
print(f"Columns: {len(df.columns)}")
df.printSchema()
df.show(10, truncate=False)

---

Querying with SQL

-- Filter rows
SELECT *
FROM <mart_catalog_name>.<schema_name>.<table_name>
WHERE column_a = 'value'
  AND column_b IS NOT NULL
LIMIT 100;

-- Aggregate
SELECT column_c, COUNT(*) AS n
FROM <mart_catalog_name>.<schema_name>.<table_name>
GROUP BY column_c
ORDER BY n DESC;

You can also run SQL inline from a Python cell:

result = spark.sql("""
    SELECT column_c, COUNT(*) AS n
    FROM <mart_catalog_name>.<schema_name>.<table_name>
    GROUP BY column_c
    ORDER BY n DESC
""")
result.show()

---

Converting to pandas

For smaller datasets or when using pandas-based libraries:

# Convert a Spark DataFrame to pandas
pdf = df.limit(100000).toPandas()

print(pdf.shape)
pdf.head()

Memory

Only convert to pandas when the dataset is small enough to fit in memory. For large tables, filter or aggregate in Spark first, then convert.

---

Visualisation

Databricks has built-in charting — click the chart icon below any SQL or DataFrame output to configure a chart without writing any code.

You can also use matplotlib or seaborn:

import matplotlib.pyplot as plt

# Numeric column — histogram
fig, ax = plt.subplots(figsize=(10, 6))
pdf['numeric_column'].hist(bins=30, ax=ax)
ax.set_title('Distribution of numeric_column')
ax.set_xlabel('Value')
ax.set_ylabel('Count')
plt.tight_layout()
plt.show()

# Categorical column — horizontal bar chart
counts = pdf['column_name'].value_counts()
fig, ax = plt.subplots(figsize=(10, 8))
counts.plot.barh(ax=ax)
ax.set_xlabel('Count')
ax.set_title('Distribution of column_name')
ax.invert_yaxis()  # largest at top
plt.tight_layout()
plt.show()

---

Saving results to your workspace catalog

-- Create a schema for your project (run once)
CREATE SCHEMA IF NOT EXISTS <workspace_catalog_name>.my_project;

-- Save a result as a table
CREATE TABLE <workspace_catalog_name>.my_project.my_filtered_data AS
SELECT * FROM <mart_catalog_name>.<schema_name>.<table_name>
WHERE column_c IS NOT NULL;

# Save a filtered DataFrame
(
    df.filter(df["column_c"].isNotNull())
      .write
      .mode("overwrite")
      .saveAsTable("<workspace_catalog_name>.my_project.my_filtered_data")
)

See Saving and sharing results for more detail.

---

Quick reference

Task	SQL	Python
List catalogs	SHOW CATALOGS	spark.catalog.listCatalogs()
List schemas	SHOW SCHEMAS IN catalog	spark.catalog.listDatabases()
List tables	SHOW TABLES IN catalog.schema	spark.catalog.listTables("schema")
Read a table	SELECT * FROM catalog.schema.table	spark.table("catalog.schema.table")
Row count	SELECT COUNT(*) FROM ...	df.count()
Save results	CREATE TABLE ... AS SELECT ...	df.write.saveAsTable(...)