Using R in Databricks¶

R is supported in Databricks via the classic all-purpose cluster. Serverless compute only supports Python and SQL — R notebooks must be attached to the classic cluster.

Attaching to the classic cluster¶

The classic cluster is shared across the workspace. To use it:

Open your notebook
Click the cluster selector at the top of the notebook
Select the shared classic cluster
Wait for it to attach (a few seconds if already running)

Accessing the data mart with SparkR¶

library(SparkR)

# Load a table from the mart
df <- sql("SELECT * FROM <mart_catalog_name>.<schema_name>.<table_name> LIMIT 1000")

# Show schema
printSchema(df)

# Convert to R data frame for analysis
rdf <- collect(df)
head(rdf)

Filtering with SQL¶

result <- sql("
  SELECT *
  FROM <mart_catalog_name>.<schema_name>.<table_name>
  WHERE column_a = 'value'
    AND column_b IS NOT NULL
")

rdf <- collect(result)
nrow(rdf)

Analysis with R¶

Once collected into an R data frame, use standard R packages:

library(dplyr)
library(ggplot2)

# Summarise
rdf %>%
  group_by(group_column) %>%
  summarise(n = n(), mean_val = mean(numeric_column, na.rm = TRUE))

# Plot
ggplot(rdf, aes(x = numeric_column)) +
  geom_histogram(bins = 30) +
  labs(title = "Distribution of numeric_column") +
  theme_minimal()

Mixing R and Python in the same notebook¶

You can use both languages in the same notebook by switching cell types. To share data between them, use a temporary view:

# Python cell — create a temp view from a Spark DataFrame
df.createOrReplaceTempView("my_temp_view")

# R cell — read the temp view into R
rdf <- collect(sql("SELECT * FROM my_temp_view"))

Installing R packages¶

Unlike Python on serverless (which requires wheel files from a volume), R packages on the classic cluster install directly — the cluster is VNet-injected and routes through Nexus:

install.packages("dplyr")
install.packages("ggplot2")

Session-scoped vs persistent packages

Packages installed with install.packages() are session-scoped and lost when the cluster restarts. To persist packages across restarts, ask the TVS SDE team to add them as cluster libraries instead.

Saving results back to the workspace catalog¶

library(SparkR)

# Convert R data frame back to Spark
spark_df <- createDataFrame(rdf)

# Save to workspace catalog
saveAsTable(spark_df, "<workspace_catalog_name>.my_project.my_r_result", mode = "overwrite")