Viewing your data¶

This page explains how to open and inspect your delivered CSV files on your VM. No coding is required — you can view data directly using tools that are already installed.

See Understanding your data first if you haven't already, particularly for the folder structure and which files to ignore.

Viewing CSV files in VS Code¶

VS Code is pre-installed on all OUH VMs and has several extensions that make viewing CSV files straightforward.

Rainbow CSV¶

Rainbow CSV highlights each column in a different colour, making it much easier to read raw CSV content at a glance.

Open VS Code
Go to File → Open File and navigate to your CSV file
The file will open with columns colour-coded automatically

You can also click on any cell to see which column it belongs to in the status bar at the bottom.

Edit CSV¶

The Edit CSV extension (janisdd.vscode-edit-csv) provides a spreadsheet-style grid view of CSV files, similar to opening a file in Excel.

Open a CSV file in VS Code
Click the Edit CSV button that appears in the top-right corner of the editor, or right-click the file tab and choose Open with Edit CSV
The file opens as a grid — you can scroll, sort columns, and filter rows

Large files

Edit CSV is best suited to files under ~50 MB. For larger files, use the chunked approach described below.

Excel Viewer¶

The Excel Viewer extension (GrapeCity.gc-excelviewer) provides a similar grid view and also supports .xlsx files.

Open a CSV file in VS Code
Click the Open Preview button in the top-right, or right-click and choose Open with Excel Viewer

Peeking at a file without opening it fully¶

For large files, it is useful to inspect just the first few rows before loading the whole thing into Python or R.

Windows (PowerShell)¶

Open a terminal in VS Code (Terminal → New Terminal) and run:

# Show the first 10 lines
Get-Content "C:\path\to\your\data\demographics\demographics.csv" -TotalCount 10

# Show the header row only
Get-Content "C:\path\to\your\data\demographics\demographics.csv" -TotalCount 1

# Count the number of rows (subtract 1 for the header)
(Get-Content "C:\path\to\your\data\demographics\demographics.csv").Count

Linux (terminal)¶

# Show the first 10 lines
head -n 10 /path/to/your/data/demographics/demographics.csv

# Show the header row only
head -n 1 /path/to/your/data/demographics/demographics.csv

# Count the number of rows (subtract 1 for the header)
wc -l /path/to/your/data/demographics/demographics.csv

Working with large files¶

Some clinical datasets can be very large — tens of millions of rows. Opening these directly in a viewer will be slow or may crash. The recommended approach is to either split the file first, or read it in chunks using Python or R.

Option 1 — Split the file into smaller chunks¶

This creates several smaller files you can open individually.

Windows (PowerShell):

$inputFile = "C:\path\to\your\data\diagnosis\diagnosis.csv"
$outputDir = "C:\path\to\your\data\diagnosis\chunks"
$chunkSize = 500000  # rows per chunk (adjust as needed)

New-Item -ItemType Directory -Force -Path $outputDir

$header = Get-Content $inputFile -TotalCount 1
$reader = [System.IO.StreamReader]::new($inputFile)
$reader.ReadLine() | Out-Null  # skip header

$chunkNum = 1
$rows = [System.Collections.Generic.List[string]]::new()

while (-not $reader.EndOfStream) {
    $rows.Add($reader.ReadLine())
    if ($rows.Count -ge $chunkSize) {
        $outFile = Join-Path $outputDir "chunk_$chunkNum.csv"
        $header | Set-Content $outFile
        $rows | Add-Content $outFile
        $rows.Clear()
        $chunkNum++
    }
}

# Write any remaining rows
if ($rows.Count -gt 0) {
    $outFile = Join-Path $outputDir "chunk_$chunkNum.csv"
    $header | Set-Content $outFile
    $rows | Add-Content $outFile
}

$reader.Close()
Write-Host "Done. Created $chunkNum chunk(s) in $outputDir"

Linux (terminal):

INPUT="/path/to/your/data/diagnosis/diagnosis.csv"
OUTPUT_DIR="/path/to/your/data/diagnosis/chunks"
CHUNK_SIZE=500000  # rows per chunk

mkdir -p "$OUTPUT_DIR"

# Save the header
HEADER=$(head -n 1 "$INPUT")

# Split the file (excluding header) into chunks
tail -n +2 "$INPUT" | split -l $CHUNK_SIZE - "$OUTPUT_DIR/chunk_"

# Add the header back to each chunk
for f in "$OUTPUT_DIR"/chunk_*; do
    mv "$f" "$f.csv"
    sed -i "1s/^/$HEADER\n/" "$f.csv"
done

echo "Done. Chunks written to $OUTPUT_DIR"

Each chunk file can then be opened in Edit CSV or Excel Viewer in VS Code.

Option 2 — Read in chunks with Python¶

If you want to process the data rather than just view it, use pandas to read in chunks:

import pandas as pd

filepath = r"C:\path\to\your\data\diagnosis\diagnosis.csv"

# Read and process 100,000 rows at a time
chunk_size = 100_000
results = []

for chunk in pd.read_csv(filepath, chunksize=chunk_size):
    # Example: filter to rows of interest before collecting
    filtered = chunk[chunk["some_column"] == "some_value"]
    results.append(filtered)

# Combine the filtered chunks
combined = pd.concat(results, ignore_index=True)
print(f"Matching rows: {len(combined):,}")

Option 3 — Read in chunks with R¶

library(readr)

filepath <- "C:/path/to/your/data/diagnosis/diagnosis.csv"
chunk_size <- 100000

# Read and process in chunks using a callback
results <- list()
chunk_num <- 1

read_csv_chunked(
  filepath,
  callback = function(chunk, pos) {
    # Example: filter rows of interest
    filtered <- chunk[chunk$some_column == "some_value", ]
    results[[chunk_num]] <<- filtered
    chunk_num <<- chunk_num + 1
  },
  chunk_size = chunk_size,
  show_col_types = FALSE
)

combined <- bind_rows(results)
message("Matching rows: ", nrow(combined))

Installed VS Code extensions¶

The following extensions are pre-installed on all OUH VMs:

Extension	ID	What it does
Rainbow CSV	`mechatroner.rainbow-csv`	Colour-codes CSV columns for easy reading
Edit CSV	`janisdd.vscode-edit-csv`	Grid/spreadsheet view of CSV files
Excel Viewer	`GrapeCity.gc-excelviewer`	Grid view, also supports .xlsx
Jupyter	`ms-toolsai.jupyter`	Run Jupyter notebooks inside VS Code
Jupyter Cell Tags	`ms-toolsai.vscode-jupyter-cell-tags`	Tag and organise notebook cells
Python	`ms-python.python`	Python language support
Pylance	`ms-python.vscode-pylance`	Python type checking and IntelliSense
R	`REditorSupport.r`	R language support
R Debugger	`RDebugger.r-debugger`	R debugging support
MSSQL	`ms-mssql.mssql`	SQL Server / Azure SQL querying
Docker	`ms-azuretools.vscode-docker`	Docker container management
YAML	`redhat.vscode-yaml`	YAML language support
PowerShell	`ms-vscode.powershell`	PowerShell language support
GitLens	`eamodio.gitlens`	Enhanced Git integration
Parquet Viewer	`dvirtz.parquet-viewer`	View Parquet files
CodeSnap	`adpyke.codesnap`	Screenshot code snippets