Data Catalog - R2 Storage (landbruget-data)

Data lake with 130+ datasets across bronze/silver/gold medallion layers in Cloudflare R2.

Discovering Data

Use rclone to browse R2 — never use gsutil or google.cloud.storage.

# List top-level directories
rclone lsd r2:landbruget-data/

# List datasets in a layer
rclone lsd r2:landbruget-data/bronze/
rclone lsd r2:landbruget-data/silver/
rclone lsd r2:landbruget-data/gold/

# List snapshots (timestamped subdirectories) for a dataset
rclone lsd r2:landbruget-data/silver/subsidies/

# List files in a snapshot
rclone ls r2:landbruget-data/silver/subsidies/

# Find latest snapshot for a dataset
rclone lsd r2:landbruget-data/gold/field_production_2024/ | tail -1

Important: Some folder names contain spaces (e.g., silver/animal welfare/, silver/work permits/). Always quote paths when scripting.

Reading Data with DuckDB

DuckDB with R2 auth is the primary way to query data. Use StorageAccess from backend/common/storage/core.py:

from common.storage.core import StorageAccess

storage = StorageAccess()

# Read a parquet file into DuckDB
storage.create_table_from_storage_parquet("my_table", "landbruget-data/silver/subsidies/20260401_020000/data.parquet")

# Query it
result = storage.execute_query("SELECT cvr_number, COUNT(*) FROM my_table GROUP BY cvr_number")

Or use DuckDB directly after auth setup:

import duckdb
from common.storage.filesystem import setup_duckdb_cloud_auth

conn = duckdb.connect()
setup_duckdb_cloud_auth(conn)

# Query directly from R2
result = conn.execute("""
    SELECT cvr_number, SUM(area_ha) as total_area
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet')
    GROUP BY cvr_number
""").fetchdf()

Environment Variables

# R2 credentials (required)
R2_ACCESS_KEY_ID=<access-key>
R2_SECRET_ACCESS_KEY=<secret-key>
R2_ACCOUNT_ID=<account-id>

# Bucket name (defaults to "landbruget-data")
R2_BUCKET=landbruget-data
# Or: STORAGE_BUCKET=landbruget-data

Medallion Architecture

• bronze/ — Raw data exactly as received (133 datasets)
• silver/ — Cleaned, validated, standardized (126 datasets)
• gold/ — Analysis-ready, joined datasets (86 datasets)

Full Dataset Inventory

Bronze (133 datasets)

Silver (126 datasets)

Gold (86 datasets)

Other Top-Level

• api/ — API-related data
• cvr_collections/ — CVR collection data

Key Identifiers

Dataset Quick Reference

Okonomi (Finance)

Landbrugsareal (Agricultural Land)

Miljo (Environment)

Husdyr (Livestock)

Medarbejdere (Employees)

Cross-Dataset Joins

CVR-based joins (most common)

# Join subsidies with pesticides on CVR using DuckDB
conn.execute("""
    SELECT s.cvr_number, s.tilskudsberetigt, p.PesticideName
    FROM read_parquet('r2://landbruget-data/silver/subsidies/*/data.parquet') s
    JOIN read_parquet('r2://landbruget-data/gold/pesticide_disaggregation_2023_2024/*/data.parquet') p
    ON s.cvr_number = p.cvr_number
""")

Field-based joins

# Join field production with environmental analysis
conn.execute("""
    SELECT fp.field_id, fp.yield_estimate, fe.bnbo_overlap_pct
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet') fp
    JOIN read_parquet('r2://landbruget-data/gold/field_environmental_analysis_fields_2024/*/data.parquet') fe
    ON fp.field_id = fe.field_id
""")

Data Update Schedule

Related Skills

• okonomi/ - Financial data: subsidies, property values
• landbrugsareal/ - Field and crop data: FVM marker, production
• miljo/ - Environmental data: pesticides, nitrogen, BNBO
• husdyr/ - Livestock data: CHR, movements, welfare
• medarbejdere/ - Employee data: inspections, safety

Troubleshooting

Check R2 access

rclone lsd r2:landbruget-data/

If rclone fails

Check ~/.config/rclone/rclone.conf has an [r2] section with:

• type = s3
• provider = Cloudflare
• accesskeyid and secretaccesskey
• endpoint pointing to your R2 account

Large Files

Use DuckDB — never load large parquet files into Pandas:

conn.execute("""
    SELECT cvr_number, SUM(area_ha) as total_area
    FROM read_parquet('r2://landbruget-data/gold/field_production_2024/*/data.parquet')
    GROUP BY cvr_number
""").fetchdf()

CRS

All geometry stored in EPSG:4326 (WGS84) in Supabase. Bronze/Silver/Gold processing uses EPSG:25832 (UTM 32N).