SkillsAdd

databricks-solutions/lakebase-online-ml · Archived

databricks-spark-structured-streaming

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, implementing real-time data processing, handling stateful operations, or optimizing streaming performance.

First seen Aug 11, 2026

Installation

$ npx skills add databricks-solutions/lakebase-online-ml --skill databricks-spark-structured-streaming
View on skills.sh ↗ GitHub repository ↗

Stronger alternatives

This repository is archived — consider an actively maintained alternative.

agent-evaluation Same source · more installs
databricks-solutions/lakebase-online-ml

Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output qualit…

2 installs
analyzing-mlflow-session Same source
databricks-solutions/lakebase-online-ml

Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or intera…

1 installs
databricks-config Same source
databricks-solutions/lakebase-online-ml

Manage Databricks workspace connections: check which workspace you're connected to, switch work…

1 installs
analyzing-mlflow-trace Same source
databricks-solutions/lakebase-online-ml

Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a tr…

1 installs

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

frontend-design Same topic
anthropics/skills

Guidance for distinctive, intentional visual design when building new UI or reshaping an existi…

866.4K installs
agent-browser Same topic
vercel-labs/agent-browser

Browser automation CLI for AI agents. Use when the user needs to interact with websites, includ…

810.4K installs
vercel-react-best-practices Same topic
vercel-labs/agent-skills 697.8K installs
web-design-guidelines Same topic
vercel-labs/agent-skills

Review UI code for Web Interface Guidelines compliance. Use when asked to "review my UI", "chec…

617.3K installs
microsoft-foundry Same topic
microsoft/azure-skills

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and …

576.5K installs
azure-diagnostics Same topic
microsoft/azure-skills

Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe …

568.9K installs

Also in this package

Other skills from databricks-solutions/lakebase-online-ml · top by installs.

npx skills add databricks-solutions/lakebase-online-ml
agent-evaluation 2 analyzing-mlflow-session 1 databricks-config 1 analyzing-mlflow-trace 1 spark-python-data-source 1 searching-mlflow-docs 1 retrieving-mlflow-traces 1 querying-mlflow-metrics 1 mlflow-onboarding 1 instrumenting-with-mlflow-tracing 1

Browse all from databricks-solutions/lakebase-online-ml

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Repository health

Stars 1
License LICENSE.md
Default branch main
Open issues 0
Status Archived

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 2,777 B
  • docs SUMMARY.md 266 B

History

  1. First seen on skills.sh
  2. First recorded snapshot · 1 installs

SKILL.md

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

Pattern Description Reference
Kafka Streaming Kafka to Delta, Kafka to Kafka, Real-Time Mode See [kafka-streaming.md](kafka-streaming.md)
Stream Joins Stream-stream joins, stream-static joins See [stream-stream-joins.md](stream-stream-joins.md), [stream-static-joins.md](stream-static-joins.md)
Multi-Sink Writes Write to multiple tables, parallel merges See [multi-sink-writes.md](multi-sink-writes.md)
Merge Operations MERGE performance, parallel merges, optimizations See [merge-operations.md](merge-operations.md)

Configuration

Topic Description Reference
Checkpoints Checkpoint management and best practices See [checkpoint-best-practices.md](checkpoint-best-practices.md)
Stateful Operations Watermarks, state stores, RocksDB configuration See [stateful-operations.md](stateful-operations.md)
Trigger & Cost Trigger selection, cost optimization, RTM See [trigger-and-cost-optimization.md](trigger-and-cost-optimization.md)

Best Practices

Topic Description Reference
Production Checklist Comprehensive best practices See [streaming-best-practices.md](streaming-best-practices.md)

Production Checklist

  • Checkpoint location is persistent (UC volumes, not DBFS)
  • Unique checkpoint per stream
  • Fixed-size cluster (no autoscaling for streaming)
  • Monitoring configured (input rate, lag, batch duration)
  • Exactly-once verified (txnVersion/txnAppId)
  • Watermark configured for stateful operations
  • Left joins for stream-static (not inner)