Apache Spark

Apache Spark 4.3 is here, bringing stateful real-time streaming to the engine you already use for batch, analytics, and AI.

⚡ Stateful Real-Time Mode: streaming aggregations, dropDuplicates, and transformWithState run with P99 end-to-end latency under 150 ms in reported benchmarks
🕰️ SCD Type 2 in Auto CDC: preserve history from Python, SQL, and Spark Connect
🪶 Project Feather: opt-in shuffle-free execution for eligible small queries, plus Arrow-native caching
🌐 Spark Connect: REST endpoints for sessions and operations, and a more complete JDBC driver
🤖 SQL: JSON_TABLE, VARIANT transformations, ASOF JOIN, and a preview of nanosecond timestamps
📅 Feature releases move to every three months instead of every six

Thank you to the 210+ contributors and the Apache Spark community behind 1,100+ Jira tickets.

Read more 👇
databricks.com/blog/introduci…

Apache Spark Data Source V2 is Spark’s way to support native table formats, not only files.

Szehon Ho walks through why that exists. Early Spark read and wrote file formats such as Parquet and used a thin Hive Metastore catalog. Apache Iceberg and Delta Lake added a much richer metadata layer so engines can run large, transactional operations. DSV2 expands that surface so a data source can plug that metadata into Spark, including through catalogs.

🎥 youtube.com/watch?v=MOQ3G8…

#ApacheSpark #OpenSource

We are happy to announce the availability of Apache Spark 4.1.4! 🎉

Spark 4.1.4 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-4.1 maintenance branch of Spark. We strongly recommend all 4.1 users to upgrade to this
stable release.

To download Spark 4.1.4, head over to the download page: spark.apache.org/downloads.html

Spark 4.1.4 is also available on PyPI (pyspark, pyspark-connect, pyspark-client) and the Maven Central Repository.

To view the release notes:
spark.apache.org/releases/spark…

We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you.

Agents are writing a ton of Spark code these days. Most of it is error-ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air.

At OSA Con 2026, @lisancao (@databricks) will cover declarative pipelines so structural mistakes get caught before data moves, Spark Connect so session context and credentials stay out of the agent’s hands, and skills and prompts that make one-shots more successful. She’ll also cover the tradeoffs between model choice, tokens, and deployment infrastructure.

🗓️ Monday, November 2
🕓 10:35-11:00 AM PST
📍 San Francisco + online · free

🔗 Session: osacon.io/sessions/2026/…

#ApacheSpark #SparkConnect #OSACon

The core issue is that their source system is both relational and a document database. All of the documents are stored as XML. Figuring out how to handle that in production was a headache, because it is very challenging for end users to parse XML or JSON. That is where Variant comes in.

Geir Alstad walks through this in a talk on Apache Spark Variant shredding in production.

🎥 Watch the full session: youtube.com/live/imEjS0UoZ…

#ApacheSpark