Apache Spark

Lightning-fast unified analytics engine

[Apache Spark @ApacheSpark]

Apache Spark 4.3 introduces stateful operation support in Real-Time Mode with sub-150ms P99 latency. State now persists in memory and updates in place rather than checkpointing on every trigger. Existing stateful operations work with minimal code changes when the trigger is set to real-time mode.

🎥 Watch: https://t.co/aSGc9swD1t

[Apache Spark @ApacheSpark]

Millisecond latency on Structured Streaming used to mean a second engine.

Next week at QCon San Francisco, Jerry Peng (@databricks) will show how RT brings that capability into Apache Spark itself: same SQL streaming model, lower latency path.

In this session:
🔸 What RTM changes under the hood
🔸 How low-latency workloads run without leaving Spark
🔸 Why that matters for consolidating batch, ETL, and real-time on one platform

🗓️ Monday, Nov 16
📍 San Francisco

🔗 Session details: https://t.co/cAjNDhS58e

[Apache Spark @ApacheSpark]

Apache Spark 4.3 is out now, and @lisancao walks through the latest updates. ⚡

This video covers stateful real-time mode, pipeline shuffle, AUTO CDC with SCD Type 2, Spark Connect operation IDs, DataFrame. zip, nested JSON in SQL, and more.

🎥 Watch the full overview: https://t.co/JvpYgjnXUw

#ApacheSpark

[Apache Spark @ApacheSpark]

➡️ Download Spark 4.3: https://t.co/4rwQFN2EHT

[Apache Spark @ApacheSpark]

Apache Spark 4.3 is here, bringing stateful real-time streaming to the engine you already use for batch, analytics, and AI.

⚡ Stateful Real-Time Mode: streaming aggregations, dropDuplicates, and transformWithState run with P99 end-to-end latency under 150 ms in reported benchmarks
🕰️ SCD Type 2 in Auto CDC: preserve history from Python, SQL, and Spark Connect
🪶 Project Feather: opt-in shuffle-free execution for eligible small queries, plus Arrow-native caching
🌐 Spark Connect: REST endpoints for sessions and operations, and a more complete JDBC driver
🤖 SQL: JSON_TABLE, VARIANT transformations, ASOF JOIN, and a preview of nanosecond timestamps
📅 Feature releases move to every three months instead of every six

Thank you to the 210+ contributors and the Apache Spark community behind 1,100+ Jira tickets.

Read more 👇
https://t.co/HeCDbzDByY

[Apache Spark @ApacheSpark]

Apache Spark Data Source V2 is Spark’s way to support native table formats, not only files.

Szehon Ho walks through why that exists. Early Spark read and wrote file formats such as Parquet and used a thin Hive Metastore catalog. Apache Iceberg and Delta Lake added a much richer metadata layer so engines can run large, transactional operations. DSV2 expands that surface so a data source can plug that metadata into Spark, including through catalogs.

🎥 https://t.co/qVGxuXp4ha

#ApacheSpark #OpenSource

[Apache Spark @ApacheSpark]

We are happy to announce the availability of Apache Spark 4.1.4! 🎉

Spark 4.1.4 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-4.1 maintenance branch of Spark. We strongly recommend all 4.1 users to upgrade to this
stable release.

To download Spark 4.1.4, head over to the download page: https://t.co/i96ON3BtmC

Spark 4.1.4 is also available on PyPI (pyspark, pyspark-connect, pyspark-client) and the Maven Central Repository.

To view the release notes:
https://t.co/BMGZowuYdL

We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you.

[Apache Spark @ApacheSpark]

Agents are writing a ton of Spark code these days. Most of it is error-ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air.

At OSA Con 2026, @lisancao (@databricks) will cover declarative pipelines so structural mistakes get caught before data moves, Spark Connect so session context and credentials stay out of the agent’s hands, and skills and prompts that make one-shots more successful. She’ll also cover the tradeoffs between model choice, tokens, and deployment infrastructure.

🗓️ Monday, November 2
🕓 10:35-11:00 AM PST
📍 San Francisco + online · free

🔗 Session: https://t.co/kXcj0qF3IK

#ApacheSpark #SparkConnect #OSACon

[Apache Spark @ApacheSpark]

The core issue is that their source system is both relational and a document database. All of the documents are stored as XML. Figuring out how to handle that in production was a headache, because it is very challenging for end users to parse XML or JSON. That is where Variant comes in.

Geir Alstad walks through this in a talk on Apache Spark Variant shredding in production.

🎥 Watch the full session: https://t.co/dnNibkA08f

#ApacheSpark

[Apache Spark @ApacheSpark]

Agents don’t ask one question a day. They ask 10 or 100, against the same data stack and the same SLA.

At Open Lakehouse + AI in Bellevue, Felix Cheung (@nvidia) walks through GPU-accelerated Apache Spark with no code change, plus @DeltaLakeOSS, @ApacheIceberg, and Spark Connect.

🎥 Watch: https://t.co/2LisokNFt4

#OpenLakehouse #ApacheSpark

[Apache Spark @ApacheSpark]

Apache Spark can analyze your query plans and derive the dependencies for you. You don’t need to declare them anymore.

@lisancao sits down with Andreas Neumann on Spark Declarative Pipelines. Spark owns the graph, manages session lifecycle and writes, while @ApacheAirflow orchestrates Spark as an application.

🎥 Watch the full conversation:
https://t.co/foQ0jEp2FQ

#ApacheSpark #ApacheAirflow #DataEngineering

[Apache Spark @ApacheSpark]

In Apache Spark 4.2, retrieval (NEAREST BY), governed semantics (metric views), and agent access (Spark Connect) run inside the engine, next to the data and under the same permissions, instead of in separate services.

Download Spark 4.2 👉 https://t.co/i96ON3AVx4

#ApacheSpark #AI #GenAI

[Apache Spark @ApacheSpark]

Felix Cheung (Apache Spark PMC, @nvidia) and @lisancao on running Spark SQL and DataFrames on GPUs with NVIDIA’s cuDF plugin, no job rewrites.

Snapchat’s A/B experimentation pipeline: 10 PB of new data a day, processed with Spark. On GPUs, 70-90% lower cost.

Also Spark Connect, including how a Spark Connect Gateway can sit in front of multiple servers, GPU ones included.

🎥 Full talk: https://t.co/yErLFO7k2Q

#ApacheSpark #SparkConnect #cuDF

[Apache Spark @ApacheSpark]

Spark SQL picks up several additions in Apache Spark 4.2, out now: SET PATH for object resolution, scripting cursors, QUALIFY for window results, SYSTEM.BUILTIN qualification, plus Theta sketches, time_bucket, and top-K max_by/min_by. More analytical work stays in SQL.

🔗 Download Spark 4.2: https://t.co/i96ON3AVx4

#ApacheSpark #SparkSQL #SQL

[Apache Spark @ApacheSpark]

Apache Spark 4.2 is out. Python Data Sources let you build batch or streaming readers and writers in Python, no JVM or Scala, used through the standard https://t.co/WrzhJbcsVb interface. Spark 4.2 adds time and memory profiling so they're easier to tune and operate.

🔗 Download: https://t.co/i96ON3AVx4

#ApacheSpark #PySpark #Python

[Apache Spark @ApacheSpark]

Auto CDC arrives in Spark Declarative Pipelines with Apache Spark 4.2: first-class SCD Type 1. Configure how change events update a target table, and Spark handles deletions and out-of-order events, in place of hand-written merge logic.

🔗 Download Spark 4.2: https://t.co/i96ON3AVx4

#ApacheSpark #CDC #DataPipelines #ETL

[Apache Spark @ApacheSpark]

.@ApacheAirflow is a popular orchestrator, but it often has little visibility when a Spark job fails.

At Airflow Summit 2026, @lisancao sat down with Andreas Neumann to discuss Spark Declarative Pipelines. Spark owns the pipeline graph, derives dependencies, and handles session lifecycle and writes, while Airflow orchestrates Spark as an application.

🎥 Watch the full conversation: https://t.co/foQ0jEp2FQ

#ApacheSpark #ApacheAirflow #DataEngineering

[Apache Spark @ApacheSpark]

Headed to #CommunityOverCode Glasgow? 🏴󠁧󠁢󠁳󠁣󠁴󠁿 Oct 11–14 features Apache Spark sessions on:

🔸 Resource auto-sizing
🔸 Spark 4.2 heterogeneous executors
🔸 CDC patterns
🔸 PySpark transpilation
🔸 Agent-native observability

🔗 See the schedule: https://t.co/f7wTGCz8gd

#ApacheSpark

[Apache Spark @ApacheSpark]

Grounding agents in trusted data is a theme in Apache Spark 4.2: NEAREST BY for retrieval, metric views for governed numbers, Spark Connect for access. Retrieval and metrics live next to the data, under the same permissions.

🔗 Download Spark 4.2: https://t.co/i96ON3AVx4

#ApacheSpark #AIAgents #RAG #GenAI

[Apache Spark @ApacheSpark]

Schema-on-Read Without Regrets: Lessons Learned from Apache Spark Variant Shredding in Production https://t.co/PDObI2CxcD