Data Engineering Weekly

Weekly Data Engineering Newsletter. Subscribe to | Wanna talk about Data engineering? Book Me here

[Data Engineering Weekly @data_weekly]

Parquet is how columnar data sits on disk.
Arrow is how it sits in memory.

Engines that share Arrow pass data with zero copies, no serialization step.

From Hoyt Emerson's overview in DEW #292. Practice:
https://t.co/3GbNEQrjtO

#dataengineering

[Data Engineering Weekly @data_weekly]

eBay monitors 2.6B listings without slowing the fee path.

Write a tiny event ID to a local ring buffer. Add 250 columns of context later in ClickHouse.

Keep the hot path small.

https://t.co/lftWNIbLwb

#systemdesign

[Data Engineering Weekly @data_weekly]

Data modeling round, one decision per scenario:

SCD2: insert, never update
Org chart: manager_id points at employees
Data vault: hubs, links, satellites
Schema evolution: additive only

Practise on a canvas:
https://t.co/8YAb7TeOcR

#dataengineering

[Data Engineering Weekly @data_weekly]

Zalando ranks schema-mapping matches by:

ORDER BY approval_count DESC, similarity DESC

A reader's catch: a 0.81 match approved often beats a 0.97 approved once.

Flip it, or blend the two?

https://t.co/lftWNIbLwb

#sql

[Data Engineering Weekly @data_weekly]

LinkedIn moved Spark shuffle onto Apache Celeborn.

Fetch failures: ~750/day to ~20
Shuffle cost per task: down ~90%
Runtime: down ~14%

The senior Spark question: where does your shuffle data live?

https://t.co/sf8DuXl8Qz

#apachespark

[Data Engineering Weekly @data_weekly]

8 window-function problems, in order: running sum, LAG/LEAD, MoM growth, moving average, DENSE_RANK ties, QUALIFY, RANGE frames, sessions.

The trap: ROWS BETWEEN 6 PRECEDING is 7 rows, not 7 days.

https://t.co/EY7F9t6xdF

#sql #dataengineering

[Data Engineering Weekly @data_weekly]

Answer: no. Exactly day 7 means exactly, so day 6 and day 8 count for nothing. Join on activity_day = signup_date + 7, not a range, and the equality enforces "exactly" for free.

[Data Engineering Weekly @data_weekly]

Quick SQL quiz.

A user signs up, comes back on day 6, then on day 8.

Day-7 retained: yes or no?

Answer in the replies. Try it first, free:
https://t.co/DOEP0kofzp

#dataengineering #sql

[Data Engineering Weekly @data_weekly]

For engineers in India switching into data engineering:

All 200 https://t.co/DBTPZgYVS3 problems are free. SQL, Python, PySpark, hints, editorials. Plus 1 AI mock interview in week 1.

Practice for the rounds DSA prep skips.

https://t.co/EY7F9t6xdF

#dataengineering #sql

[Data Engineering Weekly @data_weekly]

@harsh_singh1507 @Azure @Microsoft Good timing on that one. Support for the Azure Functions in-process model ends Nov 10, 2026, so the isolated worker move is the part that can't wait.

[Data Engineering Weekly @data_weekly]

Ratings: 4.9, 4.8, 4.7, 4.7, 4.5

ROW_NUMBER: 1 2 3 4 5
RANK: 1 2 3 3 5
DENSE_RANK: 1 2 3 3 4

Ask for a "top 3" with RANK and you get 4 rows.

Top N per Group, free on https://t.co/DBTPZgYVS3:
https://t.co/99seO9THhE

#dataengineering #sql

[Data Engineering Weekly @data_weekly]

@sammiyoung01 Good pick. The Data Pipelines track on https://t.co/DBTPZgYVS3 opens with DAGs, sensors and idempotency, then has you debug real incidents like stuck sensors and retry storms. The beginner levels are free: https://t.co/X6N3vbEqrm Airflow or Dagster first?

[Data Engineering Weekly @data_weekly]

Want daily data engineering breakdowns, system architectures, and paper teardowns sent directly to your phone?

Join our new Data Engineering Weekly WhatsApp Channel! 🚀

👇 Join here: [https://t.co/WokOIzKb2K]

[Data Engineering Weekly @data_weekly]

Affirm processes billions in credit risk and checkout workflows every day.

How do they handle real-time ML inference without crashing checkout latency?

They built an isolated "Intelligence Layer." 🧵👇

[Data Engineering Weekly @data_weekly]

• Agentic & Multi-Model Scoring: Dynamically routing risk, fraud, and credit logic.

[Data Engineering Weekly @data_weekly]

💡 Key Takeaway:

GPU-accelerated analytics on Mac hardware is becoming plug-and-play. Registering physical optimizer rules allows high-performance Rust data frameworks like DataFusion to unlock massive speed gains effortlessly.

[Data Engineering Weekly @data_weekly]

🚀 Apache DataFusion can now leverage Apple Silicon GPUs for massive query speedups!

By registering a single optimizer rule via ArrowMetal, full SQL sorts run 6.9x to 28.8x faster on Apple Silicon—without changing a single line of SQL. 🧵👇

https://t.co/npQUf1jUpy

[Data Engineering Weekly @data_weekly]

🛡️ Smart Guardrails:

The rule evaluates warm/idle GPU costs and hand-back cases (e.g., complex float NaNs or streaming Top-K queries where CPU is better).

If conditions aren't optimal or memory limits trigger, it seamlessly falls back to standard CPU execution.