Anthony Dalke

Data practitioner both thrilled and terrified by AI. Let’s talk about it. Oh, also a White Sox and Notre Dame fan. 🇻🇦

[Anthony Dalke @anthony_dalke]

Horrible news. The first two ME games remain some of my most cherished gaming experiences. 😔

[Ryan McCaffrey @DMC_Ryan]

I sadly can't say I'm surprised after the venture-capital buyout, but I'm still devastated. RIP BioWare 😢 https://t.co/z0FXIgrkpE

[Anthony Dalke @anthony_dalke]

@0xSero Love it! Are you Catholic?

[Anthony Dalke @anthony_dalke]

Who’s spending more money: Anthropic on compute, or all the gambling/prediction market sites on advertising? 🤔

[Anthony Dalke @anthony_dalke]

This is so true. I use LLM’s to recommend books all the time. 👍

[0xSero @0xSero]

Using LLMs as a sort of recommendation algo is one of the best things ever. I find so much new music now it's wonderful.

I find more books too, I just tell it to look at my browser/youtube/goodreads and have it make checklists, i rate them often.

it gets better

[Anthony Dalke @anthony_dalke]

Seeing these eye-popping Anthropic financials reminds me how badly I’d struggle as a VC. How does anyone reliably and predictably come up with these valuations? 🤔

[Anthony Dalke @anthony_dalke]

@0xSero Ugh, I’m sorry I got confused! Do you come to Chicago often? If so, I’ll do a better job of getting my days straight next time! 😬

[Anthony Dalke @anthony_dalke]

This reminds me of a point @PatrickDeneen made in Why Liberalism Failed: although liberalism promises individuals independence from the state, the resulting disorder often leads to even *greater* state control.

[Mark W. @DurhamWASP]

“By destroying traditional social habits of the people… Liberalism can prepare the way for that which is its own negation: the artificial, mechanised or brutalised control which is a desperate remedy for its chaos.”

T.S. Eliot

[Anthony Dalke @anthony_dalke]

@0xSero Oh wait a minute, I think I misread your post as you’ll be in Chicago until *tomorrow*, the 21st! Are you only here until this evening? I worked from home today, and I live in the suburbs! 😩

[Anthony Dalke @anthony_dalke]

@0xSero Okay, so you’re at O’Hare, I assume? I work downtown. I’d love to meet you (I think we could talk a lot about faith and fatherhood, for instance), but I’d hate to make you travel all the way into the city! 😬

[Anthony Dalke @anthony_dalke]

Reading this post and the comments reminds me of the joy this website can bring ☺️

[DHH @dhh]

I normally listen to electronic music with few or repeating lyrics when needing to focus, but I've found that a looping Gregorian chant also does the trick! https://t.co/0LHzghBX54

[Anthony Dalke @anthony_dalke]

@dhh On behalf of Catholics everywhere, I welcome you to join us! 😏

[Anthony Dalke @anthony_dalke]

As someone who didn’t grow up *loving* tech per se, I don’t often feel a sense of wonder from a technology.

However, agents stand as an unavoidable exception, where even relatively simple aspects leave me feeling 🤯. Progressive disclosure is one example!

[Anthony Dalke @anthony_dalke]

I follow a strict mantra when working at a desk: A.B.C. - Always Be Charging (my phone, AirPods case, or watch)

[Anthony Dalke @anthony_dalke]

@SoxMach_pnoles Wow, can’t remember the last time I heard or saw a “Paco” Martin reference! 😂

[Anthony Dalke @anthony_dalke]

@parkerbrowndev Fascinating, thank you for replying!

[Anthony Dalke @anthony_dalke]

@parkerbrowndev LOL! Well, do you have Omarchy on your phone? And Android, rather than iOS?

[Anthony Dalke @anthony_dalke]

@0xSero If you don’t mind my asking, are you Catholic?

[Anthony Dalke @anthony_dalke]

Bookmarking for future reference… 👀

[Zach Wilson @Zachly]

Turning a vibe-coded data pipeline into production-grade software requires thinking about 100s of different engineering concepts!

AI can generate an ETL pipeline in minutes.

But getting that pipeline to process billions of records reliably, maintain data quality, and recover from failures? That’s a completely different animal!

Here’s what vibe coders building data pipelines need to learn:

1. Data Ingestion & Extraction

* Batch vs. streaming ingestion
* Change Data Capture (CDC)
* Full vs. incremental loads
* API rate limiting
* Schema inference
* Data serialization
* Idempotent ingestion
* Late-arriving data

2. Data Modeling & Storage

* Star schemas
* Snowflake schemas
* Slowly Changing Dimensions (SCDs)
* Fact and dimension tables
* Data normalization
* Partitioning
* Clustering
* File compaction
* Columnar storage
* Data lakehouse architectures

3. Distributed Data Processing

* Data shuffling
* Data skew
* Broadcast joins
* Sort-merge joins
* Predicate pushdown
* Partition pruning
* Parallelism
* Memory management
* Adaptive query execution

4. Orchestration & Reliability

* DAG orchestration
* Task dependencies
* Retries
* Exponential backoff
* Idempotency
* Backfills
* SLAs
* Data freshness
* Failure recovery
* Workflow scheduling

5. Streaming & Real-Time Data

* Event time vs. processing time
* Watermarks
* Windowing
* Exactly-once vs. at-least-once processing
* Event deduplication
* Out-of-order events
* Checkpointing
* State management
* Kafka partitions
* Backpressure

6. Data Quality & Governance

* Data contracts
* Schema evolution
* Schema enforcement
* Null handling
* Duplicate detection
* Referential integrity
* Data validation
* Data lineage
* PII handling
* Access controls
* GDPR compliance

7. Performance & Cost Optimization

* Warehouse sizing
* Autoscaling
* Query caching
* Materialized views
* Incremental materialization
* Storage optimization
* Compute optimization
* Cost attribution
* Concurrency management
* Query profiling

8. Deployment & Observability

* CI/CD for data pipelines
* Infrastructure as Code
* Environment isolation
* Unit testing
* Integration testing
* Data reconciliation
* Pipeline monitoring
* Data quality alerts
* Freshness monitoring

AI has reduced the cost of writing data transformations!

But it hasn’t eliminated the complexity of running data pipelines at scale.

You can easily vibe code your way to a pipeline that works on 10,000 rows.

But when that pipeline needs to process 10 billion rows, handle schema changes, recover from partial failures, or guarantee downstream data correctness. You’re basically cooked！

The future isn’t data engineers vs. vibe coders.

It’s data engineers who understand distributed systems, data architecture, and production reliability AND know how to leverage AI.

Writing SQL and Python is getting cheaper.

Data engineering judgment is becoming more valuable！

What would you add to the list?

Inspired by Natalie L.