James Dzidek, PhD

AI/Data Platform Engineering with a focus on Dagster, Databricks, and LLMs.

[James Dzidek, PhD @jamesdzidek]

@aakashgupta After teaching a course to 100 engineers I found this model to be very useful to convey all the moving pieces in using LLMs. Source: https://t.co/gzaWBBNr6y

[James Dzidek, PhD @jamesdzidek]

@AlexNoonan6 @dagster Suggestion: rename "Code Location" to "Code Domain". I find that "Code Location" is too vague and imprecise for new adopters of Dagster.

[James Dzidek, PhD @jamesdzidek]

@AlexNoonan6 @grok what are the main 10 lessons from the books listed in this thread?

[James Dzidek, PhD @jamesdzidek]

@AlexNoonan6 Love the reference. :)

[James Dzidek, PhD @jamesdzidek]

Nice article by @AlexNoonan6 describing three fundamental data ingestion patterns: push, pull, and poll.

[Alex Nùnez @AlexNoonan6]

Most data engineering discourse focuses on transformations and fancy tools. But most of the time, the first thing is first.

I wrote a highly nutritious blog post about the three fundamental ingestion patterns every data engineer should know:
Push: When source systems deliver to you
Pull: When you control the extraction
Poll: When you need near real-time without full streaming

Each has trade-offs in terms of control, complexity, and operational burden. The post breaks down when to use each, with working @Dagster code examples you can actually run.

I have also built an example project, which you can use to see all three patterns in action.

[James Dzidek, PhD @jamesdzidek]

@AutismCapital Manual coffee grinder. Perfect companion for my Bialetti Moka.

[James Dzidek, PhD @jamesdzidek]

A good example of the perils of erroneous statistical analysis, a coding error in this case.

https://t.co/1o096g7w7M

[Adrian Colyer @adriancolyer]

"150 successful machine learning models: 6 lessons learned at https://t.co/krenlAFkFi' Bernadi et al., KDD'19 https://t.co/teXkqmLwYx #themorningpaper

Done right, ML-based projects can lead to some of the highest business returns.

[Itai Binyamin @itai_binyamin]

Today we’re announcing the general availability of the Mapping Data Flows feature of Azure Data Factory (ADF), our productive and trusted hybrid integration service.

#dataintegration #datamodeling #data #technology #thecloud #azure https://t.co/KfJbcvMPCk

[James Dzidek, PhD @jamesdzidek]

@SQLChick @CoatesDS Congratulations Melissa!