Joseph Machado

Data stuff

Free Data Engineering Fundamentals Course:

[Joseph Machado @startdataeng]

Exercise project for anyone starting in data engineering https://t.co/F3GegLjIrH
#dataengineering #bigdata #ETL #ApacheAirflow #AWS #ApacheSpark

[Joseph Machado @startdataeng]

LLM code generation dramatically speeds up development until it doesn’t. If you’ve worked with AI-generated code, you’ve likely seen:

* Bugs are buried in AI spaghetti code that takes longer to fix than it does to write it yourself.
* Code that does something completely different than what you wanted

This post shows you how to prevent that. You’ll learn to specify requirements, so LLMs generate exactly what you need. And avoid wading through thousands of lines of code to verify correctness.

https://t.co/OhxiplpSpS

#dataengineering #llm #codegeneration

[Joseph Machado @startdataeng]

AI makes some reasonable guesses, but always verify the results.

I’m working on an article to streamline pipeline creation. We still need humans in the loop.

The image below recommends an ID-based delete-insert for a fact table, which will take forever to run if your data is large.

Add these to your skills.md file.

#dataengineering #llm

[Joseph Machado @startdataeng]

“I recommend this to everyone on linkedin asking "what to study" for data engineering. because the topics are so fundamental and work with literally every system (databricks, snowflake, redshift, bigquery, etc)”

- One of the comments I got about my Design Patterns workshop

Check it out here

https://t.co/ej6C8ajEAd

#dataengineering #pipelinedesign

[Joseph Machado @startdataeng]

@Tech_p001 Fair question.

We need to review as we would human generated code.

[Joseph Machado @startdataeng]

Working on a project that teaches you how to build data pipelines following standard design practices and speeding up development with agent harness & skills!

#dataengineering #dataproject

[Joseph Machado @startdataeng]

There are a lot of data projects available on the web. While these projects are great, starting from scratch to build your data project can be challenging.

If you are wondering how to go from an idea to a production-ready data pipeline

This post is for you! In it, we will go over how to build a data project step-by-step from scratch.

By the end of this post, you will be able to quickly create data projects for any use case and see how the different parts of data systems work together.

https://t.co/cvM3yqT1qB
#dataengineering #datapipeline

[Joseph Machado @startdataeng]

I’ve spent way too much time thinking about my brand colors.

Ultimately, I decided to go with what would make my readers' lives easier.

A bright screen makes it hard to concentrate for long intervals.

So I switched my website to use the Gruvbox theme.

Does it help?

Check it out at https://t.co/X8z1yvvZbJ

[Joseph Machado @startdataeng]

In the past few years, data modeling has gone from “well-planned data model” to “dump everything in S3 and get some data for the end-user”.

How do you ensure data correctness when data is expected immediately, and there isn't enough time to get the model right?

For solutions, read this

https://t.co/Itmf8mx1hq

#dataengineering #datamodel

[Joseph Machado @startdataeng]

https://t.co/crULNFTiv8

[Joseph Machado @startdataeng]

Updating my “Python Essentials for Data Engineers” post and came across this amazing data quality tool.

pointblank from posit.

Easy to set up and use, as any tool should be. No need to wrestle with 100s of configs/settings.

#dataengineering #python #dataquality

[Joseph Machado @startdataeng]

One of the biggest causes of data completeness issues is not knowing whether your input datasets are complete.

If you schedule pipelines guess when your upstream data may be ready, its hope driven.

Learn how to use data asset-based triggers & start your pipelines when the inputs are ready.

https://t.co/cahxTYWlgA

#dataengineering #datapipeline #apacheairflow

[Joseph Machado @startdataeng]

Poll:

[Joseph Machado @startdataeng]

Which one should I use; Gruvbox or standard?

Gruvbox is easier on the eyes

But the standard theme (Bootswatch, Flatly) is familiar

Let me know in the poll below 👇

#ux #readability

[Joseph Machado @startdataeng]

There are many data projects available online. While these projects are great, starting from scratch to build your data project can be challenging.

If you are

* Wondering how to go from an idea to a production-ready data pipeline

* Feeling overwhelmed by how all the parts of a data system fit together

* Unsure whether the pipelines you build are up to industry-standard

If so, this post is for you! In it, we will go over how to build a data project from scratch, step by step.

By the end of this post, you will be able to quickly create data projects for any use case and see how the different parts of data systems work together.

https://t.co/cvM3yqT1qB

#dataengineering #dataprojects

[Joseph Machado @startdataeng]

Using AI for development? Go read @htmx_org ‘ s post.

https://t.co/EnZb4l1RbD

Take notes.

My experience closely resembles his when building data stuff.

1. AI is good at debugging (only if we have sufficient logs & trace).
2. AI is great at creating tests (although I usually have to specify additional test cases).
3. AI will “sometimes” lead you down a complexity trap when implementing new features.

When designing new features/interfaces, I always
* Create my own plan with a clear objective and pseudocode
* Then review it with AI and iterate.

I found this to be much more time efficient and less burnout-y

#dataengineering #aidevelopment

[Joseph Machado @startdataeng]

When choosing a tool/framework, always pick the one with the simplest implementation.

But that could be extended with libraries, Plugins, Ecosystem, etc.

Here are some of my favorite tools

1. DuckDB: SQL, enhanced with extensions
2. Python: English-like syntax, enhanced with libraries
3. Quarto: Markdown-like syntax, enhanced with extensions
4. nvim: vim syntax, enhanced with extensions

The fancier/esoteric the tool, the harder it is to get productive quickly.

What are some of your favorite tools?

#tools #dataengineering

[Joseph Machado @startdataeng]

This is part of my free data design pattern workshop (next Sat), where I cover this with exercises & arguments.

Come join me, sign up here to get notified: https://t.co/MXSMSQulw8.

[Joseph Machado @startdataeng]

90% of all facts and dimensions follow a similar design pattern, with some differences in business logic.

This messy middle (fact & dims) is where DEs can lay the foundations for reliable data.

If you mess this up, your team will always face issues/questions/mismatches, etc.

Here is a V1 of this design

[Joseph Machado @startdataeng]

Whenever I design a workshop, I use @robfitz ’s “The Workshop Survival Guide”.

I see how great user design increases engagement, recommendations, and leads to higher ratings.

Here is a very rough version of a workshop that I am working on (coming soon).

Focus on

1. Takeaways for the audience. Not just watch and forget.
2. Increase play. Interweave code exercises and visuals to help understanding.

#dataengineering #userexperience