Vincent Ging Ho Yim

🇭🇰🇦🇺 • date engineering / data analytics • stenography (Plover) • logic puzzles (Pic-a-Pix, sudoku, etc.) • learning languages (🇯🇵🇰🇷) • he / him

[Vincent Ging Ho Yim @cenviity]

My DE Zoomcamp module 4 homework https://t.co/XAxrU4uFfU
SQLMesh https://t.co/TU9vuqB8Iw
Ibis https://t.co/hTeDcO83jl

[Vincent Ging Ho Yim @cenviity]

👨‍💻 Data Engineering Zoomcamp module 4 – analytics engineering with #dbt (courtesy of
@DataTalksClub)

🏃 After a couple of busy weeks at work I had fallen behind in the course, so it was time to catch up and work on some familiar territory – dbt!

#dezoomcamp #dataengineering

[Vincent Ging Ho Yim @cenviity]

🪽 Second is #Ibis, a Python library that brings a unified dataframe API against a range of backends such as Pandas, Polars, DuckDB and Spark. Imagine that – the same Python code to work with dataframes across all of these. Except no imagining needed, because Ibis is real.

[Vincent Ging Ho Yim @cenviity]

💬 Alrighty, I've got to actually make a start on module 4 on analytics engineering with #dbt now…

[Vincent Ging Ho Yim @cenviity]

but dlt makes it all so much easier and effortless!

🐤 An honourable mention to DuckDB as well, which is an in-memory database useful for quick experimentation during development and a whole lot more. No more hassle with spinning up a new database just to move some data around.

[Vincent Ging Ho Yim @cenviity]

AND I've actually been using dlt at work this past week to query data from an API for a current project.

🍝 There is a lot of code that would otherwise have been very tiresome and tedious to write when extracting and loading data, e.g. incremental loading and schema evolution…

[Vincent Ging Ho Yim @cenviity]

👨‍💻 Data Engineering Zoomcamp workshop – dlt
(courtesy of @DataTalksClub)
#dezoomcamp #learning #dataengineering

🎉 To say I'm enjoying using #dlt (from @dltHub) is an understatement. I've happily completed the homework (revision on Python generators and short bits of dlt code)…

[Vincent Ging Ho Yim @cenviity]

Looking forward a lot to module 4 on analytics engineering this coming week! It's… #dbt, very familiar ground from my current studies 😉

#dezoomcamp #learning #dataengineering

[Vincent Ging Ho Yim @cenviity]

Otherwise, it was all a bit of fun writing simple #SQL statements and queries to work with external and materialised tables in BigQuery. Oh, and I also had a refresher on partitioning vs clustering data (reminiscent of Snowflake studies last year).

[Vincent Ging Ho Yim @cenviity]

…for the homework this week, so instead I used a #Python script provided by DE Zoomcamp and modified it to suit my purposes. Once I have a bit more time I'll go back and see if I can get Mage to work (and also try building a similar data pipeline in #dlt too).

[Vincent Ging Ho Yim @cenviity]

👨‍💻 Data Engineering Zoomcamp module 3 – data warehouse

(courtesy of DataTalksClub)

This week was all about data warehouses and in particular Google #BigQuery. Unfortunately I had some issues creating a pipeline in Mage to load the Parquet files into a GCS bucket…

[Vincent Ging Ho Yim @cenviity]

I happened to have stumbled upon #dlt last month when working on a web scraping project, and it does look to be an interesting tool on the data integration front.

[Vincent Ging Ho Yim @cenviity]

I enjoyed getting in some practice on #Docker and #Terraform, coding with Mage in a container and writing configuration files to help me manage GCS infrastructure in a predictable way.

[Vincent Ging Ho Yim @cenviity]

I'm looking forward to Module 3 on data warehousing and #BigQuery in the coming week. Oh, there's also a workshop livestream from dltHub tonight (though unfortunately at 4 am AEDT, so I'll just have to watch the recording afterwards – https://t.co/60myozzMZX).