Apache Iceberg Data Lakehouse Tips

Unofficial account tweeting content on working with Apache Iceberg Data Lakehouses

[Apache Iceberg Data Lakehouse Tips @IcebergDataLake]

Experience how easy it is to take data from your source data systems, ingest them into Apache Iceberg and serve a BI dashboard from the confines of your laptop with these tutorials.

#DataLakehouse #DataLake #DataEngineering #ApacheIceberg

[Apache Iceberg Data Lakehouse Tips @IcebergDataLake]

Know Someone Learning Data Engineering, share this with them:

Hands-on with Apache Iceberg on Your Laptop: Deep Dive with Apache Spark, Nessie, Minio, Dremio… https://t.co/QltefURtV3

#DataEngineering

[Apache Iceberg Data Lakehouse Tips @IcebergDataLake]

ICEBERG METADATA TABLES

This article will walk you through a hands-on exercise to get familiar with the Iceberg metadata tables.

Read here: https://t.co/32q9KUqKhh

#DataEngineering #ApacheIceberg #DataLakehouse

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

APACHE ICEBERG MIGRATION GUIDE

https://t.co/Or8yRMuqHn

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

DREMIO.ICEBERG.DBT.NESSIE.MINIO.POSTGRES.MONGODB

If you want to try a deep end-to-end tutorial that will get you hands-on with a variety of popular data tools, try this one out.

https://t.co/jsZFBLMJ97

#DataLakehouse #DataEngineering #ApacheIceberg #Dremio #dbt #DataScience #DataAnalytics

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

RECENT DATA ARCHITECTURE/ENGINEERING/ANALYTICS CONTENT

— Apache Iceberg —

> What is Data Lakehouse Table Format?
https://t.co/Er7POv8eR4

> Comparing Iceberg to Other Lakehouse Solutions
https://t.co/b6azd3f7y2

> Iceberg Migration Guide
https://t.co/zjaiSZBWuP

> Hands-on with Managed Polaris Catalog
https://t.co/wf9v1QkE43

> Hands-on with Self-Managed Polaris
https://t.co/DpXVIKr9oQ

— Hybrid Lakehouse —

> 3 Dremio Use Cases for On-Prem Data Lakes
https://t.co/ys6H1nJ5F4

> Hybrid Lakehouse Solution: NetApp
https://t.co/hDXCtLy5Ac

> Hybrid Lakehouse Solution: Minio
https://t.co/VJzxKYDXWV

> Hybrid Lakehouse Solution: Vast Data
https://t.co/jNEwtQzELj

> Hybrid Lakehouse Solution: Pure Storage
https://t.co/lm4EIiZjGa

— Unified Analytics —

> Analysts Guide to JDBC/ODBC, REST, and Arrow Flight
https://t.co/Ww1ZN6tZ97

> Unified Lakehouse
https://t.co/ey85YBC45K

#DataEngineering #DataLakehouse #DataScience #DataAnalytics #DataArchitecture

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

HOW ICEBERG CATALOGS WORK

Iceberg tables are one part data stored in several parquet files and a second part metadata files that provide context and understanding of that data as a singular table.

The metadata entry point is a file called metadata.json which tracks the tables schemas, partition schemes and snapshots. Everytime the table changes a new metadata.json is created.

So when there is possibly dozens or hundreds of these metadata.json files, how does an engine like Dremio, Snowflake or Apache Spark know which is the right one to query the table accurately. This is where a catalog comes in like Nessie and Polaris.

A catalog acts like a traffic controller maintaining a list of tables along with the file address where the current metadata.json is stored. These references are updated at the end of a transaction after the new metadata.json is created enabling Atomicity guarantees.

A catalog directs queries to the right metadata.json and updates that list when writes are complete.

If you enjoyed this post, give it a like and a share! Also check out https://t.co/EfSOHlh2PV for a lot more Apache Iceberg education resources.

#ApacheIceberg #DataLakehouse #DataEngineering

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

OPTIMIZING ICEBERG TABLES

One the things that make Iceberg queries fast is that the metadata can be used eliminate files that don’t need scanning from the scan plan. This is great but if the data is not clustered properly or spread out across many small files, you can still see less than ideal performance.

** Compaction **

When you have more manifests and data files than you need, you are doing more file operations and slowing down performance. By rewriting these files so you can collapse the data into fewer larger files you have the opposite effect. This can be done the REWRITE_DATA_FILES or REWRITE_MANIFESTS procedures in Spark or the OPTIMIZE TABlE command in Dremio.

** Clustering **

If I only am searching for agent in the northwest region, it’d be nice if all those reps where in the same few files, this is known as clustering. When rewriting data files with Spark, there is a “sort” parameter you can pass so it can cluster the data as it rewrites the files.

By compacting and clustering you data, the Apache Iceberg metadata becomes even more powerful in skipping data files when executing queries.

Read more in my new article on maintaining Apache Iceberg lakehouses here:

https://t.co/jJ5epcl6ST

#DataLakehouse #ApacheIceberg #DataEngineering

[MinIO @Minio]

Join us on September 5th at 10am PT for a MinIO x @dremio x @Carahsoft webinar about how modern #datalakes can help government customers solve their modernization initiatives. Register here: https://t.co/Y80uY8zyur

[Dremio @dremio]

Join us for "An Apache Iceberg Lakehouse Crash Course" an in-depth series designed to provide a comprehensive understanding of Apache Iceberg, taught by Iceberg expert Alex Merced.

https://t.co/4YiSfUTVXN

[Apache Iceberg Data Lakehouse Tips @IcebergDataLake]

Basics of Lakehouse Engineering - Apache Iceberg, Nessie (2 hour Course)

https://t.co/3JkRfI9g8V

#DataEngineering #Nessie #Dremio #ApacheIceberg

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

Do you use an open table format? If so how’s your experience been, vote, reply and share!

#ApacheIcberg #DeltaLake #ApacheHudi #DataEngineering #DataLakehouse

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

Open Tables (Apache Iceberg) + Open (Nessie, Polaris, Gravitino) Catalogs = No Vendor Lock-in Lakehouses

Read More: https://t.co/q67i8pHbK2

#DataEngineering #DataLakehouse #DataLake

@dremio @SnowflakeDB @ApacheIceberg

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

NEW MEDIA ON OPEN SOURCE APACHE ICEBERG CATALOGS

- New Episode of "Datanation" which you can find on iTunes and Spotify

- Substack: https://t.co/9Mq7j0VucC

#ApacheIceberg #DataLakehouse #Dremio #Snowflake #Databricks #DataEngineering

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

DATA PROFESSIONAL FOLLOW TRAIN

- reply to this tweet, I will follow you
- follow me and everyone else who replies
- retweet to maximize reach of the train

#DataEngineering #DataAnalytics #DataScience #BigData #DataLakehouse #DataLake

[Apache Iceberg Data Lakehouse Tips @IcebergDataLake]

@AMdatalakehouse Follow me

[Dremio @dremio]

🎙️ Dive into the minds of data disruptors! 🚀 Join us on the #DataDisruptors podcast as we unravel the strategies and insights shaping the future of data leadership. Tune in for exclusive conversations that redefine the data landscape. Listen now!

🔗 https://t.co/3oLtvDojOQ

[Antony Savvas @AntonySavvas]

#Dremio paddles harder in the growing #lakehouse market with product improvements https://t.co/3vHq19YdfG via @BlocksandFiles #AI #OpenSource

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

WHAT IS A DATA LAKEHOUSE?

#DataEngineering #DataAnalytics #BigData #ApacheIceberg

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

8-BIT GAME TRAILER: Quest for the Data Lakehouse

#DataLakehouse #DataEngineering #ApacheIceberg #Dremio

[Alex Merced | Open Data Lakehouse Advocate @AMdatalakehouse]

FREE YOUR DATA AND GET A FREE BOOK

Watch to learn more

#DataEngineering #DataAnalytics #DataLakehouse #DataLake #BigData