Harry Lee

Thinker | Tinker | Engineer

[Aurimas Griciūnas @Aurimas_Gr]

What is the role of 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁𝘀 in the Data Pipeline?

In its simplest form Data Contract is an agreement between Data Producers and Data Consumers on what the Data being produced should look like, what SLAs it should meet and the semantics of it.

𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁 𝘀𝗵𝗼𝘂𝗹𝗱 𝗵𝗼𝗹𝗱 𝘁𝗵𝗲 𝗳𝗼𝗹𝗹𝗼𝘄𝗶𝗻𝗴 𝗻𝗼𝗻-𝗲𝘅𝗵𝗮𝘂𝘀𝘁𝗶𝘃𝗲 𝗹𝗶𝘀𝘁 𝗼𝗳 𝗺𝗲𝘁𝗮𝗱𝗮𝘁𝗮:

👉 Schema of the Data being Produced.
👉 Schema Version - Data Sources evolve, Producers have to ensure that it is possible to detect and react to schema changes. Consumers should be able to process Data with the old Schema.
👉 SLA metadata - Quality: is it meant for Production use? How late can the data arrive? How many missing values could be expected for certain fields in a given time period?
👉 Semantics - what entity does a given Data Point represent. Semantics, similar to schema, can evolve over time.
👉 Lineage - Data Owners, Intended Consumers.
👉 …

𝗦𝗼𝗺𝗲 𝗣𝘂𝗿𝗽𝗼𝘀𝗲𝘀 𝗼𝗳 𝗗𝗮𝘁𝗮 𝗖𝗼𝗻𝘁𝗿𝗮𝗰𝘁𝘀:

➡️ Ensure Quality of Data in the Downstream Systems.
➡️ Prevent Data Processing Pipelines from unexpected outages.
➡️ Enforce Ownership of produced data closer to where it was generated.
➡️ Improve scalability of your Data Systems.
➡️ Reduce intermediate Data Handover Layer.
➡️ …

Example implementation for Data Contract Enforcement:

1️⃣ Schema changes are implemented in a git repository, once approved - they are pushed to the Applications generating the Data and a central Schema Registry.
2️⃣ Applications push generated Data to Kafka Topics. Separate Raw Data Topics for CDC streams and Direct emission.
3️⃣ A Flink Application(s) consumes Data from Raw Data streams and validates it against schemas in the Schema Registry.
4️⃣ Data that does not meet the contract is pushed to Dead Letter Topic.
5️⃣ Data that meets the contract is pushed to Validated Data Topic.
6️⃣ Applications that need Real Time Data consume it directly from Validated Data Topic or its derivatives.
7️⃣ Data from the Validated Data Topic is pushed to object storage for additional Validation.
8️⃣ On a schedule Data in the Object Storage is validated against additional SLAs and is pushed to the Data Warehouse to be Transformed and Modeled for Analytical purposes.  
9️⃣ Consumers and Producers are alerted to any SLA breaches.
🔟 Data that was Invalidated in Real Time is consumed by Flink Applications that alert on invalid schemas. There could be a recovery Flink App with logic on how to fix invalidated Data.

Leave your thoughts in the comment section 👇

--------

Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

Also hit 🔔to stay notified about new content.
𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁!

Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: https://t.co/qgNCnGtF4A

[Aurimas Griciūnas @Aurimas_Gr]

𝗪𝗵𝗮𝘁 𝗶𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 and why should you 𝗹𝗲𝗮𝗿𝗻 𝗶𝘁 𝗮𝘀 𝗠𝗟𝗢𝗽𝘀/𝗠𝗟/𝗗𝗮𝘁𝗮 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿?

Today we look into the Kubernetes system from a bird's eye view.

𝗦𝗼, 𝘄𝗵𝗮𝘁 𝗶𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 (𝗞𝟴𝘀)?

𝟭: It is a container orchestrator that performs the scheduling, running and recovery of your containerised applications in a horizontally scalable and self-healing way.

Kubernetes architecture consists of two main logical groups:

𝟮: Control plane - this is where K8s system processes that are responsible for scheduling workloads defined by you and keeping the system healthy live.
𝟯: Worker nodes - this is where containers are scheduled and run.

𝗛𝗼𝘄 𝗱𝗼𝗲𝘀 𝗞𝘂𝗯𝗲𝗿𝗻𝗲𝘁𝗲𝘀 𝗵𝗲𝗹𝗽 𝘆𝗼𝘂?

𝟰: You can have thousands of Nodes (usually you only need tens of them) in your K8s cluster, each of them can host multiple containers. Nodes can be added or removed from the cluster as needed. This enables unrivaled horizontal scalability.
𝟱: Kubernetes provides an easy to use and understand declarative interface to deploy applications. Your application deployment definition can be described in yaml, submitted to the cluster and the system will take care that the desired state of the application is always up to date.
𝟲: Users are empowered to create and own their application architecture in boundaries pre-defined by Cluster Administrators.

✅ In most cases you can deploy multiple types of ML Applications into a single cluster, you don’t need to care about which server to deploy to - K8s will take care of it.
✅ You can request different amounts of dedicated machine resources per application.
✅ If your application goes down - K8s will make sure that a desired number of replicas is always alive.
✅ You can roll out new versions of the running application using multiple strategies - K8s will safely do it for you.
✅ You can expose your ML Services for other Product Apps to use with few intuitive resource definitions.
✅ …

❗️Having said this, while it is a bliss to use, usually the operation of Kubernetes clusters is what is feared. It is a complex system.
❗️Master Plane is an overhead, you need it even if you want to deploy a single small application.

Are you deploying your ML applications in Kubernetes? What are the main pain points you are facing? Let me know in the comments 👇

--------

Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

Also hit 🔔to stay notified about new content.
𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁!

Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: https://t.co/qgNCnGtF4A

[Harry Lee @harryleesan]

Verifying myself: I am harryleesan on https://t.co/q1O5BcWnce. NUr_d9ktetUhwJaWLEUWq8bxzSyIXNuK6RY1 / https://t.co/LCogAOi3Tq

[Marques Brownlee @MKBHD]

You're telling me this is just a thing that can happen to you on the road out of nowhere? 🫠

[Dio Rettori @rettori]

Cloud indeed is the best thing at making technical debt be actual debt.

[Alex Xu @alexxubyte]

/1 Why is a credit card called a “𝐜𝐫𝐞𝐝𝐢𝐭” card?

Why is a debit card called a “𝐝𝐞𝐛𝐢𝐭” card?

An example of a debit card payment is shown in the diagram below. 👇

[Harry Lee @harryleesan]

Came across this great cli tool to visualize nodes and costs of EKS! https://t.co/BRlyPHAgA0

[LearnKube @learnk8s]

In this article, you'll learn how to use locust, Helm and Kustomize to make load testing easy at a scale

➜ https://t.co/Gx9wrn673y

[Harry Lee @harryleesan]

Likewise! It was great to finally meet @danieloh30 in person! Too bad didn’t get to meet more Ambassadors! @CNCFAmbassadors @KubeCon_

[Daniel Oh @danieloh30]

Finally, glad to meet you @harryleesan @CNCFAmbassadors @KubeCon_

[Kubesploit @kubesploit]

In this article, you will learn about Kubernetes security and architecture by reviewing reports from Chekov — a tool designed to find misconfigurations before they’re deployed

➤ https://t.co/I7bDyIBDw1

[LearnKube @learnk8s]

Talos is a modern Linux distribution for running Kubernetes: secure, immutable, and minimal

Talos is fully open source & production-ready

All system management is done via an API - there is no shell or interactive console

➜ https://t.co/d5s7Q3cekM

[Harry Lee @harryleesan]

@gbelrose Drunk kit sounds much more fun!

[Harry Lee @harryleesan]

Good to start my Saturday morning with some Elasticsearch Index Lifecycle Managment

[Kubesploit @kubesploit]

In this article, you will learn why PodSecurityPolicies never made it as a GA feature, why they had to be replaced and what you should consider going forward

➤ https://t.co/k2z1kcUdFT

[Kube Architect @K8sArchitect]

Kubernetes Services are not capable of load balancing long-lived connections such as gRPC

In this article, you will learn how to properly load balance your gRPC applications in Kubernetes by leveraging a service mesh

➤ https://t.co/5Td8aBgJcc

[Kubesploit @kubesploit]

dexter is an OIDC (OpenId Connect) helper designed to create a hassle-free Kubernetes login experience powered by Google or Azure as Identity Provider

All you need is a properly configured Google or Azure client ID & secret

➤ https://t.co/4uca90IEvW

[LearnKube @learnk8s]

The AWS team released the ACK project, which allows users to create AWS services from within a Kubernetes cluster

In this tutorial, you will learn how to create an S3 bucket with a "Bucket" CRD

➜ https://t.co/T0Li3G92VV

[Harry Lee @harryleesan]

@GrindstoneXL Hughwind is the fastest way to search, buy and sell AI solutions https://t.co/bg2xiDIkVT

[AWS Open Source @AWSOpen]

New blog post: Easily Running Open Policy Agent Serverless with AWS Lambda and Amazon API Gateway
https://t.co/GjOHUKkvVK

[Harry Lee @harryleesan]

@gbelrose Some people just want to watch the world burn 😂