All Paid Courses (Free for First 4500 People)
๐ฃ๐ฎ๐ถ๐ฑ ๐๐ผ๐๐ฟ๐๐ฒ ๐๐ฅ๐๐ (PART - 1)
1. Artificial Intelligence
2. Machine Learning
3. Prompt Engineering
4. Claude,Chatgpt,Grok
5. Data Analytics
6. AWS Certified
7. Data Science
8. BIG DATA
9. Python
10. Ethical Hacking
(72 Hours only )
Like + RT + comment ' Drive '
Must Follow me so I can DM you.
China deployed a 1,300-ton data center on the ocean floor off the coast of Hainan.
No cooling towers or air conditioning. Just cold seawater doing the job for free.
The result? Cooling costs drop from ~50% of total energy to under 10%.
โThey were careless peopleโฆ they smashed up things and creatures and then retreated back into their money or their vast carelessness or whatever it was that kept them together, and let other people clean up the mess they had made.โ
- F. Scott Fitzgerald, The Great Gatsby
Marian Croak is the Inventor of the tech powering Zoom, FaceTime & WhatsApp.
She Holds 200+ patents and was Inducted into the National Inventors Hall of Fame. ๐
Today was the first meeting of the navigators and captains of the Session 3 of the Djangonaut Space ๐
I'm honored to be a navigator of the Team Saturn and to be in the company of so many great people ๐ซ
https://t.co/9X5rRASAKw
Look at the photo ๐
#Django#Djangonaut#Community
What are the most common ๐จ๐๐ฒ ๐๐ฎ๐๐ฒ๐ ๐ณ๐ผ๐ฟ ๐๐ฎ๐ณ๐ธ๐ฎ?
We have covered lots of concepts around Kafka already. But what are the most common use cases for The System that you are very likely to run into as a Data Engineer?
๐๐ฒ๐โ๐ ๐๐ฎ๐ธ๐ฒ ๐ฎ ๐ฐ๐น๐ผ๐๐ฒ๐ฟ ๐น๐ผ๐ผ๐ธ:
๐ช๐ฒ๐ฏ๐๐ถ๐๐ฒ ๐๐ฐ๐๐ถ๐๐ถ๐๐ ๐ง๐ฟ๐ฎ๐ฐ๐ธ๐ถ๐ป๐ด.
โก๏ธ The Original use case for Kafka by LinkedIn.
โก๏ธ Events happening in the website like page views, conversions etc. are sent via a Gateway and piped to Kafka Topics.
โก๏ธ These events are forwarded to the downstream Analytical systems or processed in Real Time.
โก๏ธ Kafka is used as an initial buffer as the Data amounts are usually big and Kafka guarantees no message loss due to its replication mechanisms.
๐๐ฎ๐๐ฎ๐ฏ๐ฎ๐๐ฒ ๐ฅ๐ฒ๐ฝ๐น๐ถ๐ฐ๐ฎ๐๐ถ๐ผ๐ป.
โก๏ธ Database Commit log is piped to a Kafka topic.
โก๏ธ The committed messages are executed against a new Database in the same order.
โก๏ธ Database replica is created.
๐๐ผ๐ด/๐ ๐ฒ๐๐ฟ๐ถ๐ฐ๐ ๐๐ด๐ด๐ฟ๐ฒ๐ด๐ฎ๐๐ถ๐ผ๐ป.
โก๏ธ Kafka is used for centralized Log and Metrics collection.
โก๏ธ Daemons like FluentD are deployed in servers or containers together with the Applications to be monitored.
โก๏ธ Applications send their Logs/Metrics to the Daemons.
โก๏ธ The Daemons pipe Logs/Metrics to a Kafka Topic.
โก๏ธ Logs/Metrics are delivered downstream to storages like ElasticSearch or InfluxDB for Log/Metrics discovery respectively.
โก๏ธ This is also how you would track your IoT Fleets.
๐ฆ๐๐ฟ๐ฒ๐ฎ๐บ ๐ฃ๐ฟ๐ผ๐ฐ๐ฒ๐๐๐ถ๐ป๐ด.
โก๏ธ This is usually coupled with ingestion mechanisms already covered.
โก๏ธ Instead of piping Data to a certain storage downstream we mount a Stream Processing Framework on top of Kafka Topics.
โก๏ธ The Data is filtered, enriched and then piped to the downstream systems to be further used according to the use case.
โก๏ธ This is also where one would be running Machine Learning Models embedded into a Stream Processing Application.
๐ ๐ฒ๐๐๐ฎ๐ด๐ถ๐ป๐ด.
โก๏ธ Kafka can be used as a replacement for more traditional messaging brokers like RabbitMQ.
โก๏ธ Kafka has better durability guarantees and is easier to configure for several separate Consumer Groups to consume from the same Topic.
โ๏ธHaving said this - always consider the complexity you are bringing with introduction of a Distributed System. Sometimes it is better to just use traditional frameworks.
--------
Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.
Also hit ๐to stay notified about new content.
๐๐ผ๐ปโ๐ ๐ณ๐ผ๐ฟ๐ด๐ฒ๐ ๐๐ผ ๐น๐ถ๐ธ๐ฒ ๐, ๐๐ต๐ฎ๐ฟ๐ฒ ๐ฎ๐ป๐ฑ ๐ฐ๐ผ๐บ๐บ๐ฒ๐ป๐!
Join a growing community of Data Professionals by subscribing to my ๐ก๐ฒ๐๐๐น๐ฒ๐๐๐ฒ๐ฟ: https://t.co/qgNCnGtF4A