Sai Krishna Teja

Software Engineer

[Swapna Kumar Panda @swapnakpanda]

Best YouTube Playlists to learn Programming:

❯ C
https://t.co/rWsqnIGIE5

❯ C++
https://t.co/ivCdizORZ1

❯ Python
https://t.co/G1w3dCW558

❯ JavaScript
https://t.co/s8E5Vky2QR

❯ Java
https://t.co/yO1jNhS5iz

❯ C#
https://t.co/XvYD6WWQ3L

❯ SQL
https://t.co/C0SB4vsvaZ

❯ Go
https://t.co/yUENUeO81J

❯ PHP
https://t.co/PoHe5FQN7J

❯ Swift
https://t.co/8ztayGZ6r8

❯ Kotlin
https://t.co/AiL1hWk6R5

❯ Dart
https://t.co/uyi29sjuXJ

❯ Ruby
https://t.co/4clJ1JTzus

❯ Rust
https://t.co/yZWu2cgSUd

❯ TypeScript
https://t.co/Ls248WhenJ

❯ R
https://t.co/FAVMpexL6V

❯ React
https://t.co/pfxEUStSji

❯ Next.js
https://t.co/eqR2Wywvbw

❯ Node.js
https://t.co/8ZtStnDzf0

❯ HTML/CSS
https://t.co/cNlsiHV7L9

❯ Machine Learning
https://t.co/f4P4Ad0PCU

❯ Deep Learning
https://t.co/4UjJonuUXN

❯ DSA
https://t.co/lIHpLKDGj4

[Aurimas Griciūnas @Aurimas_Gr]

What do you need to know about 𝗖𝗗𝗖 (𝗖𝗵𝗮𝗻𝗴𝗲 𝗗𝗮𝘁𝗮 𝗖𝗮𝗽𝘁𝘂𝗿𝗲)?

Change Data Capture is a software process used to replicate actions performed against Operational Databases for use in downstream applications.

There are several use cases for CDC. Two of the main ones:

➡️ 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 𝗥𝗲𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻 (refer to 3️⃣ in the Diagram).

👉 𝗖𝗗𝗖 can be used for moving transactions performed against Source Database to a Target Database. If each transaction is replicated - it is possible to retain all ACID guarantees when performing replication.
👉 𝗥𝗲𝗮𝗹 𝘁𝗶𝗺𝗲 𝗖𝗗𝗖 is extremely valuable here as it enables Zero Downtime Source Database Replication and Migration. E.g It is extensively used when migrating on-prem Databases serving Critical Applications that can not be shut down for a moment to the cloud.

➡️ Facilitation of Data Movement from Operational Databases to Data Lakes (refer to 1️⃣ in the Diagram) or Data Warehouses (refer to 2️⃣ in the Diagram) for Analytics purposes.

👉 There are currently two Data movement patterns widely applied in the industry: 𝗘𝗧𝗟 𝗮𝗻𝗱 𝗘𝗟𝗧.
👉 In the case of ETL - data extracted by CDC can be transformed on the fly and eventually pushed to the Data Lake or Data Warehouse.
👉 In the case of ELT - Data is replicated to the Data Lake or Data Warehouse as is and Transformations performed inside of the System.

There is more than one way of how CDC can be implemented:

➡️ 𝗣𝘂𝗹𝗹 𝗕𝗮𝘀𝗲𝗱 𝗖𝗗𝗖

👉 A client queries the Source Database and pushes data into the Target Database.

❗️Downside 1: There is a need to augment all of the source tables to include indicators that a record has changed.
❗️Downside 2: Usually - not a real time CDC, it might be performed hourly, daily etc.
❗️Downside 3: Source Database suffers high load when CDC is being performed.
❗️Downside 4: It is extremely challenging to replicate Delete events.

➡️ 𝗣𝘂𝘀𝗵 𝗕𝗮𝘀𝗲𝗱 𝗖𝗗𝗖

👉 Triggers are set up in the Source Database. Whenever a change event happens in the Database - it is pushed to a target system.

❗️ Downside 1: This approach usually causes highest database load overhead.
✅ Upside 1: Real Time CDC.

➡️ 𝗟𝗼𝗴 𝗕𝗮𝘀𝗲𝗱 𝗖𝗗𝗖

👉 Transactional Databases have all of the events performed against the Database logged in the transaction log for recovery purposes.
👉 A Transaction Miner is mounted on top of the logs and pushes selected events into a Downstream System. Popular implementation - Debezium.

❗️ Downside 1: More complicated to set up.
❗️ Downside 2: Not all Databases will have open source connectors.
✅ Upside 1: Least load on the Database.
✅ Upside 2: Real Time CDC.

--------

Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space.

Also hit 🔔to stay notified about new content.
𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁!

Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: https://t.co/qgNCnGtF4A

[Phil Eaton @eatonphil]

Nice post. Hacking on databases and programming language standard libraries (especially containers and string routines), and trying to beat your own perf on benchmarks w/ ever larger data, is a great way to learn a lot of CS.

More fun than leetcode too!

https://t.co/e0dy8TZGIW

[Adam Boscarino @ajbosco]

This looks pretty cool. “WarpStream is an Apache Kafka protocol compatible platform built directly on top of S3. It’s asingle, stateless Go binary so there are no local disks to manage, no brokers to rebalance, and no ZooKeeper to operate.” https://t.co/cCCXH8WsRS

[Architecture Notes @arcnotes]

Database Sharding Explained https://t.co/TPkdXNRGCV

[Aleksey Charapko @AlekseyCharapko]

DistSys reading group spring term reading list: https://t.co/HD3c6jIXb8

We discuss one paper per week (almost) every Wednesday!

[Distributed Bytes @DistribSystems]

Evolution of search engines architecture - Algolia New Search Architecture Part 1
https://t.co/1L0DSLhDSW

[Michael Nitschinger @daschl]

I'm so happy and proud that this multi-year effort is finally coming together. Today we released @couchbase Server 7.0! https://t.co/4Rbx50OYst 🎉👑

[Couchbase @couchbase]

Couchbase Announces Closing of Initial Public Offering.
https://t.co/FKMRKF529q

[Couchbase @couchbase]

Permanent art installation needed NYC? We got you!

#CouchbaseIPO

[Sai Krishna Teja @saikrishnatejak]

@GreenplyPlywood Yes please elaborate how to indentify Domino print

[Shashidhar Reddy @Shashi_Reddy]

Amazing, outstanding, Jawdrop!! Fantaboulous #HarleenDeol

[Couchbase @couchbase]

Have you saved the date for #CBOnline?

This 2-day event will be action-packed w/ Couchbase content & will feature product experts, customers, partners, & community members highlighting our newest products and capabilities and so much more.

https://t.co/Q6D803YBeQ

[Couchbase @couchbase]

Learn how to use global secondary index partitioning for more than partitions in this blog: https://t.co/dE1p1TKWgl

[bvbvnirdalumni @bvbvnirdalumni]

These hallways have witnessed young impressionable kids bloom into capable minds. We request both Bhavans and NIRD management to ensure thatthis establishment continues to nurture youngminds @BhavansNews @NIRDPR_India
#savebvbvnird #savebhavans #savebvbvstudents #savebvbvteachers

[bvbvnirdalumni @bvbvnirdalumni]

Smt. Nirmala Prabhakar, ex-faculty of BVBV NIRD talks about the dedicated staff at BVBV NIRD and urges both NIRD and Bhavan's to protect their future
Watch her talk: https://t.co/fPz0OuxHZ7

@BhavansNews @NIRDPR_India

#Teachertalks #savebvbvnird #savebhavans #savebvbvteachers

[Lindsey @lindsey]

My spring 2021 distributed systems course finished up last week, and the full set of lecture videos is now up: https://t.co/2VJqDweZuY

No distributed systems background necessary. If you watch these lectures and get something out of them, I'd love to hear from you!

[bvbvnirdalumni @bvbvnirdalumni]

బివిబివి ఎన్.ఐ.ఆర్.డి పాఠశాల విద్యార్థులు, ఉపాధ్యాయులు ఎదురుకుంటున్న అస్థిర పరిస్థితులను, మా పాఠశాలను రక్షించుకునేందుకు మేము చేసే ప్రయత్నాలను మీ పత్రిక లో ప్రచురించినందుకు మా హృదయ పూర్వక ధన్యవాదాలు

@sakshinews

#savebvbvnird #savebhavans #savebvbvstudents #savebvbvteachers

[bvbvnirdalumni @bvbvnirdalumni]

We are grateful to @MVenkaiahNaidu @KishanReddyOfc @bandisanjay_bjp @DrRanjithReddy @asadowaisi @vhrcongress for their continued support to save BVBV NIRD and for their concern towards students, teachers of BVBV NIRD

#savebvbvnird #savebhavans #savebvbvstudents #savebvbvteachers

[bvbvnirdalumni @bvbvnirdalumni]

We deeply appreciate your coverage of the predicament faced by our students and teachers. Thank you for calling attention to our campaign to save BVBV NIRD.

@harleenminocha @oratorgreat
@DeccanChronicle

#savebvbvnird #savebhavans #savebvbvstudents #savebvbvteachers