Code & Clicks

Proud Indian | Becoming decent enough software engineer

[Code & Clicks @VPCodeClicks]

10/75

Today I learned about high-level design of a web-crawler system.

What is politeness and robot.txt file in this sense? How should we make fault-tolerant and persistent system. How URL deduplication would be happening here and how to watch out for crawler traps?

[Code & Clicks @VPCodeClicks]

9/75

Today I learned about high-level designing job-schedulers. How cron-jobs handles future job scheduling and recurring ones as well. How delayed message delivery and SQS’s visibility timeout helps in immediate job scheduling. How at-least once job execution is ensured.

[Code & Clicks @VPCodeClicks]

8/75

Yesterday, I learned about high-level design of YouTube in terms of video upload and viewing usecase.

Upload process is almost like designing dropbox.

[Code & Clicks @VPCodeClicks]

7/75

On 15th, I learned about high-level designing a distributed cache system.

I got to learn various read/write scaling techniques.

[Code & Clicks @VPCodeClicks]

@Ayussheth Sure, count me in.

[Code & Clicks @VPCodeClicks]

5/75

Today I learned about the high-level design of Google Docs. How real-time collaboration works, how concurrent users can edit the same document, and how changes are propagated and synchronized across clients with minimal latency.

Pretty cool system design problem.

[Code & Clicks @VPCodeClicks]

4/75

Today I learned about high-level designing a brokerage system that facilitates customer orders and provides a customer stock data. We are not building an exchange, instead we would be utilising exchange APIs.

In this, users can see live prices of stocks and manage orders.

[Code & Clicks @VPCodeClicks]

3/75

Today I learned about designing basic setup of UBER at high level

Covered:
• Rider & driver workflows
• Matching and dispatch
• Real-time location tracking
• Scalability and reliability considerations

I would share new things I learned today in respect to Uber design

[Code & Clicks @VPCodeClicks]

When you say Top-K in the last hour, do you mean:
1. Sliding Window — Last 60 minutes from NOW —> Rankings continuously change
2. Tumbling Window — Fixed buckets (e.g., 2 PM–3 PM) —> Rankings computed per completed window

1st is complex and require continuous expiring old events

[Code & Clicks @VPCodeClicks]

1/75

Today learning about building Top-k system. Examples can be top-k trending youtube videos, spotify songs, instagram reels/posts or articles.

The core problem is: Given a huge stream of events, continuously maintain the K most popular/relevant items.

#75Hard

[Code & Clicks @VPCodeClicks]

What exactly does Top-K mean?

Top-K videos of: All time? Last hour/day/month?

This single clarification can completely change the design. One lead to simple counters + heap and other lead to stream processing, time buckets, event expiration, and significantly more complexity.

[Code & Clicks @VPCodeClicks]

I’m going to do a 75 hard challenge to upskill my software engineering skills. I would try to solve (maybe half-solve) a design problem each day and keep posting my progress on x. Problems may involve HLD/LLD and sometimes DSA.

#grind #75Hard

[Code & Clicks @VPCodeClicks]

@suni_code In incremental IDs case
3. Pre-assign IDs interval (like 1-1M to 1st server, 1M-2M to 2nd server and so on, and assign new range only when exhausted)

[Code & Clicks @VPCodeClicks]

@suni_code If generated IDs are UUIDs, then it doesn’t matter that much, but

In case of incremental IDs, there are 2 approaches I can think of
1. If there are M servers, each ith server can only generate C*M+i as ID
2. Maintain a counter(c*X) in Redis, each server fetches X continuous IDs

[Code & Clicks @VPCodeClicks]

@suni_code When a connection drops, the client doesn't start over. Instead, it queries which parts already uploaded. The implementation requires the client to track the upload session identifier.

[Code & Clicks @VPCodeClicks]

@suni_code This is a solved problem by different blob storage.

With S3, you can chunk the file into multiple small parts, use multipart API to upload directly to S3. For every successful upload, S3 would notify.

[Code & Clicks @VPCodeClicks]

@pradeep_ That’s more like blessing than a privilege I guess.

[Code & Clicks @VPCodeClicks]

Know how to use enemies for your own profit,

The wise man profits more from his enemies than a fool from his friends.

[Code & Clicks @VPCodeClicks]

Never put too much trust in friends, learn how to use enemies.

[Code & Clicks @VPCodeClicks]

Mao’s strategy of constant conflict
1. Never pick a fight with someone. You are not sure you can defeat.
2. If you have no apparent enemies, you must sometimes set up a convenient target event, turning a friend into an enemy.

[Code & Clicks @VPCodeClicks]

Without enemies around us, we grow ology and enemy, and or heels, serpent orbits, keeping us focused and alert.