A little difficult to write this, but here goes.
Things haven't been great financially for me lately. And honestly, with Diwali and the festive season around, it feels a little harder than usual.
I'm not sharing this for sympathy. I'm trying to find a way forward,
1/5
I've spent the last 7 years learning, building, breaking things and figuring out my way around technology.
From software development to Data Engineering, it's been quite a journey.
Today, I work with Azure, Databricks, PySpark and SQL.
I want to share more of what I learn along the way:
→ Practical Data Engineering
→ Technology & AI
→ Career growth
→ Productivity and continuous learning
No gurus. No overnight success stories.
Just practical lessons, honest experiences and conversations with people who love technology.
If you're on a similar journey, let's connect.
What are you currently learning?
Azure Data Factory interview question:
You need to ingest 100 SQL Server tables into ADLS Gen2.
Would you create 100 pipelines?
I'd build a metadata-driven framework using:
- Configuration tables
- Lookup activity
- ForEach activity
- Reusable child pipelines
- Dynamic source and sink paths
- Audit logging
But how would you handle the 3 tables that fail while the other 97 succeed?
My Engineering grades
1st -> 5.96
2nd -> 6.78
3rd -> 6.77
4th -> 7.57
5th->7.93
6th-> 7.48
7th -> 6.74
8th -> 8.0
Not sure if things went well
My Engineering grades
1st -> 8.7
2nd -> 8.6
3rd -> 8.52
4th -> 8.4 (realised this shit is a scam)
5th->8.2
6th->7.8
7th ->7.6
8th ->7.7
I ended up pretty alright post college tbh!
One SQL concept every Data Engineer should understand properly:
ROW_NUMBER() vs RANK() vs DENSE_RANK().
They might look similar, but they behave differently when duplicate values exist.
Interviewers love testing this with real-world ranking problems.
Don't just memorise the syntax. Understand the difference.
SQL interview question:
You have an Orders table with customer_id, order_id, order_date and amount.
How would you retrieve the latest order placed by each customer?
Would you use:
1. ROW_NUMBER()
2. RANK()
3. A JOIN with MAX(order_date)
What's your approach?
A few days ago, I shared that I'm struggling financially.
That post crossed 155K impressions.
I'm still in the same situation. Things haven't magically changed.
But I've decided to do something about it using the skills I've spent the last 7 years building.
Here's what I'm trying. 🧵
1/6
PySpark interview question:
You have two DataFrames.
One contains 10 million records. The other contains only 5,000 records.
You need to join them.
Would you use a broadcast join?
Why or why not?
The answer isn't just about row count. Think about the actual size in memory, join strategy and executor capacity.
Your PySpark job takes 40 minutes instead of 5.
What do you do?
Most people immediately think about increasing cluster size.
But before spending more on compute, I'd investigate these 7 things.
A practical Spark performance checklist 🧵
If you're preparing for a Data Engineering interview, you can check out my guides or book a ₹99 career guidance session.
And if you know someone who might benefit, please share this thread with them.
I'm not expecting anyone to feel sorry for me.
I'm simply trying to earn through the knowledge and experience I've built over the years.
Every booking, purchase and genuine share makes a difference.
Thank you for reading. ❤️
6/6