Paritosh Dutta

Data Engineer 👨‍💻🇮🇳

[Paritosh Dutta @paritosh741]

🔥 Spark Interview Problem 🔥

Q: Your Spark job processes 10 TB of Parquet data daily but takes 4 hours. 80% of execution time is spent in a single wide transformation. How do you diagnose and optimize it?

Asked at: Netflix | Databricks | Uber
Difficulty: Hard

Answer

Start by checking the Spark UI for the slow stage — look at shuffle read/write bytes, spill metrics, and task duration variance. 80% in one stage usually means data skew or an expensive shuffle.

Key optimization steps:

Steps:

Check if partition count is adequate — spark.sql.shuffle.partitions default 200 is too low for 10 TB; set to 2000-4000.

Use .explain() to verify the physical plan — look for BroadcastHashJoin vs SortMergeJoin.

Enable AQE (spark.sql.adaptive.enabled=true) to auto-coalesce small partitions and handle skew.

If Parquet columns are selectively read, verify predicate pushdown is happening.

Check for UDFs that block Catalyst optimization — replace with built-in functions.

Config:

spark.conf.set(
"spark.sql.adaptive.enabled", "true"
)
spark.conf.set(
"spark.sql.adaptive.skewJoin.enabled", "true"
)
spark.conf.set(
"spark.sql.shuffle.partitions", "3000"
)

df.explain(mode="cost")

Interview tip: Always check spill-to-disk metrics first. They indicate memory pressure and can help identify expensive stages.

#Spark #PySpark #DataEngineering #BigData #InterviewPrep

[Srijan Das @srijann25]

I have never expected accountability or sensitivity from the government in any branch be it Education, Health or basic things however this is another level of arrogance.

When govt sees everything with lense of haar jeet this is what will happen.

[Srijan Das @srijann25]

Bumrah for PM 🙏🎊🎉

Legend 🙌

#INDvsENG

[Srijan Das @srijann25]

Also, we are playing bilateral mostly on 230-240 roads nowadays. This was bound to happen once.

This T20WC has been really good in terms of the competitive pitches throughout the tournament.

[Paritosh Dutta @paritosh741]

@striver_79 Perfect explanation.

[Srijan Das @srijann25]

Same old election is rigged rhetoric. 🤦‍♂️

Why can’t the Congress party accept a loss gracefully?
Your alliance is leading(winning) in Jammu and Kashmir but you’re stuck with this rhetoric.

[Liverpool FC @LFC]

A milestone appearance for Mo 🇪🇬👑

3️⃣5️⃣0️⃣ with the Reds 🔴

[Paritosh Dutta @paritosh741]

@ParivahanSewa it’s been 3 months since I have applied for Learning License but it is still in “Printing Pending” state. I can’t apply for DL due to this issue. Please help asap as I tired of this highly inefficient system.

#ParivahanSewa
#driving #sarathi #license

[Paritosh Dutta @paritosh741]

@CSKian716 Lmao tru, they always seem to take knee jerk decisions. Rahane came back , scored in one game and wow became vc again wtff

[Paritosh Dutta @paritosh741]

@AdeevB @srijann25 And your break up 🥲

[Paritosh Dutta @paritosh741]

@Ishansharma7390 If people can make LinkedIn a dating app then anything can happen 😂

[Paritosh Dutta @paritosh741]

India V/S Australia Test Series

The Fall of Adelaide
The Resurgence of Melbourne
The Great Escape of Sydney
And now get ready for :-
The Seige of Brisbane 🔥🔥

#IndiavsAustralia #Vihari #BCCI #Ashwin #Rahane #stevesmith

[Paritosh Dutta @paritosh741]

@srijann25 @aryankaushik_ @official_dda Exactly , @OfficialSdmc @official_dda should look into the matter . It is very shocking and disheartening that children are being kept away from enjoying sports .

[Srijan Das @srijann25]

There are so many DDA and SDMC parks/grounds but we aren’t allowed to play.
We play cricket and some caretaker comes to stop us from playing. Why are most parks only for walking/yoga purposes? Why can’t you let us play in some of the parks.
@aryankaushik_ @official_dda

[Paritosh Dutta @paritosh741]

@OfficialSdmc Dwarka Sector 22 , kids are not allowed to play sports in any of the park in sector 22 , this is absolutely shameful .You built hundreds of open gyms but there is no benefit if facilities of sports are not there . Open gyms are useless if there is no play area .

[Srijan Das @srijann25]

Read the news not headlines.

[Paritosh Dutta @paritosh741]

My future India 15

#BCCI #TeamIndia @imVkohli @klrahul11 @RishabhPant17 @ShreyasIyer15 #BleedBlue

[Srijan Das @srijann25]

@EmmanuelMacron Namaste(नमस्ते) used as form of greeting by President of France and Chancellor of Germany 🙏🙏

[Paritosh Dutta @paritosh741]

Barca condition right now :-
#Barca #BarcaBayern #BartomeuDimision #Bartomeu #Messi