Aman Khan | Data Engineering

Data Engineer · 3 yrs SAP→Snowflake pipelines in production. Now MSc AI @ BTU Cottbus. Pipelines, warehousing, and what's underneath the SQL.

[Aman Khan | Data Engineering @amankhan_data]

wrapped up my data warehouse project: a star schema pulling sales from 3 branch dbs into one model, loaded with an etl i wrote in oracle pl/sql.
deduped ~47k customer records, turned 35 duplicate products into the real 29. diagram + repo below.

https://t.co/O1N8w5Sf9U
#Data #DWH

[Aman Khan | Data Engineering @amankhan_data]

Curious: how many of you have built a system from scratch to understand how it works underneath?

Not just used a database, but built a tiny one.
Not just used an HTTP server, but implemented one.
Not just used a cache, but built one.

[Aman Khan | Data Engineering @amankhan_data]

In Oracle, when a row grows and no longer fits its block, the row moves — but the ROWID doesn't.

The old block keeps a stub pointing to the new location.

No index breaks.
You just pay an extra I/O on every read of that row until the table is reorganized.

Small storage detail.

[Aman Khan | Data Engineering @amankhan_data]

9/Two engines. Two architectures. Same advice:

leave empty space in the page now, so updates don't relocate later.

PCTFREE and fillfactor are same knob.

I'd used one and read about the other and never connected them — because I learned both as settings instead of consequences.

[Aman Khan | Data Engineering @amankhan_data]

1/ For 3 years I wrote SQL against Oracle. ROWID was just there useful, reliable, never in the way.

That's what a good abstraction does. You never need to look underneath it

I looked underneath it. Its the thing that makes indexes possible

How a database actually stores a row:

[Aman Khan | Data Engineering @amankhan_data]

8/Postgres can't do that.

MVCC means an UPDATE writes a new row version. If it lands on a different page, every index pointing at the old TID gets rewritten.

Escape hatch: HOT — new version fits on the same page, no index touched.

Prevention knob: fillfactor.

[Aman Khan | Data Engineering @amankhan_data]

"Data chaos" isn't messy data. It's this chain:

You store the same fact in many places (redundancy)
→ they drift apart (inconsistency)
→ and your question needs all of them at once (distribution)

The scary part isn't wrong answers. It's plausible wrong answers.

#data #dwh

[Aman Khan | Data Engineering @amankhan_data]

Beilstein, Germany 🇩🇪

#Europe #Germany

[Aman Khan | Data Engineering @amankhan_data]

The raw data itself was messed up. One branch had a price stored as '.0.71' (yes, two decimal points).
Another had quantities written out as words like 'three' or 'four' instead of numbers.
wrote small functions just to clean this before doing anything else with it.

#Data #DWH

[Aman Khan | Data Engineering @amankhan_data]

Next step after the schema was loading real data into it.

First real problem:
Same customer registered in 3 different branches, but each branch gives them a different internal id.
Ids were useless for finding duplicates, so matched people by name + full address instead.

#Data

[Aman Khan | Data Engineering @amankhan_data]

haven't posted here in a while,
let me start sharing what i've been building again.
for my data warehouse project I designed a star schema (mE/R model) with customers, products, branches and time all rolling into one fact table.

more on this over the next few days.
#Data #DWH

[Aman Khan | Data Engineering @amankhan_data]

@Misbahtwts Khair Mubarak brother

[Aman Khan | Data Engineering @amankhan_data]

Eid Mubarak ✨️

[Aman Khan | Data Engineering @amankhan_data]

Bottom line: Matrix decomposition lets you work smarter, not harder on huge data — smaller representations, faster math, better insights 🚀

#Matrix #SVM #PCA #Decomposition #DataScience #DataAnalysis

[Aman Khan | Data Engineering @amankhan_data]

Why matrix decomposition is a superpower for huge data 👇

Matrix decomposition is powerful for huge data because it helps you reduce size, cost, and complexity without losing key information.

Here’s why it matters 👇🧵

[Aman Khan | Data Engineering @amankhan_data]

Real-world uses: • Recommendation systems (user–item matrices) • NLP embeddings & topic modeling • Image & signal compression • Large-scale analytics & ML pipelines.