Announcing Atomit ๐ฅ
Are you a student? Meet Atomit, your AI-powered study buddy that adapts to you and your courses. It's built to help you learn better by turning complex concepts into bite-sized atoms.
Join the waitlist! We have a lot in store ๐ฅ
https://t.co/JqP93K0LME
nice architectural plan bro ๐
some advice i can give regarding this:
when you pull from an external api, your first move should always be dumping the raw response into an object store like s3 or even a local json file
and why is that ?
tbh if you try to clean the data on the fly and your logic fails on record 9,999 out of 10,000, that means you have to ping their api server again, doing that on a paid api is going to cost you money because you're repeatedly hitting their server everytime an error comes up in your code.
so always decouple your extraction from your transformation. get the data safe on your storage first, then run your transformation.
extract -> load raw -> transform
Iโve mostly worked with static datasets in Excel and Power BI. Now Iโm taking the next step, working with real-time data.
My next focus:
API โ Python โ SQL โ Power BI
Instead of downloading datasets, I want to:
- Pull data directly from APIs
- Clean and transform it using Python
- Store and query it with SQL
- Build dashboards in Power BI
The goal is to move closer to how data is actually handled in real-world scenarios.
Still learning, but excited to build this end-to-end workflow.
If youโve worked with APIs before, Iโd appreciate any tips or resources. @chidirolex@_VictorUgwu@Smanmalik83@iam_daniiell@ObohX
#DataAnalytics #PowerBI #Python #SQL
You should never vibe code mission critical Data Engineering applications.
- Not the pipeline that feeds your regulatory submission.
- Not the transformation that calculates patient dosing.
- Not the reconciliation logic your finance team signs off on.
Use AI to build it. Absolutely.
But do not use AI to review it for you.
That's the human expert's job โ and it's non-negotiable.
The code runs. The tests pass. The output looks plausible.
That's the danger.
Let AI accelerate the build.
But the review? That's where domain expertise earns its keep.
@Ubunta ran into this exact thing building a flight ticketing pipeline. used AI to write some dbt tests for NUC amounts based on standard industry logic. tests passed, but the companyโs internal logic was different. silent failures are the worst ๐ญ
i need you guys to promise me something
this thingโs hard to build. like really hard. and yes itโs gonna be completely free, idc.
i just need yโall to promise me youโre gonna use it and put your friends on and use it. not download and keep, actually use and post about it
thanks