Prince

Student at Haldia Institute of Technology | Share what I learn | Newbie

[Prince @RajPrince_07]

Here is my linktree url -
https://t.co/pCumC5qA1s

[Prince @RajPrince_07]

@Arshad_sayed_03 @vidya4sure @harshkr4227 @Mantu_kumar91 @vidystdotcom @piyushgarg_dev

[Prince @RajPrince_07]

@karanveer04 @Mantu_kumar91

[Prince @RajPrince_07]

@vidya4sure @Mantu_kumar91

[Prince @RajPrince_07]

@shubhstwt

[Prince @RajPrince_07]

@Mantu_kumar91

[Prince @RajPrince_07]

@Mantu_kumar91

[vidya4sure @vidya4sure]

Join me tonight w/ @RajPrince_07 @Mantu_kumar91 @harshkr4227 on X & YT

YT channel "vidya4sure"

Go to LIVE app: vidyst[dot]com

[Prince @RajPrince_07]

@vidya4sure @Mantu_kumar91 @harshkr4227 Chilla Chilla ke sbkoo batade

[Prince @RajPrince_07]

@vidya4sure @harshkr4227 @Mantu_kumar91 @shubhstwt 😀😀

[vidya4sure @vidya4sure]

Going live in few minutes w/ my friends @Mantu_kumar91 @RajPrince_07 @harshkr4227

https://t.co/6xyk7rCQ0m

[Prince @RajPrince_07]

@kunalstwt @vidya4sure

[Prince @RajPrince_07]

Today I worked on a Databricks orchestration project.

I created a Job with 2 pipelines, a notebook, and a Python script.

Ingested database data into Bronze and split it into valid and quarantined records.
Added a health check to allow or stop the job based on quarantined rows.

[Prince @RajPrince_07]

@CoderArmy @Mantu_kumar91

[Prince @RajPrince_07]

@Hiteshdotcom I applied for BETA users but seats are full.

[Prince @RajPrince_07]

@ducati_dada

[Prince @RajPrince_07]

Auto Loader is a Databricks feature for incremental file ingestion. It provides efficient file discovery and supports schema inference/evolution. It can ingest files from supported cloud/object-storage locations.

#Databricks

[Prince @RajPrince_07]

Medallion Architecture -
Bronze - We store the raw data into this layer . It is append only . We do not apply any filter .
Silver - We do some refinement  on data stored in Bronze . Deduplication , schema check ,casting data types .
Gold - We apply some aggregation .
#Databricks

[Prince @RajPrince_07]

With Deletion Vectors, deleted rows are marked instead of rewriting the whole Parquet file. The same concept used with DELETE, UPDATE and MERGE.
I also learned how Delta handles duplicate batch writes using txnAppId and txnVersion to provide exactly-once semantics.
#Databricks

[Prince @RajPrince_07]

@Nitaj333 @Mantu_kumar91

[Prince @RajPrince_07]

Today I learned about Time Travel and RESTORE in Databricks :-
- Time Travel helps us read data from an older version, while RESTORE rolls the table back to that version.
- If only a few rows are corrupted, restoring the whole table may remove other correct changes.

#Databricks