Rajat Venkatesh

[Rajat Venkatesh @vrajat]

@psankar @jsensarma Replacing velox in spark and trino with gpu accelerated versions is another trend. Same problems.

[Glauber Costa @glcst]

I am excited to announce that we are officially writing a new version of Postgres. In Rust - and creating the LLVM of databases in the process.

In the span of a year, we have rewritten SQLite. Keeping the compatibility, increasing its feature set. MVCC, Types, (Live) Materialized Views, among other things. In the process of doing that, we have realized: At the end of the day, what makes SQLite special is that it compiles SQL to a database-specific bytecode. So why can't we compile *Postgres* to the same bytecode?

Turns out we can. I ran an experiment called pgmicro as a way to prove this hypothesis, and it works very well. It is time to make this official, and put the weight of Turso behind it. We shall give the world a modern take on Postgres. Wire compatible, but built on a new architecture.

We have already heard of others wanting to extend this. MySQL? Redis? the sky is the limit. What can we do if we do for databases what LLVM did for compilers? To prove how powerful the SQLite bytecode is, we are actually running DOOM compiled to the unmodified SQLite instruction set. And because Turso runs natively in the browser, you can play the game in your browser. With the database executing it.

Read the full story below! 👇

[Rajat Venkatesh @vrajat]

@rishdotblog Also data catalog is never up to date. Asking the human to guess gives directionally correct results. Update the llm memory and rerun when clear definition is available. Essentially maintain a nano-catalog of the human’s knowledge.

[Rajat Venkatesh @vrajat]

@hsinha1445 There is also an incentive to play fast & loose in certain scenarios to generate profits. The downside is limited to no profit share but for the client there may be real loss of capital.

[Rajat Venkatesh @vrajat]

@mitsuhiko @ecenanovic Primary reason is that copilot reviews are pretty good especially with rust. I am not yet at expert and the reviews help me to up-level.
Secondary reason is cost.

[Rajat Venkatesh @vrajat]

@jsensarma @tweetmayank One more: https://t.co/rO5UzIoBSG

[Rajat Venkatesh @vrajat]

@criccomini @googlecloud Thank you for the kind words

[Rajat Venkatesh @vrajat]

@K2_181 😮 You are contemplating buying a house ?!?

[Rajat Venkatesh @vrajat]

@seltzi_boi Agree that lineage can become unmanageable very quickly. Change over time also complicates it. One way to simplify is to overfit for a use case like PII detection or data quality triage.

[Rajat Venkatesh @vrajat]

Parsing and analyzing SQL queries provides superpowers for monitoring data health. For example, popular (or unused) tables and columns, column-level lineage and freshness of the data. The challenge is that it is hard to find open-source parsers for even open-source databases.

[Rajat Venkatesh @vrajat]

Longer version of my survey at https://t.co/L0wVfbeVW7

[Rajat Venkatesh @vrajat]

(3) Roll your own parser. This option is not for the faint hearted. Projects like Apache Calcite reduce the effort to support custom syntax but still have steep learning curve. Also there is a risk of your project joining the long list of abandoned projects.

[Rajat Venkatesh @vrajat]

Bangalore skies continue to amaze. Halo around the sun.

[Rajat Venkatesh @vrajat]

Wow!

[vicki @vboykis]

Next year, there will be a paper called BeachHouse with a diagram of a data warehouse situated perilously on the edge of what once was a datalake but has now just become an unmanageable ocean of Parquet and YAML files, intermingled all together in one dir. https://t.co/WaZlVtbHda

[. @_pgauthler]

I recently bought health insurance for few family members. The kind of person I'm, this meant an uncontrollable spiral into research

I read through at-least

- 15 policy documents,
- handbooks+circulars published by IRDAI (the regulatory authority),
- and several blogs

[The Fifth Elephant @fifthel]

How Zeotap achieved #GDPR compliance - Sathish Kumar shares technical details and learnings - https://t.co/skjztiGelm

Participate in #Data #Governance meetings to share and learn. @makash @abh1sek @DevanganaK @vrajat @pingali @drgoda @SwiggyTech @gojektech @rohitkilam