Dan Sotolongo

Streaming @SnowflakeDB; before @google, @twitter. I like distributed systems, types, and databases. Also, doing stuff outside.

[Dan Sotolongo @sortalongo]

@julianhyde Thanks! We're very excited about the use cases it can enable, and to share what we've learned through the process.

[Dan Sotolongo @sortalongo]

@julianhyde I think there's certainly a strong analogy, if not a direct correspondence between the two. Dependent types let you apply logic in the types, so they can certainly be used to enforce constraints. How mutation and transactions fit into it is more complicated...

[Dan Sotolongo @sortalongo]

@julianhyde sign me up! I have thoughts :)

[Tyler Akidau @takidau]

Incremental processing is the foundation of efficient stream processing across the entire latency spectrum, yet SQL lacks a native mechanism for querying changes to a table over time. Today we present our #SIGMOD paper describing Snowflake Change Queries https://t.co/DtkYvIs397

[Dan Sotolongo @sortalongo]

@embano1 @SnowflakeDB Haha perhaps not the best choice of name… it was intended as a deep talk for people familiar with the 101 article 😅

[lloyd tabb @lloydtabb]

Question for Data Twitter. What are your most important SQL Window Function use cases?
Rank?
Sequence numbers?
Rolling Averages?
Row Totals?
Cumulative Sums?

What else?

[Dan Sotolongo @sortalongo]

@christzolov @fhueske @takidau That conveyor belt animation at the end is 👌

[Caltech Astro Outreach @CaltechAstro]

Galactic atmospheres influence how galaxies live and die, but how do they work? Join us Friday, December 2 @ 7PM PT for a public astronomy lecture on galactic atmospheres. It will be hosted in-person at Caltech and streamed on Youtube Live. See you there!
https://t.co/Xv50pedzTe

[Dan Sotolongo @sortalongo]

New recording (no registration required!) at https://t.co/nCIYRIsMMV

[Dan Sotolongo @sortalongo]

The #current2022 talk I gave with @takidau is online! It’s called “Streaming 101 Revisited: A Fresh Hot Take”. We talk about what’s missing in existing models and propose new ideas to make declarative stream processing easier and more powerful than ever.

[Dan Sotolongo @sortalongo]

Slides at https://t.co/kDfWKFAAtO

[Dan Sotolongo @sortalongo]

@emaxerrno @ccomeau79 @takidau .@ccomeau79 Here’s how I’d put it: we start with normal SQL (just like Mz), then add in multi-temporality, changelogs, and better watermarks.
Like @emaxerrno said: lower latencies. That puts pressure on declarative models to keep up, so we need new concepts.

[Dan Sotolongo @sortalongo]

@mim_djo @jaceksan @SteveMcCanne https://t.co/WLPQOvTWYr

[Dan Sotolongo @sortalongo]

@jaceksan @mim_djo @SteveMcCanne Or Iceberg row level delete files

[Dan Sotolongo @sortalongo]

@bothra90 They seem to be a “used by few, needed by many” kind of thing. Some who use them swear by them, but not enough DBs support them for them to be mainstream.
Arguably, anyone building SCD2 tables would benefit from them.

[Dan Sotolongo @sortalongo]

@bothra90 E.g. if you have a row with valid_to=null, and the table is final where valid_to < 2022-10-19T00:00, that means the row cannot be deleted until after the 18th. So any query that needs to join with this row before that time can do so, but joins later may have to wait for progress.

[Dan Sotolongo @sortalongo]

@bothra90 Yes, of course!
This schema is intended to be a [bitemporal table](https://t.co/T4E2pYpuOY). That’s supported by some databases natively, but one can implement it manually in others.
Typically, valid_to is set to 9999-12-31 or NULL until the row is deleted.