Interesting.
Stochastically Subsampled Self-Attention. Self-attention lets each token attend to every other token, creating information pathways between tokens, but there are n^2 may paths information can flow through (where n is sequence lenth).
Only a small amount of those information pathways are useful for next-token prediction, so there are sparse sub-networks within a model that can be constructed and trained independently to learn useful behavior -- in theory.
But these pathways are dependent on input, so you can't manually hard-code which to prune.
So they randomly sample sparse subnetworks, reducing computation and memory by about 4-8x.
Same spirit as stochastic depth or dropout.
The question to always ask yourself is,
if you only have
one shot at life, given that time is the only asset you can't control, but anything else that exists is malleable, would you want to spend your life the exact way you're spending it now?
And if not, why not?
For one day, time will cease to exist. There is nothing more important than time.
I don't see why any answer to the former question should be dissatisfactory, as uncompromisingly, one should always be intentional with time, with respect to how one wants to spend it.
Even if it requires extreme change.
It doesn't matter what you're optimizing for, whether it be for the moment or the outcomes, this has always seemed to apply.
There's one startup that I'm extremely excited to see come out of stealth.
I've seen this idea be passed around so many times, by different startups, yet all eventually pivoted to different dimensions.
I'm fairly certain this specific project has been in the works for about 7-8 years, but recently has been made to something real.
I have a gut feeling this one will work out.
building the threads clone, loving the progress so far
-> Added the Create Thread screen -
-> Using Clerk for authentication -
-> Convex as the backend database -
-> Integrated webhooks for automation
what you use for auth and db, especially for mobile dev?