Top Tweets for #distributedLearning
Echo turns reinforcement learning into a distributed party: capybaras sample experiences, train policies together, sync models, and loop smarter—scalable, efficient, and powered by everyday compute.
#EchoRL #DistributedLearning #AIInfra #GradientNetwork
The future of AI is no longer confined to the cloud. 2026 is the Year of Distributed Intelligence. We're bringing #GenAI to the very edge—powering smart devices, real-time insights, and truly decentralized learning. The revolution is on-device! 🧠✨
#EdgeAI #DistributedLearning
Imagine an internet of models.
All contributing, learning, and evolving in real time.
FractionAI makes that vision real.
#FractionAI #DistributedLearning
@FractionAI_xyz
🎓📷
We've been working on Didactica, a revolutionary learning platform that connects students and teachers through the power of blockchain technology.
#EdTech #Blockchain #DistributedLearning #ZeroWallet #Education #Innovation #ZEROCAT #Didactica
https://t.co/dgyVTkczRj
Education should be modern and trustworthy - and blockchain technology helps us deliver on that promise. Interested in trying it out or collaborating? Let's connect! #EdTech #Blockchain #DistributedLearning #ZeroWallet #Education #Innovation #ZEROCAT #Didactica
Exploring the power of distributed learning with @PrimeIntellect's 10B model! Fascinating how @GoogleDeepMind's DiLoCo uses Adam for inner and Nesterov for outer optimization, reducing bandwidth by 400x while maintaining efficiency. #DistributedLearning #AIResearch
Distributed learning?
Recently, @PrimeIntellect have announced their 10B distributed learning (https://t.co/CqIRQeiLlV), what is it exactly?
Going back to the origin, Federated Learning (FL, https://t.co/E6hP7PW4ol) aims to train a model across a fleet of phones. To handle the low bandwidth between them, the gist is that each phone will perform multiple training steps (forward/backward & update) independently from each other. After a hundred of steps, each phone's replica of the model computes its trajectory (aka an outer gradient): it's simply a delta in the parameters space between where the model is now vs where it was at first.
Then, an all-reduce (thus no need of centralized server!) averages all the replicas outer gradients. Thus the peak communication is as high as in data-parallel (https://t.co/m5kZdW65Vs), but because it's only done rarely, its cost is amortized.
Now, those "outer gradients" are quite different from your classical gradients, but we can actually use an optimizer on those! FedOpt (https://t.co/K9tdwmWrXw) proposes to optimize it with Adam, further accelerating training.
Most of the FL algorithms used however as inner/local optimizer SGD. This is suboptimal to train current large-scale transformer. In DiLoCo (https://t.co/q21VQaxLSJ) from my team at @GoogleDeepMind, we use Adam as inner optimizer, but more importantly Nesterov as outer optimizer -- this simple change is ridiculously powerful:
With DiLoCo, you can now be as flops/token efficient as data-parallel training while using two orders of magnitude less bandwidth. A bunch of papers (https://t.co/OwZwsT2Qgv, https://t.co/cLGQEpa0Ky, https://t.co/ykctgYWdIO) have then re-used that winning recipes on large-scale transformers, including OpenDiLoCo.
In particular, OpenDiLoCo from @PrimeIntellect, is training a 10B (!) DiLoCo model across the world using Hivemind (https://t.co/ffNYHBB1HW), while synchronizing every 100 steps, and downcasting the outer gradients in int8. That's a 400x bandwidth reduction. This reduction is so big that, as @samsja19 noted, their speed bottleneck now is checkpointing on disk, not communication.
Godspeed @PrimeIntellect! 🫡
The other approach that has been made somewhat public is Distro from @NousResearch. For now, there is very little details, so it's hard to say how legit it is (read their report). But we know that one of the trick used is adding a regularization loss, forcing all replicas to stay close towards the global model that is synced once in a while. This is very reminiscent of ElasticSDG (https://t.co/ZQDQ6gbclF @ylecun) and FedProx (https://t.co/eNmRmjfcCZ), which have also been re-visited recently with PAPA (https://t.co/g6ISWub9kx).
I strongly suspect it is still combined with some kind of outer optimization, in order to remain flops-efficient. We'll see.
The main point of this post is to show that there is very little new things in the recent distributed works, but that i think a lot of research on federated/distributed was artificial and done on toy settings. Now, the time is ripe for distributed training, see this @SemiAnalysis_ post: https://t.co/cZ51k2XdOw. There is lot of alpha into re-visiting past ideas, and investing heavily on the engineering parts to make them work, for real, at scale.
#skillsforesight or #skillsforecast? #blendedlearning vs #distributedlearning?🤔
Check-out the 2024 "Terminology of #Europeaneducation & training policy", a multilingual glossary with 430 terms on EU policy priorities, #skillsintelligence & #Employment https://t.co/zXkBfxiE9o
Exciting, non-RL tweet for a change. Long has written his first paper in his PhD with our group. If you are into #continuallearning, #multiagent learning or #distributedlearning, check it out! If you are not, check it out anyway. 😄
1/4 📢 Excited to share our latest work on Distributed Continual Learning (DCL)! Large-scale AI models are powerful, but they're usually trained once due to costs. DCL lets AI constantly adapt & improve by learning from others.
Exploring the Intersection of Distributed Computing and Machine Learning: Rice University's Colloquium
#academicinstitutions #AI #algorithms #artificialintelligence #computingplatforms #distributedlearning #Engineering #FAIR #Google
https://t.co/uvC0NJwDUB
I am very excited to have a chance to contribute to the tutorial on continual and pervasive AI at ECML PKDD 2023 in Turin. Thanks to my co-instructors Davide Bacciu, Claudio Gallicchio, Antonio Carta. https://t.co/3oVIscUXIT #ai #pervasiveai #distributedlearning
The articles in our September issue include a tutorial on the training of spiking #NeuralNetworks , an overview of #deepreinforcementlearning for smart grid operations, and an overview of privacy and security for #distributedlearning. https://t.co/fDwSp6yAM7
🆕 Fresh addition to ENCRYPT #BlogSeries
👉 "Ensuring Data Privacy in Distributed Learning"
by Martin Zuber, at @CEA_List, is now available at https://t.co/MRGoBkR4VK
💡 Discover how #HomomorphicEncryption techniques are safeguarding sensitive information in #DistributedLearning
Beyond excited to expand our work with the @ICRC
--
Thank you @EssTechEPFL and @ETH4D for this phenomenal opportunity! ❤️
#studentpower #impactthroughacademia #llms #distributedlearning #hiring #swe
@EPFL_en @ICepfl @CIS_EPFL
The "slides" notebook from my talk on a #meetup about "Barrier Scheduling and #DistributedLearning on #ApacheSpark with #LightGBM"
➡️ https://t.co/j4Ip4wqEYG
There are a few order-related flows but overall it's ready for...public consumption 😉 Enjoy! ❤️
What a #meetup! Just finished presenting what I managed to learn about #DistributedLearning on #ApacheSpark (and #Databricks) using libraries like #PyTorch #xgboost ❤️
➡️ https://t.co/enERuOMfbW
Enjoyed it so much that I can't wait for next meetups! Feeling so...(xg)boosted! 😉
AusCAT network in #RadOnc. The #distributedlearning approach outlined in this article has the potential for decision support models for patients for whom clinical trial evidence is not applicable
@InghamMedPhys @MattFField @shalinivinod13 @Ozradonc et al
https://t.co/7mFo3bln31
Catch the #AINET Workshop "Enabling #6G and #Sustainable Digital Transformation by Intelligent Network Automation" at @EuCNC.
@BehnamShariati is co-organizing & @PooyanSafari is presenting our #DistributedLearning framework #DLFi.
👉https://t.co/sZENFl0Ezh
La semana pasada en el @fabacademany los alumnos se han enfocado en la fabricación de placas electrónicas. Han tenido que fresarla, soldar los componentes y comprobar que funciona correctamente.
#MakeYourJob #digitalfabrication #distributedlearning #fabacademy2023
😍 La semana pasada en el programa formativo distribuido @fabacademany, los estudiantes tenían que fabricar algo grande 😅. Para ello Pedro, nuestro alumno ha aprendido a utilizar la fresadora #cnc.
#MakeYourJob #digitalfabrication #distributedlearning #fabacademy2023 #leonesp
🤩 La semana pasada en el @fabacademany, nuestro alumno Pedro se han enfocado en el diseño de placas electrónicas. Además tuvo la ayuda de nuestro antiguo alumno y ahora instructor @CyberMODE, donde le enseño a diseñar en KiCAD.
#digitalfabrication #distributedlearning #leonesp
Last Seen Hashtags on Sotwe
Most Popular Users
Elon Musk 
@elonmusk
241.7M followers
Barack Obama
@barackobama
119M followers
Cristiano Ronaldo
@cristiano
114.4M followers
Donald J. Trump
@realdonaldtrump
111.9M followers
Narendra Modi
@narendramodi
107.2M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.4M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
83.9M followers
Lady Gaga
@ladygaga
75.4M followers
Virat Kohli
@imvkohli
73.4M followers
Kim Kardashian
@kimkardashian
70.9M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
66.4M followers
Bill Gates
@billgates
65.2M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.3M followers
CNN
@cnn
61.8M followers
X
@x
60.7M followers