Mayank Bhaskar

Founder & CEO of an AI Research & Engineering Labs | Weekly @twimlai GenAI meetup | @Cohere_Labs Community Lead & Researcher | @AILucknow | Prev @TCS @Infobeans

[Mayank Bhaskar @cataluna84]

🔥 TinyAya v0.3 follow-up: the run early-stopped — so we launched the anneal leg it skipped. It's descending right now, and it's already beating the plateau.

For everyone who asked for the nitty-gritty after the last post — here's what's actually happening on the TPU pod as I write this. 🧵👇

Why an anneal leg exists at all. The 110,463-step run trained on a WSD (warmup–stable–decay) schedule: 1,100 warmup steps, then a long plateau at lr 1.716e-4, with an 11,000-step linear decay to zero scheduled for the end. But we also ran stop-anytime early stopping — 10 validation cycles (2,500 steps) without improvement ends the run. At step 65,250 of 110,463, the plateau flattened out (best val composite 2.9048 @ step 62,750) and the early stop fired… which meant the scheduled decay never ran. Our round-3 ablation had shown the decay phase is where WSD harvests its loss drop — it beat the plateau on every metric at probe scale. So the model was parked one phase short of done.

⚙️ The anneal leg, mechanically. We resumed the exact step-65,250 checkpoint — weights, full Adam optimizer state, and RNG restored from GCS — and extended max_steps to 76,250. With the WSD scheduler's total set to start+11,000, the resume lands directly inside the decay: no warmup replay, no discontinuity. Verified from the first logged points: lr = 1.7121e-4 at step 65,275, stepping down by exactly 3.9e-7 per 25 steps — the predicted linear slope of 1.716e-4 ÷ 11,000. Every other hyperparameter is byte-identical to the plateau leg (global batch 32 = 2 rows/chip × 16 chips, grad-clip 1.0, same seed, same 3,200-sample val gate every 250 steps), and the data order continues deterministically mid-epoch-2. Early stopping is OFF for the leg — it's a fixed-length descent by design.

📉 Is it working? Yes — measurably, within the first 750 steps.
• plateau best: 2.9048 (step 62,750)
• step 66,000: 2.8922
• step 67,250: 2.8827 — text token accuracy 95.2%, text loss 0.458, semantic-codebook accuracy 40.6%

Every validation cycle since the descent began has set a new best. ~9,000 steps (~6 h) still to go as the lr grinds toward zero.

🔢 Scale, in tokens: each step processes 32 sequences × 300 frames × 9 tokens/frame (1 text + 8 audio codebooks) ≈ 86k supervised tokens → the anneal alone covers ~950M tokens, bringing the full run to ~6.6B by step 76,250. Hardware: the same multi-host TPU v6e-16 (4 hosts × 4 chips, SPMD data-parallel), ~1.40 s/step, zero spot preemptions across the entire 2.2-day plateau leg.

🧯 One honest ops note for the monitoring nerds: W&B briefly showed the run as "crashed" mid-anneal — it was a heartbeat lapse during the boot's one-time XLA compile of the checkpoint-save graph, not a failure. The process was at 200% CPU the whole time and recovered on its own. Multi-day TPU runs are 50% training and 50% learning which alarms are real.

📦 Every checkpoint is kept (log-spaced early + every 1,000 steps) — the full trajectory ships as a Pythia-style suite for anyone who wants to study how a speech-to-speech model learns, including how the anneal reshapes it.

📈 Watch the descent live: https://t.co/i0LMP0d1WX
🔧 The full engineering trail — plan, hyperparameter rationale, resume drills (PR #10): https://t.co/PdbvUMGVYy
📝 Project blog: https://t.co/pgnC9Az03A

Release evals (ASR-chrF++, MOS, BLASER) and the Hugging Face suite land after the descent finishes. The next post has final numbers. 🎯

TPUs via the TRC @Google grant and project in the @Cohere_Labs Expedition Tiny Aya 2026

#SpeechAI #MachineTranslation #TPU #LearningRateSchedules #OpenSource #MLOps

[Mayank Bhaskar @cataluna84]

🗣️🔁 TinyAya v0.3 — an open Turkish↔Hindi speech-to-speech translation model — and the long-horizon training run is past its halfway mark.

🧠 v0.3 translates spoken Turkish to spoken Hindi (and back) directly in the audio-token domain: a Cohere2 LLM backbone (LoRA-fine-tuned) fused with Moshi's frozen depth decoder, generating Mimi codec tokens frame by frame — text and 8 audio codebooks per 80 ms frame.

🔊 Audio samples included as well.

⚡ The current run: 110,463 steps on a multi-host TPU v6e-16 pod (spot capacity via Google's TPU Research Cloud), now past 50% with roughly a day and a half to go ⏱️

📦 Every checkpoint is kept — log-spaced early saves plus every 1,000 steps — so the full training trajectory ships as a Pythia-style suite for anyone who wants to study how a speech-to-speech model learns, not just the final weights.

🛡️ Making a multi-day spot-TPU run survivable was half the work, and that engineering trail is public: preemption self-healing with automatic resume, dataset preflight gates so a wrong corpus can never silently train, multi-host rendezvous, a coarse-to-fine codebook curriculum against deep-codebook collapse, and an 8-stage release-evals harness (ASR-chrF++, BLASER-2.0, MOS deltas, latency) with frozen, digest-verified eval sets.

📈 Live training run: https://t.co/i0LMP0d1WX
🔧 Full implementation & hardening story (PR #10): https://t.co/PdbvUMGVYy
🏷️ v0.3 code freeze: https://t.co/PHeyHaLSVk

🤗 Model checkpoints and eval numbers land on Hugging Face once the run completes and release evals pass. More soon.

#SpeechAI #MachineTranslation #TPU #OpenSource #SpeechToSpeech

[Mayank Bhaskar @cataluna84]

@Aleks_FUT PC...

[Fut Sheriff @FutSheriff]

🚨ULTIMATE PLUS EDITION GIVEAWAAAAAAY! 🔥🔥🔥🔥

• Follow @FutSheriff & @Aleks_FUT
• RT ✅

THAT’S ALL! WINNER SOON 👀🌟⏳

[Mayank Bhaskar @cataluna84]

@AsyFutTrader PC... Will appreciate a copy

[ASY FUT @AsyFutTrader]

🚨 FC 27 FREE COPY ULTIMATE+ GIVEAWAY 🔥🔥

To join:
Repost 🔁
Comment your platform 🎮
Must be following me ✅️

Winners Sunday #FC27

[Claude @claudeai]

Introducing Claude Opus 5.

It's a thoughtful and proactive model that comes close to the frontier intelligence of Fable 5 at half the price.

[Cohere Labs @Cohere_Labs]

Cohere Labs 🤝Twitch

When we started planning ML Summer School, we never imagined 7,000+ people from around the world would want to learn alongside us 🤯

Because the response has been far bigger than we planned for, we're moving all sessions to a Twitch livestream so we can host all of you!

You'll still be able to earn a certificate of completion, so nothing changes there 📜

If you haven't registered yet, there's still time! Please register through Zoom so you'll receive all the session reminders, have the full schedule added to your calendar, and stay up to date throughout the week: https://t.co/kl8pzgG7SV

[Mayank Bhaskar @cataluna84]

@futagentt PC

[Anton Lozhkov @anton_lozhkov]

For over two years, the largest open code dataset was The Stack v2… until today.

🥞 The Stack v3 is out: the largest open code dataset ever released: 114 TB, 770 languages, 224M repositories, ~5T tokens of deduplicated and filtered source code. Fully open, no restrictively licensed code included.

The upgrade:
- v2 (2024): 68 TB raw -> 2 TB / ~550B tokens, 618 languages filtered
- v3 (2026): 114 TB raw -> 15.9 TB / ~5T tokens, 713 languages filtered

C++ x15, TypeScript x7.5, Rust x7, Python x4.8. Even the COBOL corner of GitHub got a bigger slice.

Part of that is two fresh years of open source. Part of it is a bug we found in v2's deduplication - story below. 👀

Things that make v3 different:
1. Contents inline. The #1 complaint about v2 was "cool dataset, where's the actual code?" v2 shipped file IDs from the Software Heritage graph, and fetching contents was a DIY treasure hunt. v3 is self-contained: sources embedded directly, one row = one repository. Download finishes -> you start training.
2. Fresh crawl. v2 was a 2023 snapshot of even older crawls. v3 is a direct re-crawl of GitHub at the latest commit, completed by August 2025: 224M repositories, 44B files. Forks were only included if they had 5+ stars.

Two ways in:
🥞 stack-v3-train - near-deduplicated, quality-filtered, PII-redacted, contents inline. Point load_dataset at it and go.
https://t.co/px2LW3v6VB
🏗 stack-v3-full - the entire 114 TB corpus as an HF Storage Bucket: every duplicate kept with cluster IDs, stubs for excluded files. Roll your own dedup, filters, and mixes.
https://t.co/SQzP5gOmzl

The record isn't really ours to claim - it's millions of everyone's repositories, neatly stacked. Thanks to every developer who keeps their code public, the BigCode community, and friends who built the infrastructure that made this possible!

[Mayank Bhaskar @cataluna84]

Understanding Speculative Decoding
Deep dive into speculative decoding feat. Evan Carlson

From the ICML 2023 Oral @ https://t.co/GPlr4lWyG1

https://t.co/zHpwhnKJKO

[Percy Liang @percyliang]

⚡! Tokenization is the first thing you do in CS336 (language models from scratch). If Marcel can work his magic for the rest of the LM pipeline, the world will be a better place.

[FPV Labs @fpv_labs]

1/ Announcing Vidur, an end-to-end system to generate fine-grained action labels on raw robot and human videos. Vidur leads across all metrics on WGO and EgoExoLearn Bench across temporal segmentation, semantic precision, semantic recall, and end-to-end action-label quality 🧵

[Abhinav @Abhinavsns]

We got CUDA kernels running on @Apple Metal GPUs! 🍎⚡️

Thanks to GPT 5.6 Sol, our existing CUDA/HIP-style simulation kernels in OpenFPM now directly execute on Apple Silicon GPUs.

Here is how we got there, and what the first real Dam break benchmark says. 🧵

[Google Cloud India @GoogleCloud_IN]

We are LIVE! 🚨 The Cloud Technical Series: Agentic AI Edition is kicking off right now!

Tune in to see how industry leaders are moving from AI pilots to enterprise-grade production. Join the livestream here 👇 https://t.co/Z1cuUMZ4kj

[Mayank Bhaskar @cataluna84]

@drfeifei Congratulations 🚀

[Alaya Lab @alayastd]

Full technical details of AlayaWorld have just been released.
AlayaWorld is a full-stack, open-source video world model that supports 720p, 24 FPS streaming video generation with camera control and text-driven event generation.
https://t.co/OtbAUToIJX

[Mayank Bhaskar @cataluna84]

@alayastd Congratulations 🎊

[Mayank Bhaskar @cataluna84]

RAW Worklog | Part 54 | Long Horizon Run of S2S Translation & nanoGPTJAX https://t.co/HUa99UyBFM

[Mayank Bhaskar @cataluna84]

@novasarc01 🤩

[Prince Canuma @Prince_Canuma]

Excited to introduce Nativ 🚀

Run frontier open models locally on your Mac. No accounts, no subscriptions, no cloud.

⚡ Built on mlx-vlm — multimodal + fastest on Apple Silicon
🔒 100% private — every token generated on your machine
🛠 Plug your coding agents into a local endpoint
📊 Live telemetry — tokens/sec, memory, usage
🆓 100% open source, MIT licensed, free forever

Your Mac is more capable than you think. Stop renting intelligence.

Download for macOS 👇
https://t.co/JoWC2hLXYA

Github repo 🌟
https://t.co/ZXxv58SoWu

[Noam Brown @polynoamial]

Long-running models can solve hard open-ended problems, but their persistence can create safety risks that shorter-horizon evaluations miss.

We’re sharing what we learned from studying a long-running model, and how those findings are shaping our approach to evaluations, alignment, monitoring, and user control.

https://t.co/yePIzJGsAU