chodedestroyer5 @chodedestroyer5 - Twitter Profile

Managing groups of game objects just got easier. Collections, now in Studio Beta, use live queries to group instances and bind your logic to the whole group, reducing boilerplate code. Learn more on the #RobloxDev forum below. https://t.co/XjVIoBASGq

146

630

58

88

299K

chodedestroyer5 @chodedestroyer5

21 days ago

@tracklist @ohavyj OMGOMGOMGOMG NOOOO OWTFF WQIHECKJQWEHWHY WHY WHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHYWHY

1

0

82

chodedestroyer5 @chodedestroyer5

21 days ago

@niitroxen Wow

0

2

0

80

chodedestroyer5 retweeted

niitroxen @niitroxen

22 days ago

raw footage and stuff i've been working on, uploaded an edited vid too: https://t.co/qwPfHDC4FF #Roblox #RobloxDev #Anime #VFX

3

62

4

12

2K

chodedestroyer5 retweeted

Charles

@charles_v11

29 days ago

Little peak into what we've been working on: - Run any model from any harness from one API key (claude code, codex, open code, pi, etc) - Access to new fusion models pushing the cost and performance frontier - Local-cloud routing with configurable model pools! Very very soon Works where you do!

charles_v11's tweet photo. Little peak into what we've been working on:
- Run any model from any harness from one API key (claude code, codex, open code, pi, etc)
- Access to new fusion models pushing the cost and performance frontier
- Local-cloud routing with configurable model pools!
Very very soon

Works where you do!

6

31

5

4

5K

chodedestroyer5 retweeted

wafer

@wafer_ai

22 days ago

every speculative decoding step computes the target model's logits on rejected tokens, then throws them away. that's a free, on-policy training signal sitting on the floor of every llm serving stack. here's how that's getting picked up: speculative decoding speedup is roughly (1 − α^(K+1)) / ((1 − α)(1 + Kc)), where α is acceptance rate and K is the lookahead. this function is a geometric series, so it compounds really fast. for example, a 0.10 drop in α erases ~25% of your throughput. the fundamental problem we need to solve for here is that the draft model in your serving stack was trained on generic datasets like sharegpt or ultrachat. but your model traffic is very specific: voice agents for specific functionalities, coding agents, chatbots with specific context, etc. so the distributions don't match. this means that α naturally drifts from 0.75 to 0.65 over weeks and you lose a quarter of your speedup. the fix is to keep training the draft from live traffic. and the elegant part is that every verify pass already produces the labels. for every node in the speculative tree, accepted AND rejected, the target's logits are right there in the kernel output. you just have to harvest them and use them. so the reward reduces to reverse-KL on the verifier's logits at every tree position. no REINFORCE, no baseline, and no entropy bonus. it's just one custom tree-attention mask that lets a single forward/backward produce gradients for every accepted and rejected branch at once. the systems side is uglier and more interesting IMO. together's aurora paper is the cleanest public reference for this loop. the trainer never loads the target model. it only holds the ~1B-parameter EAGLE-3 draft (one transformer block fed by concatenated hidden states from three target layers, low, mid, high). the inference server streams over RDMA: those 3-layer hidden states plus the top-1024 verifier logits per position. 256 KB compressed to 2 KB per token. what if you wanted to wire this into your own vllm stack? the pieces are coming together. RFC 33118 added hidden state extraction. PR 23075 added /collective_rpc, the hook for atomic draft reload. one thread draining into a ring buffer, one AdamW step every 5k tokens with top-k forward KL with a canary that rolls back if held-out acceptance rate drops more than 5%. together open-sourced aurora as the full reference (ray + mooncake + patched sglang). sglang itself exposes enable_aux_hidden_states and update_weights_from_disk out of the box. specforge from the lmsys team is the production eagle-3 trainer. snowflake's arctic ships as a vllm plugin with YAML recipes for custom drafts. speculative decoding has been leaking free supervision for two years. the fix is to collect this data to make sure your draft model doesn't drift in your real world prod data. @gpuemi technical deep dive :)

wafer_ai's tweet photo. every speculative decoding step computes the target model's logits on rejected tokens, then throws them away. that's a free, on-policy training signal sitting on the floor of every llm serving stack.

here's how that's getting picked up:

speculative decoding speedup is roughly (1 − α^(K+1)) / ((1 − α)(1 + Kc)), where α is acceptance rate and K is the lookahead. this function is a geometric series, so it compounds really fast. for example, a 0.10 drop in α erases ~25% of your throughput.

the fundamental problem we need to solve for here is that the draft model in your serving stack was trained on generic datasets like sharegpt or ultrachat. but your model traffic is very specific: voice agents for specific functionalities, coding agents, chatbots with specific context, etc. so the distributions don't match. this means that α naturally drifts from 0.75 to 0.65 over weeks and you lose a quarter of your speedup.

the fix is to keep training the draft from live traffic. and the elegant part is that every verify pass already produces the labels. for every node in the speculative tree, accepted AND rejected, the target's logits are right there in the kernel output. you just have to harvest them and use them.

so the reward reduces to reverse-KL on the verifier's logits at every tree position. no REINFORCE, no baseline, and no entropy bonus. it's just one custom tree-attention mask that lets a single forward/backward produce gradients for every accepted and rejected branch at once.

the systems side is uglier and more interesting IMO. together's aurora paper is the cleanest public reference for this loop. the trainer never loads the target model. it only holds the ~1B-parameter EAGLE-3 draft (one transformer block fed by concatenated hidden states from three target layers, low, mid, high). the inference server streams over RDMA: those 3-layer hidden states plus the top-1024 verifier logits per position. 256 KB compressed to 2 KB per token.

what if you wanted to wire this into your own vllm stack?

the pieces are coming together. RFC 33118 added hidden state extraction. PR 23075 added /collective_rpc, the hook for atomic draft reload. one thread draining into a ring buffer, one AdamW step every 5k tokens with top-k forward KL with a canary that rolls back if held-out acceptance rate drops more than 5%.

together open-sourced aurora as the full reference (ray + mooncake + patched sglang). sglang itself exposes enable_aux_hidden_states and update_weights_from_disk out of the box. specforge from the lmsys team is the production eagle-3 trainer. snowflake's arctic ships as a vllm plugin with YAML recipes for custom drafts.

speculative decoding has been leaking free supervision for two years. the fix is to collect this data to make sure your draft model doesn't drift in your real world prod data.

@gpuemi technical deep dive :)

13

135

13

108

18K

chodedestroyer5

@chodedestroyer5

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users