aranku

technically member of staff @magicailabs

[aranku @aranku_]

In awe of what our small team has achieved through focused compounding on pretraining. Next stop, RL.

[Magic @magicailabs]

Frontier pretraining is said to be a big-lab-only game. We don’t have 100k chips yet, so there’s only one way: algorithmic efficiency. Our new recipe matches DeepSeek V4 Pro’s pretrain using 50x less compute – that’s roughly half the FLOPs used for GPT3, or ~$0.5M on GB200.

https://t.co/uBs5o5Arsu

[Eric Steinberger @EricSteinb]

We're hiring for a new team aiming to train AI SWEs to robustly complete long-horizon work on a no-restrictions computer via the GUI.

Today's models excel at small, Olympiad-type coding tasks but struggle in complex codebases and aren't easy to integrate into existing enterprise workflows.

Completing tasks that go beyond just writing code is a key requirement for autonomy.

Our goal with this work is to train our models to interact with software tools and platforms the way people do, and to make integration into existing workflows and teams easier.

We're integrating this throughout the entire modeling stack, from model architecture through to pretraining and RL. The most interesting piece is designing scalable, realistic environments within which models can practice these skills.

This role is in-person in SF, Seattle or NYC (relocation support is provided).

If this sounds interesting, please reach out! There's a lot of work to do.

[Magic @magicailabs]

Excited to announce we’re building an Applied Team focused on post-training. Come explore what's possible with our new (and still unreleased) LTM2 models and their 100M token context window. Apply here: https://t.co/Tx0QSIM9vI

[aranku @aranku_]

@iwanttobeinmars @magicailabs Thanks! Should be fixed now :)

[Magic @magicailabs]

LTM-2-Mini is our first model with a 100 million token context window. That’s 10 million lines of code, or 750 novels.

Full blog: https://t.co/oFz4A9ynVZ

Evals, efficiency, and more ↓

[aranku @aranku_]

@CFGeek @haus_cole Prefix tuning isn’t very expressive https://t.co/Yci7GpaCw1

[Magic @magicailabs]

We've raised $117M from @natfriedman and others to build an AI software engineer.

Code generation is both a product and a path to AGI, requiring new algorithms, lots of CUDA, frontier-scale training, RL, and a new UI.

We are hiring!

[aranku @aranku_]

@giffmana @DimitrisPapail Unfortunately, it collides with your fellow colleagues https://t.co/KfNnSQT1Iv

[aranku @aranku_]

@finbarrtimbers This is the closest work summarising all the things I mentioned off the top of my head: https://t.co/bpdbIsGNb2

[aranku @aranku_]

@finbarrtimbers This is evidenced by other open-ended generation tasks finding more favourable results with top-p/k sampling. Once model-based metrics (BLEURT or COMET which are more correlated with human preferences) became more prevalent in MT, beam search was shown to underperform.

[aranku @aranku_]

@finbarrtimbers The “success” of beam search in MT was a result of BLEU score being the leaderboard metric at the time. Candidate sentences generated by beam search tended to correlate well with bleu score (as long as num beams wasn’t too high).

[aranku @aranku_]

@AlbertQJiang @TaliaRinger Will any dataset details be included at all going forward (e.g. was instruct tuning data included or is this just a base model)?

[sophia @cis_female]

SEND US DATA AND WE'LL PAY YOU $100/GB: the Magic data bounty https://t.co/LQwJQElkf3

[aranku @aranku_]

@abacaj The same is true of human raters/annotators (different people would rate outputs differently).

[Eric Steinberger @EricSteinb]

AI with long-term memory!

*A lot* of work left to do but happy to share a little more about what we've been up to.

It's been incredibly fulfilling to work with a wonderful team and the trust of our backers towards this milestone. Thank you for the opportunity <3

[Magic @magicailabs]

Watch LTM-1 reuse and synthesize information across files:

[Magic @magicailabs]

Meet LTM-1: LLM with *5,000,000 prompt tokens*

That's ~500k lines of code or ~5k files, enough to fully cover most repositories.

LTM-1 is a prototype of a neural network architecture we designed for giant context windows.

[aranku @aranku_]

@cis_female Ah gotcha, missed that

[aranku @aranku_]

@FelixHill84 Not a paper but this was the first blogpost to make the connection between CoT and pretraining on code iirc https://t.co/6DcOdiz22Q

Though we have seen models like UL2 be able to do CoT without explicit code pretraining.