Ivan Chelombiev

Research Eng @IsomorphicLabs. Curious about learning. Technology, sustainability and architecture enthusiast.

[Ivan Chelombiev @savelichic]

1/n While everybody’s been busy packing for #NeurIPS2023, our team at @graphcoreai has been busy with this beauty. Let me introduce:
✨SparQ Attention✨
TL;DR This is a plug-and-play inference Attention block for pre-trained LLMs, which evaporates the KV cache bandwidth 🧵

[Max Jaderberg @maxjaderberg]

The Iso team has cooked something incredible: our new technical report unveils the latest results from our drug design engine, the IsoDDE, progressing far beyond AlphaFold 3. This breaks new ground compared to AF and other similar methods by a significant degree across all key benchmarks. 1/7

[Isomorphic Labs @IsomorphicLabs]

Today we share a technical report demonstrating how our drug design engine achieves a step-change in accuracy for predicting biomolecular structures, more than doubling the performance of AlphaFold 3 on key benchmarks and unlocking rational drug design even for examples it has never seen before.

Head to the comments to read our blog.

[Simon Evans @DrSimEvans]

+++NEW ANALYSIS+++

UK electricity was the cleanest ever in 2024, with emissions per unit falling by more than two-thirds in a decade

Highlights:
🏭end of coal power after 142yrs
🔥fossil fuels at record-low 29% share
🌄renewables at record-high 45%

https://t.co/zsH79J7HeK
1/9

[Gabriele Berton @gabriberton]

Libraries and tools that every deep learning project should use: loguru, tqdm, torchmetrics, einops, python 3.11, black. Optional: prettytable. Good for debugging: lovely_tensors. Any other ones I've missed?

Below a few words on each of them:

[Ivan Chelombiev @savelichic]

@EdConwaySky the “evil twin” chart is just a mirage! if you use a zero-aligned y-axis, you can see coal consumption has been stagnant in china for the last 10 years, despite a big rise in power usage. considering that solar displaces coal, there’s good reason to hope coal use will go down

[Ivan Chelombiev @savelichic]

@EdConwaySky @IEA the “evil twin” chart is just a mirage! if you use a zero-aligned y-axis, you can see coal consumption has been somewhat stagnant in china for the last 10 years, despite a big rise in power usage. considering that solar displaces coal, there’s good reason to coal will go down

[Nicolas Fulghum @nicolasfulghum]

☀️ Solar power has scaled up faster than any other source of electricity in history ↗️

It took just 8 years to go from 100 TWh to 1000 TWh.

It will only take 3 years to go from 1000 TWh to 2000 TWh!

[Ivan Chelombiev @savelichic]

@Haonan_Wang_ Did you try observing the effect of float16 vs bloat16 by any chance? I wonder if the increased precision of float16 can compensate sufficiently not require float32 in training

[Simon Evans @DrSimEvans]

Today is the day the UK becomes the first G7 country to completely phase out coal power

Since opening "Jumbo", the world's first coal power plant in 1882, the UK's coal plants have burned through a whopping 4.6bn tonnes of coal, emitting 10.4GtCO2 – which is more than most countries have ever released

Here's the story of how that all came to an end:

https://t.co/jOMqWjo4Vs

[Roman Gaditskii @gadirom_]

SGD vs Adam💡
I punish x10 for distance to mouse in the loss function,
but Adam's gradient normalisation eliminates the effect.
#MLX #SwiftUI #MachineLearning

[Tom Harwood @tomhfh]

Why oh why is growth at zero?

[zed @zmkzmkz]

it's 1:28AM and I just finished this abomination. fully illustrated toy calculation of 1 transformer layer.

why would I make this? idk ask my thesis advisor, "not everyone knows how a transformer works, you have to give an example"

[Mark Cummins @mark_cummins]

It’s no secret that LLM training data is running out. How close are we to the limit? To answer that, here's an estimate of the total amount of text in the world from every major source:

[Daniel Kaiser @spectate_or]

llama3-120b randomly creates a new word "prefaceate" that has 0 search results on google

[Ziming Liu @ZimingLiu11]

MLPs are so foundational, but are there alternatives? MLPs place activation functions on neurons, but can we instead place (learnable) activation functions on weights? Yes, we KAN! We propose Kolmogorov-Arnold Networks (KAN), which are more accurate and interpretable than MLPs.🧵

[The Economist @TheEconomist]

Europe’s carbon price is its biggest climate achievement. It is part of the reason that the continent’s emissions fell by a steep 15.5% in 2023 https://t.co/4KyiqJDYa6 👇

[Ivan Chelombiev @savelichic]

@samswoora @tophinity SF life sounds fun

[Graphcore Research @GCResearchTeam]

At 2pm today Graphcore researchers @luka_ribar & @savelichic will be presenting at @letsunifyai's popular reading group.

We'll be covering our recent SparQ paper - a method for increasing LLM inference throughput by sparsifying attention.

Live stream: https://t.co/8XSpilGc08

[Santiago Ortiz @moebio]

Take a look into the mind of the machine! visit my new project here: https://t.co/Q8CIvwcsby

I repeated the same completion prompt "Intelligence is " hundreds of times and used this to peer into the statistical and semantic behavior of chatgpt