Roman Novak

Research Scientist @OpenAI, ex @GoogleDeepMind, Google Brain 🇺🇦

[lia 🇺🇦 ꕤ @kvituchalia]

The attack on Kyiv was absolutely horrid, but literally no one else will talk about it other than Ukrainians.

29 locations in one city attacked with missiles, an entire block of flats COLLAPSED and there are definitely casualties, yet its radio silence here.

[Kate Knibbs 🏄🏻‍♀️ @Knibbs]

Workers at OpenAI and Google have filed an amicus brief in support of Anthropic https://t.co/xJxxgls2b7

[Mo Bavarian @mobav0]

Anthropic SCR designation is unfair, unwise, and an extreme overreaction. Anthropic is filled with brilliant hard-working well-intentioned people who truly care about Western civilization & democratic nations success in frontier AI. They are real patriots.

Designating an organization which has contributed so much to pushing AI forward and with so much integrity does not serve the country or humanity well.

[Boaz Barak @boazbaraktcs]

I signed this and encourage anyone who cares about American AI leadership to do the same.

https://t.co/1YETDvpiuO

[Anton Gerashchenko @Gerashchenko_en]

A genocide against Ukrainians is unfolding right now.
It already has a name: "Kholodomor" (exhaustion through cold)

This crime is being deliberately committed by Russia.

In legal terms, it has a clear definition - genocide.

Under Article II of the UN Convention on the Prevention and Punishment of the Crime of Genocide, one of the defining acts of genocide is:

"Deliberately inflicting on a group of people conditions of life calculated to bring about its physical destruction in whole or in part."

Many Ukrainian cities - Kyiv foremost among them - have been without electricity, heating, and water for the second week in a row, amid severe frosts.

What does this mean?

It means people are being condemned to death by Russia.

This is what is happening:

▪️ Elderly people cannot leave their apartments to buy food or water because elevators do not work - or because they are physically unable to walk down the stairs;

▪️ No communication: mobile phones are dead, there is no way to call for help or an ambulance;

▪️ Sick people cannot use life-sustaining medical equipment because there is no electricity at home;

▪️ Mothers of small kids carry their children and strollers up staircases in high-rise buildings - along with water and food;

▪️ No hot meals and no possibility to cook food at home;

▪️ Sewer system failures, creating critical and dangerous conditions;

▪️ Illness caused by extreme cold and the inability to stay warm;

▪️ Children cannot attend schools because there is no heating. Distance learning is also impossible without electricity;

▪️ Kindergartens are closed, mothers cannot work, earn money to feed their children;

▪️ Stray animals are freezing to death on the streets;

▪️ Even pets are dying from the cold - parrots, aquarium fish, animals kept in terrariums;

▪️ Animals in zoos are freezing;

▪️ Collections of rare plants in Kyiv’s botanical gardens have frozen and died;

▪️ Small businesses are earning nothing, forced to spend money on generators and fuel - causing direct losses to the economy;

▪️ Severe harm to the environment and public health due to the constant operation of massive numbers of generators;

▪�� Public transport functions far worse due to power outages; electric transport does not operate at all, leaving people unable to reach work, doctors, or essential services;

▪️ Burst pipes and freezing temperatures are destroying homes, making them uninhabitable;

▪️ Hospitals are forced to cancel planned surgeries, operating in emergency mode and at full capacity;

▪️ Rates of depression, anxiety disorders, and burnout are sharply increasing.

Russia wants to make large Ukrainian cities uninhabitable.

This is a humanitarian catastrophe deliberately caused by Russia.

[Roman Novak @ARomanNovak]

Democracy dies in cringe

[The White House @WhiteHouse]

“Have you said thank you once?” –@VP JD Vance

[Mike Pence @Mike_Pence]

Mr. President, Ukraine did not “start” this war. Russia launched an unprovoked and brutal invasion claiming hundreds of thousands of lives. The Road to Peace must be built on the Truth.🇺🇸🇺🇦

“Russia Invades Ukraine in Largest European Attack Since WWII” @FoxNews (February 24, 2022) https://t.co/ra8R9PI9oj

[Lechao Xiao @Locchiu]

1/5. Excited to share a spicy paper, "Rethinking conventional wisdom in machine learning: from generalization to scaling", https://t.co/AZxPsMYyIa.
You might love it or dislike it!
NotebookLM: https://t.co/OB7nwhLnss
While double-descent (generalization-centric, data-efficiency) is cool, "skydiving” 🪂 (scaling-centric, compute efficiency) is adventurous!
The thinking is different.
Generalization-centric: reducing generalization gap using regularization.
Scaling-centric: reducing approximation error via scale -- as generalization gap has yet to emerge.
Two central principles of the generalization paradigm may not be applicable:
1) regularization as the primary guiding principle;
2) model comparison via a validation set.

[Katie Everett @_katieeverett]

We've gotten some great questions about the notion of alignment in our width-scaling parameterization paper! https://t.co/It6Zvyxnkg

A deep dive into the alignment metric and intuition 🧵 [1/16]

[Peter J. Liu @peterjliu]

We recently open-sourced a relatively minimal implementation example of Transformer language model training in JAX, called NanoDO.

If you stick to vanilla JAX components, the code is relatively straightforward to read -- the model file is <150 lines. We found it useful as a fork-able example for researchers to easily hack on and experiment rapidly. While we do not provide SOTA configs, we hope it can help researchers get started to try ideas, especially if new to LMs and JAX -- which is truly under-rated.

Repo: https://t.co/EkkYrUeFJf

[Jascha Sohl-Dickstein @jaschasd]

Have you ever done a dense grid search over neural network hyperparameters? Like a *really dense* grid search? It looks like this (!!). Blueish colors correspond to hyperparameters for which training converges, redish colors to hyperparameters for which training diverges.

[Avi Singh @avisingh599]

Excited to announce our new work on using synthetic data for improving mathematical problem solving and code generation in LLMs!

arxiv: https://t.co/1Fz3fD8MoF

A small amount of fine-tuning can lead to large gains (&gt;6% on Hendrycks MATH with Palm-2)

[Ilija Radosavovic @ir413]

we have trained a humanoid transformer with large-scale reinforcement learning in simulation and deployed it to the real world zero-shot

[bucket of kets @bucketofkets]

Happy to see our paper on adversarial arithmetic released!

(https://t.co/AjRaS2dPJS)

[Mitchell Wortsman @Mitchnw]

Sharing some highlights from our work on small-scale proxies for large-scale Transformer training instabilities: https://t.co/mCNFJYO2z8

With fantastic collaborators @peterjliu, @Locchiu, @_katieeverett, many others (see final tweet!), @hoonkp, @jmgilmer, @skornblith!

(1/15)

[Roman Novak @ARomanNovak]

@mathladyhazel Actually a round object looks square in polar coordinates, and vice versa

[Jaehoon Lee @hoonkp]

Today at 11am CT, Hall J #806 we are presenting our paper on infinite width neural network kernels! We have methods to compute NTK/NNGP for extended set of activations + sketched embeddings for efficient approximation (100x) for compute intensive conv kernels! See you there!

[Amin Karbasi @aminkarbasi]

This paper on "Fast Neural Kernel Embeddings for General Activations" will be presented tomorrow, Tue 29 Nov 11 a.m. CST - 1 p.m. CST, Hall J #806

[UkraineWorld @ukraine_world]

A Kyiv family has come to a gas station (where there is electricity) specifically to be able to plug in the inhaler their little girl needs to breathe

[Dr Bohdana Kurylo @BohdanaKurylo]

Due to #Russian missile strikes against civilian infrastructure, my elderly relatives (80+ y/o) have no electricity, water or heating. It's freezing cold. They barely have any food b/c local supermarkets are often closed. Mobile signal is also down, so I can't even reach them.