Papers with Code

Our mission is to organize science by converting information into useful knowledge.

[Papers with Code @paperswithcode]

Read about the research at: https://t.co/jMxj3cov9S

[Papers with Code @paperswithcode]

Thank you everyone for trying the Galactica model demo. We appreciate the feedback we have received so far from the community, and have paused the demo for now. Our models are available for researchers who want to learn more about the work and reproduce results in the paper.

[Patrick Mineault @patrickmineault]

Galactica is basically GPT-3 for science. It can write whitepapers, reviews, wikipedia pages and code. It knows how to cite and how to write equations. It's kind of big deal 1/ 🧵

[Jim Fan @DrJimFan]

Today a 120B model called “Galactica” is open-sourced by @paperswithcode. It’s capable of writing math notations, citations, code, chemical formula, DNA, etc. Here’s why I think Galactica is a huge milestone in open foundation models, scientific automation, and responsible AI: 🧵

[Eric Topol @EricTopol]

Foundation #AI models come to science👍

[Sergei Kalinin @Sergei_Imaging]

The new language model for science https://t.co/EgXNqkoP37. Upon few quick tries, it seems to generate professional text in the areas I am familiar with. And 7 years ago we were *joking* about ML writing papers!

[Jim Fan @DrJimFan]

Haven’t been so excited by a text LM for a long time! And it’s all open! A true gift to science. 🎁

[Papers with Code @paperswithcode]

This is just the first step on our mission to organize science. And there is a lot more work to be done. We look forward to seeing what the open ML community builds with the model.

[Papers with Code @paperswithcode]

🪐 Introducing Galactica. A large language model for science.

Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more.

Explore and get weights: https://t.co/jKEP8S7Yfl

[Papers with Code @paperswithcode]

Despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. Galactica is also significantly less toxic than other language models based on evaluations.

[Papers with Code @paperswithcode]

We have explored some of the latest progress, architectural improvements, and emerging new techniques for long-range modeling. We'll continue to keep track of the progress on long-range modeling and LRA. More threads like this coming soon! Follow @paperswithcode for more.

10/10

[Papers with Code @paperswithcode]

How well do machine learning models perform on long sequences?

This is a question of high interest in ML research so let’s take a look at what we know so far?

1/10

[Papers with Code @paperswithcode]

Besides transformers, other types of models have been tested on LRA. Some of the top performing models are attained by S4 variants which are based on state space models. A recent, improved S4 variant (Liquid-S4) attained competitive results with Mega (current SoTA).

9/10