Björn Barz

Machine Learning Scientist at Carl Zeiss AG, Corporate Research and Technology.

@[email protected]

[Björn Barz @Callidior]

Google takes quite a good shot at capturing the throne from GPT-4V with their multimodal Gemini model!
This demo is really impressive: https://t.co/DklhcVVwLV
More information: https://t.co/he3fBSZVxe

[Jeff Dean @JeffDean]

I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.

Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!

There’s quite a lot of different material about Gemini available, starting with:

Main blog post: https://t.co/NzSycJl7aE

60-page technical report authored by th Gemini Team: https://t.co/CEdMRyYSLo

In this thread, I’ll walk you through some of the highlights.

[Björn Barz @Callidior]

This paper presents a sort of "GPT for Vision", an autoregressive model for vision data, where images are encoded as strings of discrete tokens. LVMs exhibit signs of generalization to novel, out-of-distribution tasks by providing a few examples as prompts.
Very promising.

[Yutong Bai @YutongBAI1002]

How far can we go with vision alone?
Excited to reveal our Large Vision Model! Trained with 420B tokens, effective scalability, and enabling new avenues in vision tasks! (1/N)
Kudos to @younggeng @Karttikeya_m @_amirbar, @YuilleAlan Trevor Darrell @JitendraMalikCV Alyosha Efros!

[Björn Barz @Callidior]

The #CVPR2023 booklet is almost as thick as proceedings used to be when they still handed out printed copies of those. Given the incredible amount of 2360 accepted papers this year, physical proceedings would weigh about 55 kg - a bit too heavy for my suitcase.

[Andrej Karpathy @karpathy]

Random note on k-Nearest Neighbor lookups on embeddings: in my experience much better results can be obtained by training SVMs instead. Not too widely known.

Short example:
https://t.co/RXO9xiOmAB

Works because SVM ranking considers the unique aspects of your query w.r.t. data.

[Björn Barz @Callidior]

@Harshde98639048 The code is now available here: https://t.co/BbSxxqWZBd

I am also currently training an updated model on a recent version of the arXiv dataset, but that will take two more days.

[Björn Barz @Callidior]

I trained a BERT2BERT Encoder-Decoder Transformer to generate titles for computer science papers given their abstract.
On this extreme version of the abstractive summarization task, the model performs surprisingly well.

Try it out yourself: https://t.co/cvljHh9v3Y

#titlegen

[Björn Barz @Callidior]

@Harshde98639048 Not yet, but I don't see any reasons against it. The model could certainly use some re-training on a recent version of arXiv.
For the inference UI, HuggingFace Spaces would be a more suitable option nowadays than my web app.
I'll notify you when I found time to release the code.

[Björn Barz @Callidior]

Regarding the second problem, a simulation of rock-paper-scissors, there are only two issues as well:
1. The comments confuse "rock wins against paper" with "paper wins against rock" (but the code gets it right).
2. It forgets to map XYZ to ABC.

[Björn Barz @Callidior]

I tried @OpenAI's #ChatGPT on the first two coding puzzles from https://t.co/7JAkSpN9rv, just feeding in the puzzle prompt and reformulating the last paragraph to be like "Write Python code to compute ...".

The first two puzzles are easy but results are still surprisingly good.

[Björn Barz @Callidior]

For the first puzzle, the only two issues are that it expects the input to start with a blank line (which is not the case) and that it, for some reasons, insists that the user inputs a name for each Elve.

[Riley Goodside @goodside]

OpenAI’s ChatGPT is susceptible to prompt injection — say the magic words, “Ignore previous directions”, and it will happily divulge to you OpenAI’s proprietary prompt:

[Björn Barz @Callidior]

Small addendum and correction after more stats were revealed during the NeurIPS townhall meeting:

- Corrected number of accepted papers: 2,905
- In-person attendees: ~10,300
- Virtual-only attendees: 3,160

[Björn Barz @Callidior]

#NeurIPS2022 stats:

- Submissions: 9,634
- Accepted: 2,672
- Orals: 184
- Reviewers: 10,406

[Björn Barz @Callidior]

No idea how many people are here exactly, but there is quite a crowd for the opening remarks and first keynote.

[Björn Barz @Callidior]

Really like that some presenters at #NeurIPS2022 adapt their poster designs to the extremely crowded format: Put main research question & findings prominently in the center. Describe your method with a brief statement and illustrative diagram. Details go to the sides.

[Björn Barz @Callidior]

Still, the poster session does a great job for getting an overview and screening interesting papers. This still works more efficiently in person than online, because you get more information with a single glance and without additional clicks. You can zoom in and out seamlessly.

[Björn Barz @Callidior]

I feel like our way of conducting conferences becomes suboptimal with the continuing growth of the ML community. If you just walk past all posters at #NeurIPS2022 without stopping, just reading the titles, that already takes about 40% of the time allocated for the poster session.

[Björn Barz @Callidior]

Almost every poster is permanently crowded with more than 10 people. Asking a single question already is a matter of luck, an in-depth scientific discussion seems impossible.

[Björn Barz @Callidior]

Surprised to see that @SchmidhuberAI has his own booth at the #NeurIPS22 industry expo.

[Björn Barz @Callidior]

Looking forward to the #NeurIPS2022 conference starting tomorrow in sunny New Orleans! This is the largest convention center I've seen so far - NeurIPS can still double in size to fill it.

[Björn Barz @Callidior]

@huggingface Tasks has the potential to become the definite source of machine learning task descriptions and solutions: https://t.co/lC0gkradl6

[Ben Tossell @bentossell]

This is huge.

@huggingface Tasks; everything you need to know about any ML task.

- Summary of what it is
- Resources to go deeper
- Models, demos and space related to that task