Enes TAYLAN

AI Expert / ML Researcher, Boğaziçi Computer Engineering (PhD-c), Bilkent Computer Science (BS) & International Relations (MA)

[Biology+AI Daily @BiologyAIDaily]

Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization

1. This study explores the effectiveness of subword tokenization methods—Byte-Pair Encoding (BPE), WordPiece, and SentencePiece���in representing protein sequences, highlighting how their linguistic origins limit their performance in biological contexts.

2. Key findings show that vocabulary size significantly impacts tokenization performance. Smaller vocabularies ensure more shared tokens and better domain boundary preservation, while larger vocabularies lead to greater divergence and less effective segmentation.

3. BPE demonstrates superior contextual specialization and marginally better domain boundary preservation in smaller vocabularies, whereas SentencePiece excels in encoding efficiency, achieving lower fertility scores.

4. The analysis of linguistic laws reveals partial adherence to Zipf’s and Brevity laws, but substantial deviations from Menzerath’s law, suggesting protein sequences might follow unique organizational principles distinct from natural languages.

5. All tokenization methods struggle to maintain protein domain integrity, especially with increasing vocabulary sizes, underscoring the need for protein-specific tokenization approaches that respect biological sequence structures.

6. The results advocate for the development of specialized tokenizers tailored to the complexities of protein sequences, moving beyond adaptations of natural language tokenization strategies.

@enestaylan @suyunuCS
💻Code: https://t.co/vosafzIIXa
📜Paper: https://t.co/U6Bpj5EQJI

#Bioinformatics #ProteinSequence #NaturalLanguageProcessing #LinguisticLaws #SubwordTokenization

[Alex Rives @alexrives]

We have trained ESM3 and we're excited to introduce EvolutionaryScale.

ESM3 is a generative language model for programming biology. In experiments, we found ESM3 can simulate 500M years of evolution to generate new fluorescent proteins.

Read more: https://t.co/iAC3lkj0iV

[Enes TAYLAN @enestaylan]

Yep it is just curve fitting and that’s the beauty of it. With just curve fitting in next token prediction, amazing capabilities emerge. Search engine analogy is inherently wrong.

[François Chollet @fchollet]

The idea that you can build general cognitive abilities by fitting a curve on "everything there is to know" is akin to building a search engine by listing "every query anyone might ever make". The world changes every day. The point of intelligence is to adapt to that change.

[Andrej Karpathy @karpathy]

With many 🧩 dropping recently, a more complete picture is emerging of LLMs not as a chatbot, but the kernel process of a new Operating System. E.g. today it orchestrates:

- Input & Output across modalities (text, audio, vision)
- Code interpreter, ability to write & run programs
- Browser / internet access
- Embeddings database for files and internal memory storage & retrieval

A lot of computing concepts carry over. Currently we have single-threaded execution running at ~10Hz (tok/s) and enjoy looking at the assembly-level execution traces stream by. Concepts from computer security carry over, with attacks, defenses and emerging vulnerabilities.

I also like the nearest neighbor analogy of "Operating System" because the industry is starting to shape up similar:
Windows, OS X, and Linux <-> GPT, PaLM, Claude, and Llama/Mistral(?:)).
An OS comes with default apps but has an app store.
Most apps can be adapted to multiple platforms.

TLDR looking at LLMs as chatbots is the same as looking at early computers as calculators. We're seeing an emergence of a whole new computing paradigm, and it is very early.

[Andrej Karpathy @karpathy]

Around 5 years ago we were very proud of these state of the art results in image generation, trained on 32x32 "images" of CIFAR-10. You can kind of make out little wheel shapes, car/plane parts, and organic structures and textures. Pretty cool right

[Raphaël Millière @raphaelmilliere]

We will host a pre-conference debate on Friday, March 24th on the question: "Do Language Models Need Sensory Grounding for Meaning and Understanding?"

The debate will feature @Jake_Browning00, @davidchalmers42, @LakeBrenden, @ylecun, @glupyan &amp; Ellie Pavlick (@BrownCSDept).

[Enes TAYLAN @enestaylan]

@chelseabfinn @_eric_mitchell_ @yoonholeee @SashaKhazatsky @chrmanning Hello, you can try the DetectGPT on paper abstracts maybe https://t.co/pHK0J1ldwn

https://t.co/Pnu2W6YSDR

[Enes TAYLAN @enestaylan]

@jerome_massot @chelseabfinn I think no. It just shows average writing skills of a person is worse than LLMs in LLMs’ view of language. That’s also why they can approximate direction of derivatives with another LLM.

[Lex Fridman @lexfridman]

Here's my conversation with John Carmack (@ID_AA_Carmack), legendary programmer &amp; engineer. At over 5 hours, this is officially the longest conversation I've had on the podcast, and we can talk many more times. This was really fun and a huge honor for me. https://t.co/RuWzACGeGm

[Enes TAYLAN @enestaylan]

https://t.co/spnAqBYdLU

[Özgür Özdamar @OzgurOzdamar]

Bölümümüz hocalarının düzenlediği etkinliğe bekleriz. Sıkıcı bir etkinlik olmayacağı bence garanti 😉

[Enes TAYLAN @enestaylan]

2019 https://t.co/UFhgF2LkOL

[AI at Meta @AIatMeta]

In a paper published today in @PNASNews researchers at FAIR find that biological structure and function emerge in representations of language models trained on massive databases of protein sequences. Read the paper: https://t.co/EBd0D2XF0O

[Yann LeCun @ylecun]

Language is an imperfect, incomplete, and low-bandwidth serialization protocol for the internal data structures we call thoughts.

[Enes TAYLAN @enestaylan]

“When your scope for action is greatest, the knowledge on which you can base this action is always at a minimum. When your knowledge is greatest, the scope for action has often disappeared.”

Henry Kissinger

[Özgür Özdamar @OzgurOzdamar]

Bilkent Uluslararası İlişkiler Bölümü hesap açmış. Kitaplar, makaleler, konferanslar, toplantılardan haberdar olmak için şuradan takip edelim: @BilkentIRDept

[Enes TAYLAN @enestaylan]

@MatttGFX no answer, in Bayesian sense you zigzag between prior and posterior

[Enes TAYLAN @enestaylan]

“Science is a way of thinking much more than it is a body of knowledge.” ― Carl Sagan

[Enes TAYLAN @enestaylan]

Forecasting the US 2020 elections | The Economist https://t.co/So0vaMDht0

[Google DeepMind @GoogleDeepMind]

Looking for a few more favourite resources from the team? Today’s #AtHomeWithAI picks are from research scientist @TaylanCemgilML! (1/6)