booydar

Aydar Bulatov, PhD

[Aran Komatsuzaki @arankomatsuzaki]

Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity

[booydar @booydar]

Even the shoulders of giants are made of ants.

[AK @_akhaliq]

In Search of Needles in a 10M Haystack

Recurrent Memory Finds What LLMs Miss

paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 10^4 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10^7 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any open neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences

[Aran Komatsuzaki @arankomatsuzaki]

In Search of Needles in a 10M Haystack: Recurrent Memory Finds What LLMs Miss

Fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 10M timesteps

https://t.co/mrBij6JGzY

[booydar @booydar]

"The difference between theory and practice is greater in practice than in theory" - Zack Freedman

[booydar @booydar]

@dooraven @MikhailBurtsev @yurakuratov here for all experiments memory size is fixed to 10 vectors, yes

[booydar @booydar]

@mattylawhon @MikhailBurtsev @_akhaliq @yurakuratov Not yet, but there will be more applied experiments in the upcoming work!

[booydar @booydar]

@dooraven @MikhailBurtsev @yurakuratov Here we show that RMT shines on specific memory-intensive tasks. Stay tuned for our future work, where we will outline more applied cases with different Transformers on real-world tasks!

[booydar @booydar]

@vrungta @MikhailBurtsev @_akhaliq @yurakuratov No, here we do create a database since the memory is small (fixed to 10 vectors) and updated every timestep. To retrieve from memory we use attention rather than similarity.

[booydar @booydar]

@generatorman_ai @MikhailBurtsev @yurakuratov It is a separate line of work but there are similarities. RMT also uses the recurrent approach (but on level of segments) and memory can be viewed as the dynamic state of a turing machine

[booydar @booydar]

о чем можно говорить с человеком, чей остров в океане не омывает тысячи рек?

[booydar @booydar]

b1 x b2 <- b3 <- b4 <- b5
loss.backward()

b1 x b2 <- b3 <- b4 <- b5 <- b6

b1 x b2 x b3 <- b4 <- b5 <- b6

1)
b1, b2
b1 <- b2
2)
b2, b3
b3 <- b2

b1 -> b2
b1 x b2

b1 -> b2 -> b3
b1 x b2 -> b3
b1 x b2 x b3

[booydar @booydar]

Две вещи наполняют душу всегда новым и всё более сильным удивлением и благоговением, чем чаще и продолжительнее мы размышляем о них — это звёздное небо надо мной и моральный закон во мне. И.Кант "Критика практического разума"

[booydar @booydar]

отношение к алкоголю: сдержанно положительное

[booydar @booydar]

из серых штанов российской дорожной сети грузно вываливалось зелёное пузо тайги

[booydar @booydar]

too many bottles of this wine you can't pronounce

[booydar @booydar]

абсолютно точно лучший год всех времен в музыке это 2015

[booydar @booydar]

"смысл" - субъективно наиболее точное выражение комплексного явления через простые общепринятые истины

[booydar @booydar]

надеюсь, никогда не забуду лучшего диджейского перформанса, который мне довелось увидеть. публика кричала от восторга, даже не знал, что так бывает. жаль, что я так и не смог понять, кто там диджей

[booydar @booydar]

you told me again that you prefer handsome men, but for me you would make an exception