Žygimantas Medelis

---

[Žygimantas Medelis @zzgmm]

@TheAhmadOsman Following instructions

[Žygimantas Medelis @zzgmm]

Thursday, I will be presenting my open-source LLM tooling work at the London Clojurians online event. I'll go over the components needed to build LLM apps and discuss whether or not these components are already a part of LLM models and APIs.
https://t.co/bw5EI8SPIz

[Žygimantas Medelis @zzgmm]

@virattt To make this more accurate tokenization needs to be taken into account. For example, based on a text blob at hand, GPT4 produces 449 tokens, Mistral makes ~17% more - 543. With that Mistral's cost is almost the same as GPT4.

[Žygimantas Medelis @zzgmm]

A demo of my latest work on Bosquet - an LLM tooling to work with Gen AI
https://t.co/dAY8O3JiB8
https://t.co/CDiTFKaldP

[elvis @omarsar0]

The worst take that I have seen these past few days is that long context models like Gemini 1.5 will replace RAG.

This couldn't be further from the truth.

Let's explore one scenario. Imagine you had data that has a complex structure, changes regularly, and has an important temporal dimension (e.g., code edits/changes and web logs), meaning one data point might be connected to other historical data points and potentially many more in the future. You can't use today's long-context LLMs (on their own) to address use cases that depend on such type of data. Not only could this data be too complex for LLMs to handle but today's max context windows are just not feasible for this type of data. I mean, you will at the end of the day need some type of clever retrieval mechanism when dealing with such type of data.

Today's long-context LLMs work great with static information (books, video recordings, PDFs, etc.) but they are yet to be battle-tested on highly evolving information and knowledge. We will make progress towards addressing some of the challenges like "lost in the middle" and handling more complex structured and dynamic data but we still have a long way to go.

To tackle these types of problems you could potentially combine RAG and long-context LLMs to build a robust system that effectively and efficiently retrieves and performs large-scale analysis of key historical information. And not even this is enough for lots of scenarios. AI-based agents add even more complexity especially because a lot of data could be changing rapidly. Properly combining all these ideas is highly likely what will happen with complex use cases as opposed to general-purpose or long-context LLMs replacing everything.

And there are plenty of examples like this. You can't forget the fact that not all data is static, a lot of it is dynamic. When thinking of these applications it's always good to remember the 3 Vs of big data: velocity, volume, and variety. This is a lesson I learned while working at, wait for it, a search company. In my opinion, different families of LLMs will help solve different types of problems. We need to move on from this idea that there will be one LLM that will rule all.

I welcome other opinions about this but most of the takes I've seen are super dismissive without concrete examples or evidence.

Lastly, I would like to end this post with a quote from @OriolVinyalsML (VP of Research & DL Lead at Google Deepmind): "RAG ("retrieval-augmented generation") isn’t done for, even though we can do 1M or more tokens in context now. In fact, RAG has some nice properties that can enhance (and be enhanced by) long context. RAG allows you to find relevant information, but the way the model accesses it may be too restrictive due to compression. Long context may help bridge that gap, similar to how L1/L2 cache & main memory work together in modern CPUs."

[Žygimantas Medelis @zzgmm]

@eating_entropy @phronmophobic Thanks for mentioning Bosquet! The project is a bit in the woods now. Ran into complex issues trying to support non-basic graph LLM generation cases. More or less like DSPy does.

[Žygimantas Medelis @zzgmm]

An exploration into LLM Red Teaming:
1. How is LLM jailbreaking defined?
2. What are the motivations and goals of those seeking ways to make LLMs misbehave?
3. What strategies are used to red team?
https://t.co/ev52pMOefm

[Žygimantas Medelis @zzgmm]

Phone numbers, person names included.

[Žygimantas Medelis @zzgmm]

Want to know if a phrase is present in one of the big data sets used for AI training? AllenAI (now making the news with OLMo) has 'What's In My Big Data?' WIMBD toolset that can search for phrases or domains in those datasets. https://t.co/RjSnnwuq1u

[Linus @thesephist]

Small rant about LLMs and how I see them being put, rather thoughtlessly IMO, into productivity tools. 📄

TL;DR — Most knowledge work isn't a text-generation task, and your product shouldn't ship an implementation detail of LLMs as the end-user interface

https://t.co/N1NJoKzUqA

[Žygimantas Medelis @zzgmm]

Done with Clojure LLM tooling available at https://t.co/dAY8O3JiB8

[Žygimantas Medelis @zzgmm]

My talk on implementing 'Large Language Models as Optimizers' paper. It covers dealing with embeddings, Qdrant as a vector store, large documents, content splitting, evaluation, and setting up complex LLM flows.

https://t.co/n8XoTBpWgr

[Scicloj @scicloj]

On Friday, we'll have the next talk of our #LLM meetup series. @zzgmm will demonstrate how recent research papers can be elegantly implemented using #Bosquet in #Clojure.

https://t.co/0V995yuWb8

[Žygimantas Medelis @zzgmm]

Sunday tinkering to help with HuggingFace 🤗 dataset access from Clojure https://t.co/ufvjVdhNX3

[Žygimantas Medelis @zzgmm]

@mantasurbonas Who knows, maybe they are linked :)
Two things CoD researchers note:
1. Variants #2-3 are usually the best
2. It is aimed at news. Doing it on a rich Wiki article is probably too much

[Žygimantas Medelis @zzgmm]

Following the CoVe implementation another chain this time on text summarization task. Chain of Density is a neat technique to produce good summaries. It is interesting to track how summarization density increases as the LLM produces versions of summaries. https://t.co/Dbthb3iCqR

[Žygimantas Medelis @zzgmm]

Chain of Verification is a multi-step prompting technique addressing the issue of LLM hallucinations. CoVe can be declared and executed in Bosquet without custom coding and piping https://t.co/zsSemm0KSN

[Žygimantas Medelis @zzgmm]

@mantasurbonas Ačiū! :)

[Žygimantas Medelis @zzgmm]

Chain of Verification is a multi-step prompting technique addressing the issue of LLM hallucinations. CoVe can be declared and executed in Bosquet without custom coding and piping https://t.co/zsSemm0KSN

[Žygimantas Medelis @zzgmm]

LLMs are not great at generating complex structured data. They will often be a lot or a bit off. Even having 50.0% instead of 50 in a financial doc might be misleading. But finetuned smaller models like Llama-7B will outperform big ones in this task. https://t.co/JvQiA1MKyk

[Žygimantas Medelis @zzgmm]

@bmorphism @scicloj @cjbtweetz @danielsz @phronmophobic I have the Outlines tab open forever :) Planning at some point to get to some form of guided generation with Bosquet.