Usama Jamil @UsamaJamil_dev - Twitter Profile

With all the buzz around Generative AI right now, what's your take? Do you see it as just a passing phase, or do you think it will bring lasting changes to how we work and live? #generatieveai #llms #chatgpt #openai #sora #gemini

0

107

Usama Jamil @UsamaJamil_dev

over 2 years ago

@umarsaif While True: Look busy do nothing, block Twitter

0

20

Usama Jamil @UsamaJamil_dev

over 2 years ago

Read MyScale's new article about filtered vector search, as well as the technologies behind implementing it. It includes #preFiltering and #postFiltering with row and column storage. https://t.co/bV2Vdsn3Ll #rag #LLMs #OpenAI #Sora #GPT5 #GenerativeAI

0

1

0

161

UsamaJamil_dev retweeted

Cameron R. Wolfe, Ph.D.

@cwolferesearch

over 2 years ago

Retrieval augmented generation (RAG) was proposed in 2020, but the idea has since been explored and expanded by a variety of papers. Here are four notable publications that study advanced concepts with RAG… (0) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: This paper proposes RAG. To retrieve relevant context, authors use the DPR bi-encoder. Unlike recent works that add RAG to decoder-only LLMs, this formulation of RAG uses encoder-decoder transformers and directly finetunes these models based on examples of high-quality output. However, only the transformer and the query encoder for DPR are finetuned (i.e., the document encoder is not) to avoid constantly rebuilding the search index. (1) How Context Affects Language Models' Factual Predictions: Pretrained LLMs have factual information encoded within their parameters, but they struggle with reliably manipulating this parametric knowledge—RAG mitigates this issue. However, early approaches to RAG use a supervised approach for RAG that directly trains the model to leverage this context. In this work, authors propose an unsupervised RAG approach, which looks more similar to most RAG pipelines used today, that uses a pretrained retrieval mechanism and generator. Such an unsupervised approach is still found to significantly benefit the model’s generations. (2) Gorilla: Large Language Models Connected with Massive APIs: Combining language models with external tools is a popular topic in AI research, but LLMs are usually taught to use a small/fixed set of tools (e.g., calculator and search engine). In this work, authors develop a retrieval-based finetuning strategy, which looks very similar to the originally-proposed RAG pipeline, to teach an LLM called Gorilla how to use over 1,600 different deep learning model APIs. By finetuning the LLM to leverage this retrieval mechanism, we see that the model becomes highly capable of generating API calls based off of documentation that is retrieved via RAG. (3) Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: This paper studies the concept of knowledge injection (i.e., incorporating data from an external dataset into an LLM). There are two primary methods of knowledge injection: 1. Finetuning (i.e., continued pretraining) 2. RAG We see in this paper that RAG is far more effective than finetuning from the perspective of knowledge injection. In fact, using RAG alone often outperforms a combination of finetuning and RAG! (4) RAGAS: Automated Evaluation of Retrieval Augmented Generation: One of the most difficult aspects of RAG is figuring out how to measure its performance. This is because there are many dimensions to the performance of RAG—accurate retrieval, properly using data that’s retrieved, and generating the correct output. In this paper, authors propose an automated evaluation framework, called RAGAS, that evaluates RAG based on three factors: - Faithfulness: the answer is grounded in the given context. - Answer relevance: the answer addresses the provided question. - Context relevance: the retrieved context is focused and contains as little irrelevant information as possible. Together, these metrics holistically characterize the performance of any RAG pipeline. Additionally, we can evaluate them in an automated fashion by prompting powerful foundation models like ChatGPT or GPT-4.

cwolferesearch's tweet photo. Retrieval augmented generation (RAG) was proposed in 2020, but the idea has since been explored and expanded by a variety of papers. Here are four notable publications that study advanced concepts with RAG…

(0) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: This paper proposes RAG. To retrieve relevant context, authors use the DPR bi-encoder. Unlike recent works that add RAG to decoder-only LLMs, this formulation of RAG uses encoder-decoder transformers and directly finetunes these models based on examples of high-quality output. However, only the transformer and the query encoder for DPR are finetuned (i.e., the document encoder is not) to avoid constantly rebuilding the search index.

(1) How Context Affects Language Models' Factual Predictions: Pretrained LLMs have factual information encoded within their parameters, but they struggle with reliably manipulating this parametric knowledge—RAG mitigates this issue. However, early approaches to RAG use a supervised approach for RAG that directly trains the model to leverage this context. In this work, authors propose an unsupervised RAG approach, which looks more similar to most RAG pipelines used today, that uses a pretrained retrieval mechanism and generator. Such an unsupervised approach is still found to significantly benefit the model’s generations.

(2) Gorilla: Large Language Models Connected with Massive APIs: Combining language models with external tools is a popular topic in AI research, but LLMs are usually taught to use a small/fixed set of tools (e.g., calculator and search engine). In this work, authors develop a retrieval-based finetuning strategy, which looks very similar to the originally-proposed RAG pipeline, to teach an LLM called Gorilla how to use over 1,600 different deep learning model APIs. By finetuning the LLM to leverage this retrieval mechanism, we see that the model becomes highly capable of generating API calls based off of documentation that is retrieved via RAG.

(3) Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: This paper studies the concept of knowledge injection (i.e., incorporating data from an external dataset into an LLM). There are two primary methods of knowledge injection:

1. Finetuning (i.e., continued pretraining)
2. RAG

We see in this paper that RAG is far more effective than finetuning from the perspective of knowledge injection. In fact, using RAG alone often outperforms a combination of finetuning and RAG!

(4) RAGAS: Automated Evaluation of Retrieval Augmented Generation: One of the most difficult aspects of RAG is figuring out how to measure its performance. This is because there are many dimensions to the performance of RAG—accurate retrieval, properly using data that’s retrieved, and generating the correct output. In this paper, authors propose an automated evaluation framework, called RAGAS, that evaluates RAG based on three factors:

- Faithfulness: the answer is grounded in the given context.
- Answer relevance: the answer addresses the provided question.
- Context relevance: the retrieved context is focused and contains as little irrelevant information as possible.

Together, these metrics holistically characterize the performance of any RAG pipeline. Additionally, we can evaluate them in an automated fashion by prompting powerful foundation models like ChatGPT or GPT-4.

4

373

88

334

30K

UsamaJamil_dev retweeted

Antonio Vieira Santos

@AkwyZ

over 2 years ago · Cork

Large language models (LLMs) are infiltrating the medical field. One in 10 doctors already use ChatGPT in day-to-day work, and patients have taken to ChatGPT to diagnose themselves. A 4-year-old boy, Alex, whose chronic illness was diagnosed by ChatGPT after over a dozen doctors failed to do so. #AI https://t.co/e9ufD3Fqjn

AkwyZ's tweet photo. Large language models (LLMs) are infiltrating the medical field. One in 10 doctors already use ChatGPT in day-to-day work, and patients have taken to ChatGPT to diagnose themselves. A 4-year-old boy, Alex, whose chronic illness was diagnosed by ChatGPT after over a dozen doctors failed to do so. #AI https://t.co/e9ufD3Fqjn

1

6

2

0

282

Usama Jamil @UsamaJamil_dev

over 2 years ago

From getting out of the field being injured yesterday and conquering the next day What a Player Shamar Joseph is <3 #AUSvWI

0

2

0

71

UsamaJamil_dev retweeted

MrBeast

@MrBeast

over 2 years ago

I’m gonna give 10 random people that repost this and follow me $25,000 for fun (the $250,000 my X video made) I’ll pick the winners in 72 hours

377K

2M

3M

63K

284M

Usama Jamil @UsamaJamil_dev

over 2 years ago

@anwaar_kakar There is not a single reason to live here except for my parents.

0

1

0

27

Usama Jamil @UsamaJamil_dev

over 2 years ago

I heard yesterday that my unconstitutional PM @anwaar_kakar was saying, "Go abroad, get some higher education, come back, and serve this country in a better way. This is not a brain drain" Why should I return? So that I can work under these 12th-pass Majors or Generals? #Shame

1

3

0

48