With all the buzz around Generative AI right now, what's your take?
Do you see it as just a passing phase, or do you think it will bring lasting changes to how we work and live?
#generatieveai#llms#chatgpt#openai#sora#gemini
Retrieval augmented generation (RAG) was proposed in 2020, but the idea has since been explored and expanded by a variety of papers. Here are four notable publications that study advanced concepts with RAG…
(0) Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: This paper proposes RAG. To retrieve relevant context, authors use the DPR bi-encoder. Unlike recent works that add RAG to decoder-only LLMs, this formulation of RAG uses encoder-decoder transformers and directly finetunes these models based on examples of high-quality output. However, only the transformer and the query encoder for DPR are finetuned (i.e., the document encoder is not) to avoid constantly rebuilding the search index.
(1) How Context Affects Language Models' Factual Predictions: Pretrained LLMs have factual information encoded within their parameters, but they struggle with reliably manipulating this parametric knowledge—RAG mitigates this issue. However, early approaches to RAG use a supervised approach for RAG that directly trains the model to leverage this context. In this work, authors propose an unsupervised RAG approach, which looks more similar to most RAG pipelines used today, that uses a pretrained retrieval mechanism and generator. Such an unsupervised approach is still found to significantly benefit the model’s generations.
(2) Gorilla: Large Language Models Connected with Massive APIs: Combining language models with external tools is a popular topic in AI research, but LLMs are usually taught to use a small/fixed set of tools (e.g., calculator and search engine). In this work, authors develop a retrieval-based finetuning strategy, which looks very similar to the originally-proposed RAG pipeline, to teach an LLM called Gorilla how to use over 1,600 different deep learning model APIs. By finetuning the LLM to leverage this retrieval mechanism, we see that the model becomes highly capable of generating API calls based off of documentation that is retrieved via RAG.
(3) Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs: This paper studies the concept of knowledge injection (i.e., incorporating data from an external dataset into an LLM). There are two primary methods of knowledge injection:
1. Finetuning (i.e., continued pretraining)
2. RAG
We see in this paper that RAG is far more effective than finetuning from the perspective of knowledge injection. In fact, using RAG alone often outperforms a combination of finetuning and RAG!
(4) RAGAS: Automated Evaluation of Retrieval Augmented Generation: One of the most difficult aspects of RAG is figuring out how to measure its performance. This is because there are many dimensions to the performance of RAG—accurate retrieval, properly using data that’s retrieved, and generating the correct output. In this paper, authors propose an automated evaluation framework, called RAGAS, that evaluates RAG based on three factors:
- Faithfulness: the answer is grounded in the given context.
- Answer relevance: the answer addresses the provided question.
- Context relevance: the retrieved context is focused and contains as little irrelevant information as possible.
Together, these metrics holistically characterize the performance of any RAG pipeline. Additionally, we can evaluate them in an automated fashion by prompting powerful foundation models like ChatGPT or GPT-4.
Large language models (LLMs) are infiltrating the medical field. One in 10 doctors already use ChatGPT in day-to-day work, and patients have taken to ChatGPT to diagnose themselves. A 4-year-old boy, Alex, whose chronic illness was diagnosed by ChatGPT after over a dozen doctors failed to do so. #AI https://t.co/e9ufD3Fqjn
I heard yesterday that my unconstitutional PM @anwaar_kakar was saying, "Go abroad, get some higher education, come back, and serve this country in a better way. This is not a brain drain" Why should I return? So that I can work under these 12th-pass Majors or Generals?
#Shame