Purak Jain

Engineer; Researcher | Marketing Science

[Bo @bo_wangbo]

today i learned something very interesting at ICLR, a term called order of evidence.

During RAG, we send the top retrieved documents as evidences and prompting them to LLMs. Normally prompting is based on cosine similarity: top match always being prompted first.

However, different LLMs might have difference "preference" on the order of evidence:

1. ChatGPT prefers top evidence, which is good.
2. GPT4 has no preference on evidence order, which means the similarity score is not being considered, you only need to decide top-K, and all evidences will be equally treated.
3. Surprisingly, Llama2 and PaLM prefers last order evidence, so you need to reverse the rank list then prompt to the LLM :)

https://t.co/fHRR0FGO26

[Werner Vogels @Werner]

Summarizing meeting notes and action items is compounding. AI can augment these skills and reduce overhead. Today, I’m open-sourcing a simple app my team has built (with help from Q) to summarize our discussions and track deliverables: https://t.co/LeBNKvxZPa #Bedrock #AmazonQ

[⿻ Andrew Trask @iamtrask]

A long-suspected theory: LSTMs are superior to Transformers but we didn't know how to scale them as well — so Transformers pulled ahead because they were easier to scale.

Enter xLSTM... very interesting work.

Key diagrams below

[DJ @DuaneJRich]

I come back to this a lot

[Simo Ryu @cloneofsimo]

https://t.co/mnhV35F7gi

Yes. Yes!!!! Everyone read this material three times!

[Shreya Shankar @sh_reya]

gave a talk at last week's Stanford MLSys seminar about evaluating LLM pipeline output quality, especially for custom tasks at scale: https://t.co/LY5uIIvWwG

if your lab is interested in data management x LLMs x HCI, i would love to give this talk & get your feedback! 🙏

[Chris Botica @chrisbotica]

@ns123abc @ID_AA_Carmack i think there were actually like 30 delivered but here there are https://t.co/xeDgQjZOGj

[Fred Jonsson @enginoid]

since I'm deeply immersed in evals right now (and the process of building them) I got a kick out of this paper from @sh_reya @jdzamfi @bjo3rn @adityagp @IanArawjo

it addresses the challenge of time-efficiently coming up with evals that are aligned with practitioners

some thoughts:

[Benjamin Warner @ ICML @benjamin_warner]

I've written up my study group lectures on implementing Transformers in PyTorch into a blog series:

Creating Transformers from Scratch:

- Part 1: The Attention Mechanism https://t.co/ZCMiCMzQIp

- Part 2: The Rest of the Transformer https://t.co/kWq9gWkvtp

[Shreya Shankar @sh_reya]

hard launching a new HCI paper on LLM evals tomorrow 🚀 (tldr: coming up with custom evals you can trust is _really_ hard, and our paper shares a number of reasons for why)

[Jeremy Howard @jeremyphoward]

This video from @UmerHAdil is *the* best resource I've seen for learning OpenAI Triton -- and one of the best deep tech tutorials of any kind I've seen.
https://t.co/83na4LcGlI

[Michael Pyrcz🌻 @GeostatsGuy]

I'm teaching spatial, stochastic simulation in my #DataAnalytics course.

To help my students make the leap from spatial estimation to spatial simulation I built out this #Python @matplotlib dashboard with my sequential Gaussian simulation codes on #GitHub @ https://t.co/9zmTEaxm8J ∀.

Students learn by playing with #DataScience methods!

[Lance Martin @RLanceMartin]

LLM app development is rate-limited by quality evals. There's a paradox of LLM, prompt, etc choice. Recently put together a short guide on setting up custom evals.

Playlist (5 min vids):
https://t.co/uEoNaloBCH

Code:
https://t.co/CGIqhRhOK6

https://t.co/kOMIOiSu0p

[Gustavo Cid @_cgustavo]

Prototyping is easy. Shipping high-quality systems is hard.

One of the most helpful papers for AI practitioners is @sh_reya et al. “Operationalizing ML: an interview study.”

Lots of learnings from many teams are distilled into a single paper
https://t.co/21S24slEcu

[Lance Martin @RLanceMartin]

RAG From Scratch

Here's a set of short (5-10 min videos) and notebooks explaining > a dozen of my favorite RAG papers. Took a stab at implementing each idea myself (all code open source) and grouped according to the diagram.

Repo:
https://t.co/WDN9Tsy84t
Video playlist:
https://t.co/ctSTXXWJ49
Some highlights:

Is RAG Really Dead?
How RAG might change with long context LLMs.
Video:
https://t.co/z1ntvtWFWL

Adaptive-RAG
Dynamically route queries based on complexity to different RAG approaches. Implemented in LangGraph w/ @cohere cmd-R.
Video:
https://t.co/3hKVdSQ0Zf
Code:
https://t.co/KjcqtCL0bW
Paper (@SoyeongJeong97 et al):
https://t.co/4bIO5km1jn

Corrective-RAG
Self-correct retrieval errors in-the-loop unit tests for doc relevance and fallback to web-search. I implemented in LangGraph w/ @MistralAI-7b + @ollama for running locally.
Video:
https://t.co/8T06OVnmW4
Code:
https://t.co/jATwYQuqaN
Paper (@Jiachen_Gu et al):
https://t.co/nBO5ZxWfvU

Self-RAG
Self-correct RAG errors with in-the-loop unit tests for doc relevance, answer hallucinations, and answer quality. Implemented in LangGraph w/ @MistralAI-7b + @ollama for running locally.
Code:
https://t.co/7bEwbcY9jv
Code (local):
https://t.co/BQZ2LxEHJ3
Paper (@AkariAsai et al):
https://t.co/lTIlj4rpoT

Query Routing
Various approaches for directing questions to the correct datasource (e.g., logical, semantic, etc).
Video:
https://t.co/Ffg7Dyx6jA
Code:
https://t.co/nMLCCYEOzB

Query Structuring
Use an LLM to convert from natural language-to-<DSL> where DSL is a domain specific language required to interact with a given database (SQL, Cypher, etc).
Video:
https://t.co/LQB5zVdA2R
Code:
https://t.co/nMLCCYEOzB
Blog:
https://t.co/Z5ynzlzmWK
2/ Deep dive on graphDBs (c/o @neo4j): https://t.co/4FvtlQXl7F
3/ Query structuring docs: https://t.co/ZcgucoC4jf
4/ Self-query retriever docs: https://t.co/FBZ7OlwFCp

Multi-Representation Indexing
Use an LLM to produce document summaries ("propositions") that are optimized for retrieval. Embed these summaries for similarity search, but return full documents to the LLM for generation.
Video:
https://t.co/XU44LDd3X7
Code:
https://t.co/102QR15FSb
Paper (@tomchen0 et al):
https://t.co/ltzx6n0BUy

RAPTOR
Cluster docs in the corpus and summarize similar ones recursively. Index them all together, resulting in lower-level docs and summaries that can be retrieved to answer questions that span detailed-to-higher level.
Video:
https://t.co/bzLQRRRtQk
Code:
https://t.co/SXqOLhxAgk
Paper (@parthsarthi03 et al):
https://t.co/0aSDyiU24H

ColBERT
Improve embedding granularity w/ a contextually influenced embedding for each token in the document and query.
Video:
https://t.co/JZ6VTl24kV
Code:
https://t.co/102QR15FSb
Paper (@lateinteraction & @matei_zaharia):
https://t.co/MuTGFlAerv

Multi-Query
Re-write the user question from multiple perspectives, retrieve documents for each re-written question, return the unique documents for all queries.
Video:
https://t.co/WiHg60yDC4
Code:
https://t.co/cxsjcvPO9a
Paper:
https://t.co/GCH3lgUjT0

RAG-Fusion
Re-write the user question from multiple perspectives, retrieve documents for each re-written question, and combine the ranks of multiple search result lists to produce a single, unified ranking w/ Reciprocal Rank Fusion (RRF).
Video:
https://t.co/DWlBCu2duf
Code:
https://t.co/cxsjcvPO9a
Repo (@Raudaschl):
https://t.co/zn5iZRNaYg

Decomposition
Decompose a question into a set of sub-problems / questions, which can either be solved sequentially (use the answer from first + retrieval to answer the second) or in parallel (consolidate each answer into final answer). Various works such as Least-to-Most prompting (@denny_zhou et al) and IR-CoT present ideas that be utilized.
Video:
https://t.co/VzxTWhnNoD
Code:
https://t.co/cxsjcvPO9a
Papers:
https://t.co/ZGRt9Gqid3 https://t.co/JhGK1vHwro

Step-back prompting
First prompt the LLM to ask a generic step-back question about higher-level concepts or principles, and retrieve relevant facts about them. Use this grounding to help answer the user question.
Video:
https://t.co/JOjfkghYC2
Code:
https://t.co/cxsjcvPO9a
Paper (@denny_zhou + colleges):
https://t.co/ZIc6uaNBDx

HyDE
LLM to convert questions into hypothetical documents that answer the question. Use the embedded hypothetical documents to retrieve real documents with the premise that doc-doc similarity search can produce more relevant matches.
Video:
https://t.co/Kiy2EFCRBT
Code:
https://t.co/cxsjcvPO9a
Paper:
https://t.co/wDCkSwuK3o

[Ben Smith @bensmithlive]

The most dangerous modern diseases aren't:

Depression. Obesity. ADHD.

This one disease is the worst. And it causes the three above.

Here it is (and how to protect yourself):

[Vipul Vaibhaw @vaibhaw_vipul]

Andrej Karpathy talking about -
- rarity of talents in distributed computing with GPUs.
- the opportunities in computer architecture because von Neumann architecture may not be optimal
- progress in precision
- sparsity in neural networks

[Hrishi @hrishioa]

https://t.co/jylXz7Zb31

leave this here