Qianru Qi

Associate Professor of Finance@Telfer_uOttawa

[Inferam @InferamAI]

The correct way to roll a burrito

[Yann LeCun @ylecun]

A clip from our Davos panel in which Daphne Koller and I talk about the saturation of LLM performance, the exhaustion of text data, and the necessity of using sensory data to get AI systems to understand the world.
I make the point that training AI systems to understand the world from video will require the use of non-generative architectures because predicting video in pixel space is too hard. I describe the idea of Joint Embedding Predictive architectures without mentioning the name: predicting in abstract representation space.

[CALL TO ACTIVISM @CalltoActivism]

HOLY SHIT! Only 17 years old, yet he destroys Republicans better than any adult could.

Every second of this gives me hope.

[Daniel Mahncke @MnkeDaniel]

Li Lu is one of the most successful investors of the last decades. But there are almost no public videos of him.

But in 2006, he held a Valuation Class at Columbia.

Here's the Video:

[Bindu Reddy @bindureddy]

LLMs Applied to Designing mRNA Vaccines Today, Potentially Designer Babies Tomorrow!

The COVID-19 vaccine was an mRNA-based vaccine - these vaccines are mRNA sequences that encode proteins that mimic the viral protein and trigger an antibody response.

Different RNA sequences can encode the same protein, and picking the right one can help increase protein expression, stability, and immunogenicity.

CodonBERT (link in alt) is a large LLM that enables the selection of an optimal sequence,

Unlike prior models, CodonBERT uses codons (3 nucleotide sequences) as inputs, which enables it to learn better representations. CodonBERT was trained using more than 10 million mRNA sequences from a diverse set of organisms.

The resulting model captures important biological concepts. CodonBERT can also be extended to perform prediction tasks for various mRNA properties. CodonBERT outperforms previous mRNA prediction methods, including on a new flu vaccine dataset.

Today, LLMs are being used to design optimum mRNA sequences. You can extend these concepts to start designing optimum DNA sequences for specific genes.

Over the next few years, we will be able to create designer sperm and eggs where you can specify the preferred traits for our offspring and generate the optimum sequences for important genes, insert/recombine them into existing DNA to produce designer babies!

The crazy part is that some of this tech is already available! 🤯

TLDR; LLMs can be used to design mRNA today and can be used to design optimal gene sequences tomorrow

[AI at Meta @AIatMeta]

Introducing 'Prompt Engineering with Llama 2' — an interactive guide covering prompt engineering & best practices for developers, researchers & enthusiasts working with large language models.

Access the notebook in the llama-recipes repo ➡️ https://t.co/TbLWc7xlD5

[Bindu Reddy @bindureddy]

AI is going to make us much smarter! This graph showcases the steep increase in decision quality amongst professional Go players, right after Alpha Go launched.

They are not only playing against the model and learning from the AI but they are also apparently inventing new moves!

When challenged by a strong and creative AI model, human creative juices kick into full gear, and we become more creative as well!

Similarly, LLMs are going to cause a step change in our abilities. It's about AI fighting us; it's about AI enabling us.

Instead of AI being superhuman intelligence, AI may just make us superhumans!! 💕

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

The volume of LLM research being released is staggering. Although there are too many new papers for any one person to read, this work can be largely distilled into a much smaller set of overlapping themes. Recently, there are three trends in LLM research that have been especially impactful…

(1) Synthetic training data: Using LLMs to generate their own training data has been a topic of interest for a long time. Examples of this include Constitutional AI, Orca, RLAIF, and Evol/Self-Instruct. However, this topic is currently experience an explosion of interest within the AI research community, resulting in several publications:
- In [1], authors show that synthetic training data can be used to train state-of-the-art embedding models.
- We see in [2] that synthetic data can be easily generated and verified for math and coding problems, which can then be used to improve the performance of LLMs.

(2) LLM safety: Since the proposal of GPT-2, safe deployments have been a priority in the development of LLMs (i.e., GPT-2 weights weren’t released to the public due to safety concerns!). Although the AI community seems to be more willing to take risks in the deployment of LLMs, safety remains a top priority for many labs. However, we have learned in recent research that ensuring the safety of LLM deployments is extremely difficult:
- Work in [3] shows that backdoor attacks trained into an LLM persist even after extensive safety training, forming sleeper agents that can deceive human users.
- We learn in [4] that training data can be extracted from nearly all LLMs, even those that have underwent lots of alignment, given the proper prompting technique.

(3) Knowledge injection: Nearly every business is interested in training an LLM over their own internal/proprietary data (e.g., BloombergGPT, EinsteinGPT, ShopAI, and more). However, it is unclear how we can best specialized a pretrained LLM over a domain-specific knowledge base. In [5], authors perform an extensive comparison of fine-tuning and RAG for this purpose finding that i) teaching an LLM new information via finetuning is very difficult but ii) RAG is incredibly capable at injecting knowledge into an LLM. This topic has also been extensively researched in the past:
- Authors in [6] propose retrieval augmented generation (RAG), showing that this approach is impactful to performance on knowledge-intensive tasks.
- LIMA [7] shows that nearly all knowledge of an LLM is learned during pretraining.
- Phi-1 [8] demonstrates that knowledgeable LLMs can be trained over smaller, curated sets of curated data (i.e., textbooks).

--------
[1] Wang, Liang, et al. "Improving text embeddings with large language models." arXiv preprint arXiv:2401.00368 (2023).
[2] Singh, Avi, et al. "Beyond human data: Scaling self-training for problem-solving with language models." arXiv preprint arXiv:2312.06585 (2023).
[3] Hubinger, Evan, et al. "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training." arXiv preprint arXiv:2401.05566 (2024).
[4] Nasr, Milad, et al. "Scalable extraction of training data from (production) language models." arXiv preprint arXiv:2311.17035 (2023).
[5] Ovadia, Oded, et al. "Fine-tuning or retrieval? comparing knowledge injection in llms." arXiv preprint arXiv:2312.05934 (2023).
[6] Lewis, Patrick, et al. "Retrieval-augmented generation for knowledge-intensive nlp tasks." Advances in Neural Information Processing Systems 33 (2020): 9459-9474.
[7] Zhou, Chunting, et al. "Lima: Less is more for alignment." arXiv preprint arXiv:2305.11206 (2023).
[8] Gunasekar, Suriya, et al. "Textbooks Are All You Need." arXiv preprint arXiv:2306.11644 (2023).

[Jerry Liu @jerryjliu0]

🪜 4 Levels of Agents for RAG 🤖

There’s an entire spectrum of agentic capabilities offering simple-to-advanced reasoning on top of your data.

Here’s a mini-🧵 on the 4 different levels of agents and how they can augment your RAG pipeline 👇

1️⃣ Tool Use: A linear pipeline that can query a set of tools given a user query.

The simplest example is a router: https://t.co/hlwMh4QbDE

2️⃣ Reasoning Loop + Memory: Simple loop around a RAG pipeline. Given a user query and previous conversation history, infer a new query to query the pipeline.

We add a simple retry-layer on top of a text-to-SQL query engine: https://t.co/MRKrhorCVe

3️⃣ Reasoning Loop + Memory + Tool Use: A loop around a pipeline designed to pick a set of tools given a user query.

Here we combine reasoning loops with tool use. A simple example here is a retry layer on top of a router: https://t.co/vyGtpplbSl

4️⃣ Fancy Reasoning Loop + Tool Use + Memory: The reasoning loop can get arbitrarily complicated, from sequential ReAct chain-of-thought to LLMCompiler-style parallel planning and tool use.

ReAct from scratch: https://t.co/eGetiu0qj5

Check out our LLMCompiler implementation here: https://t.co/sp0qfpWo9f

[Nick Dobos @NickADobos]

Building gpts tip:

-Don't use knowledge files

They destroy prompt performance and remove's the ai's ability to coherently follow the system prompt

Why?
When you insert docs, the system copies snippets of your docs into the system prompt & context window

So 10 docs. Each around 4000 letters. FLOODS the entire context window

Which means your ai is now too stupid to remember your last message. Less memory than a goldfish

And now the system prompt is 10x as long, so your instructions get ignored

Workarounds & fixes:
Pink knowledge docs are available in the context and via RAG

Blue are only via code interpreter, and work MUCH MUCH better, especially as things scale up

-Use .md files, which are not compatible with knowledge retrieval, and open them by writing code, with code interpreter. This keeps them out of the system prompt context window, only fetching when needed. Dramatically improving overall performance

(Downside here is doc search becomes way worse. Better when you want the full doc RAG'ed in. Not good for summarizing.)

-DO NOT USE ALL 20 FILES, or if you do. DO NOT MAKE THEM LARGE. I had 10 files at ~4000 characters which destroyed all multi step reasoning. It would literally forget and repeat the first message in the conversation over and over again. Swapping from .txt to .md instantly fixed this

-Only include docs which include working system prompts for the ai. Do not simply chuck a doc in here and expect it work

[cem() @cemtuncelli]

Very simple open source UI to create agent based workforces for yourself.

Just drag and drop agents, connect them together, give them specific task, and run it on your own local.

Couple examples we have built:
- An email assistant that can read your emails, compose drafts, summarize
- A research workforce to find product ideas from research papers
- An engineering workforce to fix an issue from an open source project
- A workforce to finish homework :)

build yours @ https://t.co/f2VAEKVaij
contribute @ https://t.co/C7G5LgtuKi

Big thanks to:
@pyautogen @xyflowdev

[Eric Topol @EricTopol]

The ability to do continuous, high-resolution ultrasound imaging of an organ in people on the go through a wearable patch is an important innovation
https://t.co/FMxNNQ4Mbp @NatureBiotech @heart_lung

[GynaecologyUSS @GynaecologyUS]

Would you like to learn how to do a basic gynae transvaginal ultrasound? And within 30 mins ramp it up to scanning for endometriosis? That was today's lecture for @BSUroRad Here it is: https://t.co/f1DbzVbMlu

[Lars A. Kuehn @ProfLarsKuehn]

My latest paper "Learning about the Consumption Risk Exposure of Firms" is now in the JFE.
https://t.co/Golt770xEv
We find that higher perceived cash flow exposure to consumption risk depresses investments.

[New Scientist @newscientist]

After much neglect, women’s health must get the attention it deserves https://t.co/dn5U3KwO6b

[Eric Topol @EricTopol]

Non-invasive assessment of biological age.
Use of a transformer #AI multimodal model of images from the face, tongue and retina
https://t.co/vzvDtoamxt
"superior to biological age prediction models such as DNA methylation clock, transcriptome aging clocks and blood profiles"

[Ethan Mollick @emollick]

There are many careful studies that show a huge part of the US innovation advantage is government funding of university research, and spin-outs launched by students at universities. (And the international students that universities attract)

[MindBranches @MindBranches]

Here’s a great way to use ChatGPT to learn more about technology.

Pick a topic from the MindBranch below and insert it into the following prompt:

"Act as an expert educator in the field of [topic]. I'm seeking a comprehensive understanding of this subject and would like you to guide me through a detailed exploration, covering its foundational aspects, historical development, current applications, and future prospects. Please structure our interaction in the following manner:
Introduction: Begin with a concise overview of [topic], outlining its significance and core elements.
Historical Context: Summarize the evolution of [topic], noting key developments and how they've shaped its present state.
Key Concepts and Principles: Elucidate the fundamental concepts, theories, and principles of [topic]. Ensure each explanation builds logically on the last, providing a clear and structured learning path.
Current Applications: Describe the real-world applications of [topic] today, highlighting specific industries or technologies where it's particularly influential.
Challenges and Controversies: Address any challenges, ethical dilemmas, or controversies associated with [topic], including limitations, societal impacts, or philosophical debates.
Future Trends: Discuss the potential future of [topic], detailing emerging trends, possible advancements, and expected shifts in the field.
Further Learning Resources: Suggest resources for deeper learning, such as books, academic papers, online courses, or notable experts to follow.
Interactive Q&A: I'll pose questions about areas I wish to delve deeper into or need further clarification on. Please provide detailed responses and additional insights where necessary.
I look forward to an informative and comprehensive journey into [topic]. Let's begin!"

[CNBC @CNBC]

How Luckin Coffee overtook Starbucks as the largest coffee chain in China https://t.co/R0fK62Pd40