Prompt Engineering

Curation account about Prompt engineering, a new branch based on language models like GPT-4, DALL-E 2 and LLaMa. Photo by @pollinations_ai

[Dario Garcia-Gasulla @dariogargas]

Main lesson learnt while doing research on synthetic content detection:

This is an unsolved problem that we may not solve in the short-mid term. Learn (and teach) to be cautious and critical of any digital content!

[Rohan Paul @rohanpaul_ai]

Wild idea in this paper 🤯

Current LLMs are stateless between tokens which lead to many many problems requiring reasoning across tokens (like unreliable CoT). Even a tiny bit of memory should help.

How might we store knowledge affordably yet comprehensively? Memory³ proposes an intriguing method - compressing factual data separately. Introduces a third form of memory in addition to the implicit knowledge stored in model parameters and the short-term working memory used during inference (context key-values).

👨‍🔧 LLMs struggle with inefficient knowledge storage and retrieval, leading to high training and inference costs. The paper aims to address this by introducing a more efficient memory format.

📌 Memory3 introduces explicit memory as a third memory format for LLMs, alongside model parameters (implicit memory) and context key-values (working memory). This explicit memory is implemented as sparse attention key-values, allowing for more efficient knowledge storage and retrieval.

📌 Defines a memory hierarchy for LLMs: plain text (RAG) → explicit memory → model parameters. As you move up this hierarchy, write cost increases while read cost decreases. The goal is to optimize knowledge placement across this hierarchy based on usage frequency.

📌 Memory3's architecture involves converting reference texts into explicit memories before inference. During inference, these memories are retrieved and integrated into self-attention layers. This design allows for smaller model size while maintaining performance.

📌 The explicit memory format uses intense compression to save space. It selects only the first half of attention layers as memory layers, uses grouped query attention to reduce key-value heads, and selects only 8 out of 128 tokens for each key-value head based on attention weights.

📌 The training process involves a two-stage approach: a warmup stage without explicit memory, followed by a continual train stage with explicit memory. This approach was necessary as starting with explicit memory from the beginning rendered the memories useless.

📌 Introduces a "memory circuitry theory" to formalize the concept of knowledge in LLMs. It defines knowledge as circuits (equivalence classes of subgraphs) in the computation graph, categorizing them as specific or abstract knowledge.

📌 The Memory3 model achieved better performance than larger models and RAG models on various benchmarks, while maintaining higher decoding speed. It showed particular improvements in factuality and reduced hallucination.

[Stanislas Polu @spolu]

Super interesting work from DeepSeek on MiniF2F (so happy to see our benchmark still in use \o/).

It's hard to compare this with the recent DeepMind paper but from my experience building and using MiniF2F I think ~60% pass-rate is likely comparable to DeepMind's recent result on formal IMOs.

The take-aways of the paper are super interesting:

TreeSearch: bumps from 60% (one pass generation) to 63%... not super huge. This is aligned with what I saw while working on it between 2019 and 2022. What matters is not how you test-time compute but rather how much you test-time compute.

Similar to what many people have explored, the DeepSeek model introduces tactic state and informal statement in the generative objective. The former helps the model stick with the formal system state. The latter helps the model explore strategies that are not obvious when operating in the purely formal state (an approximate informal proof can help you find an interesting cut to introduce in the formal proof).

What would I would have loved to see in the paper is:

(i) a plot of figure 3 in log-scale to confirm it's a straight line (should be until you start saturating the benchmark)

(ii) an ablation of model sizes. hot-take: A dirty little unpublished secret of AI research is that when you have perfect verification (which is the case for formal maths) test-time compute is all that matters within a very large range of model sizes: you can sample more from a smaller model and get roughly the same results as you would get from sampling less from a larger model. If confirmed of course this is bad for AGI hence why no-one dares publishing about it I guess? :p

One thing that I've never seen, but maybe I just missed it, is a very aggressive optimization of the sampling while interacting with the formal system: sample tactic, keep the cache hot, add the tactic state to the context and continue sampling. Basically interact with the formal system inside the auto-regressive sampling loop. This prevents from going back and forth to the model with a large context and should squeeze more test-time compute out of models + if well implemented would allow back-tracking in case of errors k-times during one step generation efficiently. A simpler more efficient and more elegant approach compared to tree search. Question here: do they keep each node of the tree search hot in the model KV cache?

Regardless, this is a massive achievement. 60% on MiniF2F is beyond what were my wildest dreams just 2 years ago. MiniF2F, despite its name, is. a. challenging. benchmark.

What are the takeaways at a high level?
- RL to better align the models with the formal systems seems to be the most massive bump here. So the models were doing silly formalization mistakes all along?
- In light of this result and the DeepMind result there is a great avenue of research in front of the community in places where we have either (a) perfect (good?) verification or (b) guaranteed valid data generation strategies (the two are kind of one and the same if you squint)

hot-take 2: I thiiiiiiiinnk @arcprize is an interesting target here for these reasons:
- you can generate code to infer output from input
- you can verify the code is correct on the training examples of the task
- you can perform RL using the exact same GRPO approach on failed attempts
- Remember Hindsight experience replay🤯

[Soumith Chintala @soumithchintala]

I worked on Metamate last year with Aparna Ramani and Zach Rait, secured funding, brought together the initial team (@Vjeux @bolinfest, etc.) built out the ML side of things with @shahin_sefati .
It's been really cool to build a vertical GenAI product for Meta-internal productivity, considering a large company has so many internal-specific ways of doing things.
Apart from the base Perplexity-like experience, people and teams build custom agents in a Python-like scripting language in-browser, focusing the agent on team-specific knowledge or systems -- like oncall-bots, etc.

[Dan Hendrycks @hendrycks]

NVIDIA gave us an AI pause.
They rate limited OpenAI to create a neck-and-neck competition (OpenAI, xAI, Meta, Microsoft, etc.). For NVIDIA, each new competitor is another several billion in revenue. Because of this, we haven't seen a next-generation (>10^26 FLOP) model yet.

[Alvaro Cintas @dr_cintas]

I turned Llama 3.1 into my new phone assistant.

It can answer anything, and look at how fast it does it using Groq’s API 🤯

[Jeff Dean @JeffDean]

We're starting to roll out API support for Gemini 1.5 Pro for developers. We're excited to see what you build with the 1M token context window!

We'll be onboarding people to the API slowly at first, and then we'll ramp it up. In the meantime, developers can try out Gemini 1.5 Pro in the AI Studio UI right now:

https://t.co/HG3inVgrc7

[Julian Togelius @togelius]

The most exciting thing I've seen in deep learning for a while. Very interested in where this could go.

[AK @_akhaliq]

Hugging Face presents Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a

[Emad @EMostaque]

Happy to say you can now do this with the brand new features @StabilityAI API 🤠

1. Search and replace
2. Editing, with inpaint
3. Creative upscaling up to 4k
4. Stable Video

More releases to come! 🛳️

https://t.co/gXamC0mbx2

[Emad @EMostaque]

Maybe crown needs to be more thorny

[Emergent Mind @EmergentMind]

GLAN significantly enhances LLMs' instruction-following by generating synthetic data from a pre-curated human knowledge taxonomy, showing superior performance across key domains: https://t.co/2aEYfqmUAS

[Lykon @Lykon4072]

Hand positions that are a pain to get on other models are working 99% of the time on #SD3. And in multiple styles.

[Prompt Engineering @prompteng]

We are close to 1.000 followers :) Please share this to reach it sooner!

[AK @_akhaliq]

Microsoft presents UFO

A UI-Focused Agent for Windows OS Interaction

paper page: https://t.co/OwBTddqqxV

introduce UFO, an innovative UI-Focused agent to fulfill user requests tailored to applications on Windows OS, harnessing the capabilities of GPT-Vision. UFO employs a dual-agent framework to meticulously observe and analyze the graphical user interface (GUI) and control information of Windows applications. This enables the agent to seamlessly navigate and operate within individual applications and across them to fulfill user requests, even when spanning multiple applications. The framework incorporates a control interaction module, facilitating action grounding without human intervention and enabling fully automated execution. Consequently, UFO transforms arduous and time-consuming processes into simple tasks achievable solely through natural language commands. We conducted testing of UFO across 9 popular Windows applications, encompassing a variety of scenarios reflective of users' daily usage. The results, derived from both quantitative metrics and real-case studies, underscore the superior effectiveness of UFO in fulfilling user requests. To the best of our knowledge, UFO stands as the first UI agent specifically tailored for task completion within the Windows OS environment.

[AK @_akhaliq]

ChatCell

Facilitating Single-Cell Analysis with Natural Language

paper page: https://t.co/9EkaFX1U9t

As Large Language Models (LLMs) rapidly evolve, their influence in science is becoming increasingly prominent. The emerging capabilities of LLMs in task generalization and free-form dialogue can significantly advance fields like chemistry and biology. However, the field of single-cell biology, which forms the foundational building blocks of living organisms, still faces several challenges. High knowledge barriers and limited scalability in current methods restrict the full exploitation of LLMs in mastering single-cell data, impeding direct accessibility and rapid iteration. To this end, we introduce ChatCell, which signifies a paradigm shift by facilitating single-cell analysis with natural language. Leveraging vocabulary adaptation and unified sequence generation, ChatCell has acquired profound expertise in single-cell biology and the capability to accommodate a diverse range of analysis tasks. Extensive experiments further demonstrate ChatCell's robust performance and potential to deepen single-cell insights, paving the way for more accessible and intuitive exploration in this pivotal field.

[Simon Willison @simonw]

Here's how the new "memory" feature of ChatGPT works: it uses a new tool called "bio" described in the system prompt like this:

[Nick St. Pierre @nickfloats]

Friendly reminder that --v 6 is still not the default in Midjourney

So if you haven't gone to /settings and selected --v 6 from the dropdown menu, you still might be running things in --v 5.2

Once you select it --v 6 will automatically get appended to the end of your prompt 💪

[Estado Latente @EstadoLatente]

𝗔𝗜 𝗕𝗥𝗔𝗡𝗗𝗜𝗡𝗚 nuestro #𝗚𝗣𝗧 de #inteligenciaartificial al servicio del #branding. Una inspiración para co-crear con #IA los activos estratégicos y creativos de tu marca:
❤️‍🔥
https://t.co/gQq32KVYSX
❤️‍🔥
#Marketing #ArtificialIntelligence #ChatGPT #OpenAI @ruthfalquina

[Bindu Reddy @bindureddy]

Building smart AI agents over structured and unstructured data involves several components.

Ideally, you want to connect all your data sources, use an LLM of your choice, and build smart agents that can automate a number of processes and increase employee productivity.

You can use an LLMOps platform like Abacus AI to complete this in a few hours.