detectivelyw

Computer scientist & Anime lover

[detectivelyw @detectivelyw]

@benjamincowen Ben you chose a great match to watch!!

[detectivelyw @detectivelyw]

@benjamincowen Enjoy the match Ben!! 🤣

[Haihao Shen @HaihaoShen]

💡INC v3.3 released with enhanced FP8 and INT4 quantization performance on Intel platforms. Validated on most recent LLMs e.g., DeepSeekV3/R1, Falcon, Phi etc.
https://t.co/XklzQFSYdz🌟

[Sebastian Raschka @rasbt]

Just put together a short Jupyter notebook with tips and tricks for reducing memory usage when loading larger and larger models (like LLMs) in PyTorch: https://t.co/fEx2e8E7jS

(PS: This is an LLM example but the same concepts apply to any PyTorch model)

[Rohan Paul @rohanpaul_ai]

A cool Github repo collecting LLM papers, blogs, and projects, with a focus on OpenAI o1 and reasoning techniques.

[Ethan Mollick @emollick]

Wow: Just tried PaperQA, an open source AI-powered literature review whose research paper claims it achieves "superhuman synthesis of scientific knowledge"

I tested it against papers I wrote and it seems like the real deal, putting together a good summary with accurate details.

[elvis @omarsar0]

RAG vs. Long-Context LLMs

I have yet to see a convincing paper or technical blog showing that long-context LLMs can or will replace RAG.

So far I've seen specific long-context applications where long-context LLMs thrive and current retrieval benchmarks are not convincing.

This new paper reports that longer-context LLMs suffer from diminished focus on relevant information, which is one of the primary issues that a RAG system addresses (i.e., uses more relevant information).

They propose an order-preserving RAG mechanism that improves performance on long-context question answering.

It's not perfect and in fact as retrieved chunks increases the quality of responses go up and then declines. But there is a sweet spot where it can achieve better quality with a lot fewer tokens than long-context LLMs.

I am getting better results for RAG plus the added benefit of efficiency as well. Things could change rapidly but still not counting out RAG for now.

[Rohan Paul @rohanpaul_ai]

Evaluating your LLM applications with the right metrics is super important for production-class applications.

Roughly we have two types of evaluations

1. LLM model itself is evaluated across different tasks on its conciseness, relevance, coherence, faithfulness, etc.

2. RAG applications such as Contextual Recall and Answer Correctness

This is my quick list of open-source LLM evaluation repos

- DeepEval

- OpenAI SimpleEvals

- OpenAI Evals

- RAGAs

[Rohan Paul @rohanpaul_ai]

MASSIVE work by IBM Research. AutoToS: Automated version of Thought of Search 👏

Achieved 100% accuracy across all evaluated domains 🤯

AutoToS generates sound, complete search components from LLMs without human feedback, achieving high accuracy across domains.
Original Problem 🔍:
LLMs struggle with planning tasks, particularly in terms of soundness and completeness. Existing approaches sacrifice soundness for flexibility, while the Thought of Search (ToS) method requires human feedback to produce sound search components.

Key Insights 💡:

• LLMs can generate sound and complete search components with automated feedback

• Generic and domain-specific unit tests guide LLMs effectively
• Iterative refinement improves code quality without human intervention
• Smaller LLMs can achieve comparable performance to larger models
Solution in this Paper 🛠️:

•

• Components:
- Initial prompts for successor and goal test functions
- Goal unit tests to verify soundness and completeness
- Successor soundness check using BFS with additional checks
- Optional successor completeness check
• Feedback loop:
- Provides error messages and examples to LLM
- Requests code revisions based on test failures
• Extends search algorithms with:
- Timeout checks
- Input state modification detection
- Partial soundness validation
Results 📊:

• Minimal feedback iterations required (comparable to ToS with human feedback)

• Consistent performance across different LLM sizes
• Effective on 5 representative search/planning problems:
- BlocksWorld, PrOntoQA, Mini Crossword, 24 Game, Sokoban

[Henry Ndubuaku @Henry_Ndubuaku]

One of my mentees just completed his second project with me: SciKit-Jax: https://t.co/Fft45CVUtt

Scikit-Learn does not leverage GPUs by itself and offers less flexibility in the model design. Scikit-Jax solves these!

He will polish this to accommodate multi-GPU setups once we get to distributed computing in the mentorship plan.

[Sumit @_reachsumit]

HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extraction

Combines VectorRAG and GraphRAG techniques to improve information extraction from financial documents.

📝https://t.co/KHIDWy6NNe

[elvis @omarsar0]

Combining GraphRAG and VectorRAG leads to a HybridRAG system that outperforms both individually.

It was tested on a set of financial earning call transcripts. Combining the advantages of both approaches provides more accurate answers to queries.

https://t.co/lVow0hZUcH

[Ningyu Zhang@ZJU @zxlzr]

We will present EasyEdit tomorrow at #ACL24 @aclmeeting.

Location: Mon 11:00-12:30@Convention Center A1, Poster Session A

EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models (Demo)

Can't wait to connect and chat with everyone there! 🌟 #ACL2024 #AI #NLP #LLMs #KnowledgeEditing

[elvis @omarsar0]

Nice survey paper on current practices and solutions for LLM-based agents for software engineering.

Covers important topics such as requirement engineering, code generation, test generation, and autonomous decision making.

It also includes benchmarks, metrics, and models used in different software engineering applications.

[elvis @omarsar0]

Synthesizing Text-to-SQL Data from Weak and Strong LLMs

Proposes integrated synthetic data to build a highly specialized SoTA text-to-SQL model called SENSE.

The synthetic data from strong models enhances data diversity while valuable erroneous data from weaker models combined with an executor to learn from execution feedback. Preference learning is used to instruction-tune LLMs to learn from both correct and incorrect samples.

SENSE achieves state-of-the-art results on the SPIDER and BIRD benchmarks, which bridges the performance gap between open-source models and methods that use closed-source models.

Overall, this is an interesting framework where the outputs of weaker and smaller aligned models can be cleverly integrated to achieve more generalized systems.

[Josep Ferrer @iamjosepferrer]

How Python work in our computer clearly explained👇🏻

[Sahn Lam @sahnlam]

Caching 101: The Must-Know Caching Strategies

Fetching data is 𝘀𝗹𝗼𝘄. Caching speeds things up by storing frequently accessed data for quick reads. But how do you populate and update the cache? That's where strategies come in.

🔍 Read Strategies:

𝗖𝗮𝗰𝗵𝗲 𝗔𝘀𝗶𝗱𝗲 (Lazy Loading)
- How it works: Tries cache first, then fetches from DB on cache miss
- Usage: When cache misses are rare or the latency of a cache miss + DB read is acceptable

𝗥𝗲𝗮𝗱 𝗧𝗵𝗿𝗼𝘂𝗴𝗵
- How it works: Cache handles DB reads, transparently fetching missing data on cache miss
- Usage: Abstracts DB logic from app code. Keeps cache consistently populated by handling misses automatically

📝 Write Strategies:

𝗪𝗿𝗶𝘁𝗲 𝗔𝗿𝗼𝘂𝗻𝗱
- How it works: Writes bypass the cache and go directly to the DB
- Usage: When written data won't immediately be read back from cache

𝗪𝗿𝗶𝘁𝗲 𝗕𝗮𝗰𝗸 (Delayed Write)
- How it works: Writes to cache first, async write to DB later
- Usage: In write-heavy environments where slight data loss is tolerable

𝗪𝗿𝗶𝘁𝗲 𝗧𝗵𝗿𝗼𝘂𝗴𝗵
- How it works: Immediate write to both cache and DB
- Usage: When data consistency is critical

🚀 Real-Life Usage:

𝗖𝗮𝗰𝗵𝗲 𝗔𝘀𝗶𝗱𝗲 + 𝗪𝗿𝗶𝘁𝗲 𝗧𝗵𝗿𝗼𝘂𝗴𝗵
This ensures consistent cache/DB sync while allowing fine-grained cache population control during reads. Immediate database writes might strain the DB.

𝗥𝗲𝗮𝗱 𝗧𝗵𝗿𝗼𝘂𝗴𝗵 + 𝗪𝗿𝗶𝘁𝗲 𝗕𝗮𝗰𝗸
This abstracts the DB and handles bursting write traffic well by delaying sync. However, it risks larger data loss if the cache goes down before syncing the buffered writes to the database.

–
Subscribe to our weekly newsletter to get a Free System Design PDF (158 pages): https://t.co/kNfv0DVDdf

[Rohan Paul @rohanpaul_ai]

Uncovers a critical vulnerability in the function calling process of LLMs, introducing a novel "jailbreak function" attack method that exploits alignment discrepancies, user coercion, and the absence of rigorous safety filters.

📌Jailbreak function attack exploits vulnerabilities in LLM function calling, achieving over 90% success rate across six state-of-the-art models including GPT-4, Claude-3.5-Sonnet, and Gemini-1.5-pro.

📌Attack method involves crafting a "jailbreak function" with carefully designed template, custom parameters, system parameters, and trigger prompt to induce harmful content generation within function arguments.

📌Key vulnerabilities: 1) Alignment discrepancies between function arguments and chat mode responses. 2) User ability to force execution of potentially harmful functions. 3) Lack of rigorous safety filters in function calling process.

📌Empirical study revealed significant drop in attack success when testing in chat mode or allowing models to choose function execution, confirming hypotheses about alignment issues and forced execution vulnerabilities.

📌Defensive strategies proposed: 1) Restricting user permissions on function calls. 2) Aligning function calls through additional security training. 3) Implementing safety filters for function arguments. 4) Incorporating defensive prompts in function descriptions or user prompts.

📌Inserting defensive prompts in function descriptions proved most consistently effective across tested models, reducing attack success rate to 0% for Claude-3.5-Sonnet and Gemini-1.5-pro, and to 10% for GPT-4.

📌Findings highlight urgent need for enhanced security measures in LLM function calling capabilities, emphasizing importance of considering all interaction modes in AI safety development.

[elvis @omarsar0]

LLM NEWS: New SoTA text-to-image and LLM debut.

Major developments include:

- SoTA text-to-image models debut (FLUX.1)
- Gemini 1.5 Pro leads Chatbot Arena
- Self-Reasoning layer enhances RAG accuracy
- Meta-judges improve LLMs
- Gemma 2 2B outperforms larger models
- Constraining method reduces CoT tokens
- Prompting Engineering -> Prompt Design
- LangChain drops new Agents IDE
...

Here's the detailed summary: https://t.co/oWC6VqToke