TRIPOD-LLM is out! Check out our consensus guidelines for reporting #LLM research in biomedicine. TRIPOD-LLM is intended to be a living guideline to keep up with the rapid advances in #LLMs/Gen AI. Kudos to lead author @JackGallifant
Don't do RAG
Proposes cache-augmented generation (CAG) to eliminate retrieval latency and minimize retrieval errors.
What is CAG?
CAG aims to leverage the capabilities of long-context LLMs by preloading the LLM with all relevant docs in advance and precomputing the key-value (KV) cache.
The preloaded context helps the model to provide contextually accurate answers without the need for additional retrieval during runtime.
When to apply CAG?
It's a useful alternative to RAG for cases where the documents/knowledge for retrieval are of limited, manageable size.
My thoughts: As LLMs advance in capabilities, I suspect that what we know as RAG today could change significantly either architecturally or how it's optimized. CAG is one in a growing list of developments and new ideas that have emerged recently to address limitations like poor retrieval relevancy and latency. There could also be hybrid methods that combine preloading with selective retrieval.
Don't sleep on long-context LLMs. They are here to stay.
The data science revolution is getting closer. TabPFN v2 is published in Nature: https://t.co/Ybb15pnZ5P On tabular classification with up to 10k data points & 500 features, in 2.8s TabPFN on average outperforms all other methods, even when tuning them for up to 4 hours🧵1/19
Il nostro presidente, Gianluigi Greco, è stato intervistato da Rai News.
Nell’intervento ha spiegato alcuni dei modi in cui l’AI può aiutare la nostra società.
Scopriamo di più insieme: https://t.co/0qnsNEUHeG
#AIxIA#IntelligenzaArtificiale#AI
And another 3 year postdoc position in our neuro-symbolic AI group (https://t.co/NHFTzYCmrB), this one on reasoning about the harmfulness of internet memes, a really hard multimodal problem that will need both learning & reasoning.
https://t.co/dtNgKFunA6
Just 6 years ago. This NLP reviewer was 100% certain that prompt engineering to unify all NLP problems in a single neural network and just ask it any question was completely misguided and rejected the paper. It would then be cited by the GPT2 and 3 papers. Thanks @arxiv!
How to write a grant?
1. Write it for the reviewer, not you, the applicant.
2. Communicate in stories.
3. Make your story cohesive—leave no puzzling gaps.
4. Make your story resonate to keep the reviewer reading.
5. Accept chance and noise in peer-review.
https://t.co/wHZ065DNlm
🚨 Are PostDocs Alright? 🚨
Join us on 27.11.2024, for a joint event by @LeibnizPostDocs & German Postdoc Network as we discuss the latest findings from the Leibniz PostDoc survey!
🕛12:00-13:00 CET
Register now: https://t.co/eKrdJ0Q9Yf
#postdocs#IchBinHanna
✨EXCITING NEWS! Registration for the 3rd Learning on Graphs conference is now open 📷 It is virtual, free to attend, livestreamed, and recorded 📷https://t.co/HGQdXOBRal
Meet Tülu 3 -- a set of state-of-the-art instruct models with fully open data, eval code, and training algorithms.
We invented new methods for fine-tuning language models with RL and built upon best practices in the community to scale synthetic instruction and preference data.
Demo, GitHub, technical report, and models below 👇
📢 25 november komen we in actie tegen de bezuinigingen op onderwijs en onderzoek. Laat ook je stem horen voor toegankelijk onderwijs, tegen ontslagen en tegen de langstudeerboete. Teken de petitie een kom in actie! https://t.co/ha1ZsirXO7
Incredible LLM Creation Visualization in this Site.
Click on each section, like Embedding, LayerNorm, Self Attention, and it will show you the mechanics of that section .
(link in comment)
Here are 9 AI datasets still dominated by humans 👇
It shows the insane amount of value left to capture
Will these tasks be solved through ad-hoc LLM engineering by companies or directly by LLM providers?
Thanks to @ldjconfirmed for the data!
G-Retriever (https://t.co/BSDZxKwdLP) that leverages the strengths of GraphRAG, LLMs, and GNNs, has now been integrated into the PyG library.
Thanks to @jure and the @PyG_Team team for making this possible!
Great read - "Understanding LLMs: A Comprehensive Overview from Training to Inference"
The journey from self-attention mechanism to the final LLMs.
This paper reviews the evolution of large language model training techniques and inference deployment technologies.
--------
→ The evolution of LLMs and current training paradigm
Training approaches have evolved from supervised learning to pre-training and fine-tuning, now focusing on cost-efficient deployment. Current focus is on achieving high performance through minimal computational resources.
→ Core architectural components enabling LLMs' success
The Transformer architecture with its self-attention mechanism forms the backbone. Key elements include encoder-decoder or decoder-only designs, enabling parallel processing and handling long-range dependencies.
→ Key challenges in training and deployment
Main challenges include massive computational requirements, extensive data preparation needs, and hardware limitations. Solutions involve parallel training strategies and memory optimization techniques.
→ The role of data and preprocessing in LLM development
High-quality data curation and preprocessing are crucial. Steps include filtering low-quality content, deduplication, privacy protection, and bias mitigation.
🔍 Critical Analysis & Key Points:
→ Data preparation strategies drive model quality
Processing raw data through sophisticated filtering, deduplication and cleaning pipelines directly impacts model performance.
→ Parallel training techniques enable massive scale
Using data parallelism, model parallelism and pipeline parallelism allows training billion-parameter models efficiently.
→ Memory optimization is crucial for inference
Techniques like quantization, pruning and knowledge distillation help deploy large models with limited resources.