Edgar Isusquiza

Ciencia de Datos | Big Data | Machine Learning | Economista | Políticas Públicas

[𝗔𝗹𝗲 𝗚𝘂𝘁𝗶𝗲́𝗿𝗿𝗲𝘇 @AleGutierrez_mx]

¡Buenos días, León! 👋🏻🧡

¡Estamos haciendo historia y dejando huella en nuestra corporación! 👮🏻‍♀️👮🏻‍♂️🎓

Celebramos la graduación de la primera generación de la Licenciatura en Derecho de la Universidad Metropolitana de Formación Policial y Seguridad Pública. 📜👏🏻

Seguimos profesionalizando a nuestras mujeres y hombres de la Secretaría de Seguridad, Prevención y Protección Ciudadana, porque una corporación más preparada significa un León más fuerte y seguro.

📡 ¡Sigue la transmisión y acompáñanos en este gran momento!

https://t.co/v0YxDC3oW3

[OpenAI @OpenAI]

GPT-5.2 derived a new result in theoretical physics.

We’re releasing the result in a preprint with researchers from @the_IAS, @VanderbiltU, @Cambridge_Uni, and @Harvard. It shows that a gluon interaction many physicists expected would not occur can arise under specific conditions.

https://t.co/EAZhKWacsG

[Wenhu Chen @WenhuChen]

🚀 New Paper: Pixel Reasoner 🧠🖼️
How can Vision-Language Models (VLMs) perform chain-of-thought reasoning within the image itself?

We introduce Pixel Reasoner, the first open-source framework that enables VLMs to “think in pixel space” through curiosity-driven reinforcement learning.

Current VLMs reason only in text — even when grounded in rich images or videos, their logical steps are verbalized in natural language. This restricts their ability to interrogate visual evidence and demonstrate how conclusions are drawn.
🔍 So we ask:
What if we could make VLMs "show their work" by reasoning directly in the pixel space?
Inspired by GPT-o3’s "think-in-image" ability, we propose a framework where VLMs use interactive visual operations — zoom, select-frame, highlight — to reason through complex visual inputs.

To do this, we design a two-stage training process: Instruction tuning with synthesized visual reasoning traces. Reinforcement learning with curiosity-driven reward to balance exploration between pixel and text reasoning

✨ With this, Pixel Reasoner achieves near-SoTA performance on many information-rich multimodal benchmarks:
📊 84% on InfographicsVQA
🧠 84% on V* benchmark
🧩 74% on TallyQA-Complex
It also achieves strong accuracy of 68% on MVBench (a video benchmark).

Website: https://t.co/3YUxaIJmIv
Paper: https://t.co/CHYukmu5fB
Code: https://t.co/0mQOfXbKpM
Demo: https://t.co/AWDNoffEz8 (coming soon)

[elvis @omarsar0]

// A Survey of Frontiers in LLM Reasoning //

Nice survey on reasoning LLM with focus on inference scaling, enhancing reasoning, and applications in agentic systems.

[Josep Ferrer @iamjosepferrer]

The Transformer's encoder clearly explained 👇🏻

[Shruthi Nagamani @shruthinr]

Holaaa, me mudo de país por la maestría y ando vendiendo mis cosas. Si me pueden apoyar con un rt por fa. Todo se recoge en la roma.

Tengo mil cosas más, si buscan algo específico mándeme dm y chance tengo. ✨

[LangChain @LangChain]

😍Optimization of LLM Systems with DSPy and LangChain

Recording from our webinar with @hwchase17 and @lateinteraction is up! Covers:

👨‍🏫Introduction to DSPy
🦜Similarities to LangChain

and most excitingly...
❓How LangChain <> DSPy can collaborate!

https://t.co/Swio5Tpc2U

[Virat Singh @virattt]

My financial RAG dataset is live 🗃️

You all asked me to share the 100 question dataset.

So, here it is.

Dataset details:
• 100 questions on Airbnb 2023 10-K
• synthetically generated via opus

The dataset is tiny right now, but I will continue expanding it.

Eventually, I will upload it to HuggingFace.

Let me know what the best open source libraries are for economically generating datasets.

If none exist, I will build my own and share here 🙂

[LlamaIndex 🦙 @llama_index]

Injecting Knowledge Graphs in RAG

A common question we’ve heard from users is how to best include knowledge graphs (KGs) in a RAG pipeline.

This is a fantastic post by @chiajy2000 that covers graph-related techniques across different stages - see diagram below for stages:

[Stage 1️⃣] Query Augmentation: Use KGs to augment a query with term understanding! (e.g. a company has a bank of internal terms)
[Stage 2️⃣] Document Hierarchies: Use KGs to model relationships between chunks. You can see this with @llama_index hierarchical retrieval strategies: https://t.co/u9eWHSSSvK

[Stage 2️⃣] Contextual Dictionaries: define relationships between the metadata of each chunk
[Stage 3️⃣] Recursive Retrieval: see @llama_index recursive retriever for example of recursively walking through relationships between chunks to retrieve hierarchically
[Stage 4️⃣ ] Answer Augmentation: After initial vector retrieval, use relationships to include more concepts
[Stage 5️⃣ and onwards]: Enforcing answer validity, chunk access controls, and KG personalization.

Check it out! https://t.co/E7ATNpfS2s

[Santiago @svpino]

You can build a full-stack application using Python alone.

You don't need JavaScript, CSS, or HTML.

If you are a data scientist or someone dealing with data, here is an open-source Python library that will let you build end-to-end production applications without worrying about learning web development:

https://t.co/0NQxKtJC6n

Star the repo!

Taipy works with Python. It has a library of pre-built components to interact with data pipelines, including visualization and management tools. It supports tools for versioning and pipeline orchestration.

It's open-source and comes with a Visual Studio Code extension that will get you started without writing any code.

Thanks to the team behind Taipy for collaborating with me on this post.

Adding this to your tool belt is one of the easiest ways to improve your Data Science career in 2024.

[LlamaIndex 🦙 @llama_index]

A Five Part Series to Building a Full-Stack RAG Chatbot

This is one of the most comprehensive tutorials we’ve seen to help you build RAG end-to-end (algorithms, frontend, backend) - and it’s still ongoing!

Marco Bertelli has published a series of blog posts taking you through this process:
1️⃣ Model Selection
2️⃣ Setting up Flask Backend
3️⃣ Constructing ChatEngine
4️⃣ Optimizing RAG pipeline (agentic reasoning, cost reduction, reranking)

The great thing about it being ongoing is there’s still new content to uncover. Linking the latest article on optimizing RAG below (links to previous sections):
https://t.co/ErQdmpVhr4

[LlamaIndex 🦙 @llama_index]

Chain-of-Table ⛓️📊: use LLMs to understand tabular data step-by-step

Even the best LLMs have a hard time understanding tabular data:
🚫 Dumping the table in the prompt doesn’t work given attention limits
🚫 Text-to-SQL is flaky / inflexible.

That’s when we discovered the chain-of-table framework, proposed by @zlwang_cs 💡- form a chain of tables through step-by-step reasoning + planning with a limited set of tabular operations.

Do any of the following operations in sequence: adding a column, selecting a row, groupby, sorting, etc. It’s similar to how a data scientist would transform the table into a concise/readable representation.

We were excited to implement this as a LlamaPack in @llama_index! 🦙📦

LlamaPack: https://t.co/kqPgCAv44R

Notebook: https://t.co/13zlcf4n73

ArXiv: https://t.co/sHhiXnL4nx

[elvis @omarsar0]

LLMs for Generative Information Extraction

One of the most common uses of LLMs is to analyze and extract information. It's also probably one of the easiest and most effective ways to use LLMs if done right.

Here is a new survey paper on methods, trends, domains, insights, and tasks where LLMs are effective for information extraction.

https://t.co/irqfQiW7Ct

[Andrew Ng @AndrewYNg]

New short course on advanced retrieval for RAG (retrieval augmented generation)!

RAG fetches relevant documents to give context to an LLM. In Advanced Retrieval for AI with Chroma, taught by @trychroma founder @atroyn, you’ll learn:
(i) Query expansion using an LLM to rewrite and improve a query, by either generating either additional relevant queries or a hypothetical answer to the query.
(ii) Reranking using a cross-encoder - a model trained to measure similarity between two inputs presented simultaneously. Reranking reorders retrieved documents based on the cross-encoder similarity measure.
(iii) Constructing and training an Embedding Adaptor, which is a model that adapts the embedding values to be more relevant to your use case.

Each of these techniques can help you build much better RAG systems. Please sign up for the course here: https://t.co/6N1H8agcYC

[Maxime Labonne @maximelabonne]

The LLM Course (https://t.co/2melpj9Kxr) is doing great.

I don't know why there's been a sudden increase in popularity, but it's exciting. I'm planning on releasing the LLM Engineer Roadmap very soon in January.

[Vaibhav (VB) Srivastav @reach_vb]

fuck yeah! whisper on metal powered by rust 🦀

100% local + fastt!

brought to you by 🤗candle.

[Goku Mohandas @GokuMohandas]

Excited to share our production guide for building RAG-based LLM applications where we bridge the gap between OSS and closed-source LLMs.

- 💻 Develop a retrieval augmented generation (RAG) based LLM application from scratch.
- 🚀 Scale the major workloads (load, chunk, embed, index, serve, etc.) across multiple workers.
- ✅ Evaluate different configurations of our application to optimize for both per-component (ex. retrieval_score) and overall performance (quality_score).
- 🔀 Implement LLM hybrid routing approach to bridge the gap b/w OSS and closed LLMs.
- 📦 Serve the application in a highly scalable and available manner.
- 💥 Share the 1st order and 2nd order impacts LLM applications have had on our products.

🔗 Links:
- Blog post (45 min. read): https://t.co/QHgOXPT7S0
- GitHub repo: https://t.co/GMNrsHAhpY
- Interactive notebook: https://t.co/UPXSkwDt6h

@pcmoritz and I had a blast developing and productionizing this with the @anyscalecompute team and we're excited to share Part II soon (more details in the blog post).

[Bindu Reddy @bindureddy]

Create Your Own Custom LLM ChatBot - An AI Brain for your organization

A step by step tutorial on how to choose the best LLM and infrastructure to create a Custom ChatGPT and supercharge your business!

A large language model (LLM) trained on your company's data can function like a AI brain for your business - boosting employee productivity, improving decision making and surfacing customs insights from your data

Here are the key factors to consider before getting started

Security - For a lot of organization, data can't leave their VPC or their cloud environments

Use-cases -You might want to build a chat bot on your unstructured data like your internal wiki and sharepoint docs or a chatbot that understands all your databases. It's easy to build multiple chatbots based on your use-cases

Data sources - You want to ideally connect all your important data sources including internal wikis, share point folders, contracts and even database

Best LLM - There are several LLM APIs in the market. We have found different LLMs work best based on your specific problem. Try out all the LLM options and choose the best one that works for your dataset and use-case.

Let's pick a classic use-case - let's say we a bunch of contracts and we want to build a contracts database that answer all the employee questions about any specific contract. There are several steps involved in building this custom chatbot.

Chunking - The first step is to chunk the documents into small pieces. Chunk size and overlap are important choices based on the context length of your LLMs, type of questions being asked and how the answers are distributed in the doc

Embedding - This is the process by which you convert text into numerical matrices called vectors, so we can perform operations on them

Document Retrievers (Vector DBs) - We then use a vector database to store these vectors and perform a similarity search and retrieve the relevant chunks every time someone asks the chatbot a question.

Context Generation Strategy - So the fundamental reason behind all these steps is that all the commercially available LLM APIs can only digest a limited number of words/tokens in a single call. This is called the context length of the LLM.

So we have to know to feed the LLM relevant chunks of your data, to generate responses. Based on whether you expect long threaded chats or smaller ones, you can experiment with different techniques here

Choice of LLMs To optimize your solution, it is recommended to experiment with different LLMs and determine which one delivers the best results.

You can do this on your own or you can use a platform like https://t.co/dJ2B4Jig2U that offers a range of LLM options, including GPT3.5, GPT4, Palm, Azure OpenAI, Claude, Llama2, as well as Abacus's proprietary LLMs.

Additionally, you can fine-tune an LLM on your data and use it in the Retrieval Augmented Generation technique, enjoying the best of both worlds

Evaluation of Response: This is the final and most important piece. You want to prepare a set of questions and expected answers, get each of the LLMs to generate answers and evaluate the predicted answers against the expected answers. There are several techniques to do this and the Abacus AI platform automates the evaluation for you

Deployment and Pipelines Once you have choosen which LLM works best, you can now deploy that LLM and set up pipelines. A pipeline automatically picks up new data (contracts in this case), chunks them, generates embeddings and fine-tunes new LLMs as needed. This way the system is now on auto-pilot

Monitoring: This is the final piece. Once your AI brain is set up, you will want to evaluate it on a regular basis and and see if the performance is stable. You can now set up evaluation pipelines to ensure performance doesn't deteriorate

That's it! Your Custom ChatGPT is ready to go and any of your employees can use it any time to have conversations much like they do on ChatGPT. Using a end-to-end platform like Abacus AI also gets you a fancy chat interface very much like the one on Bard or the publicly available ChatGPT where you can have threaded conversations, conversation history and provide feedback.

You can also integrate this chat bot into slack or teams and ask questions that way.

Resources - More detailed blog post - https://t.co/QAIAjBUdSp

[Aran Komatsuzaki @arankomatsuzaki]

Shepherd: A Critic for Language Model Generation

Presents a LM specifically tuned to critique model responses and suggest refinements, which closely ties with ChatGPT.

repo: https://t.co/e7kLd6U9pN
abs: https://t.co/GL9M17ZUrS

[elvis @omarsar0]

With the velocity of developments in LLMs, it's easy to think that contributing to the field is impossible.

That couldn't be further from the truth. There is a ton of more work to do with LLMs - for both practitioners and researchers.

This neat paper summarizes a comprehensive list of challenges when working with LLMs that range from brittle evaluations to prompt brittleness to a lack of robust experimental designs.

https://t.co/mR1wHHkXR5