Henry Smith

Software developer between jobs. Interested in machine learning, AI, autonomous vehicles, and riding around on a unicycle while juggling and playing harmonica.

[Akshay 🚀 @akshay_pachaar]

Traditional RAG vs. Agentic RAG, clearly explained (with visuals):

[Rowan Cheung @rowancheung]

So, Cognition AI just introduced Devin, the most capable autonomous coding agent I've seen.

Instead of just autocompleting tasks, it can write entire apps on its own in minutes.

It also correctly resolves 13.86% of issues, autonomously.

[AK @_akhaliq]

Design2Code

How Far Are We From Automating Front-End Engineering?

Generative AI has made rapid advancements in recent years, achieving unprecedented capabilities in multimodal understanding and code generation. This can enable a new paradigm of front-end development, in

[Henry Smith @ArtifIntersting]

Big if true, outperforming GPT-4 across the board is quite an accomplishment.
Congrats to Anthropic

[Anthropic @AnthropicAI]

Today, we're announcing Claude 3, our next generation of AI models.

The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision.

[Anthropic @AnthropicAI]

Today, we're announcing Claude 3, our next generation of AI models.

The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision.

[Henry Smith @ArtifIntersting]

This paper restricted model weights to just -1, 0, or 1, and were able to get comparable performance out of it. Interesting stuff: https://t.co/XbIKdUs9uf

[Hans Baier @[email protected] @hansfbaier]

CPU vs FPGA

[AK @_akhaliq]

Meta announces Aria Everyday Activities Dataset

present Aria Everyday Activities (AEA) Dataset, an egocentric multimodal open dataset recorded using Project Aria glasses. AEA contains 143 daily activity sequences recorded by multiple wearers in five geographically diverse indoor locations. Each of the recording contains multimodal sensor data recorded through the Project Aria glasses. In addition, AEA provides machine perception data including high frequency globally aligned 3D trajectories, scene point cloud, per-frame 3D eye gaze vector and time aligned speech transcription. In this paper, we demonstrate a few exemplar research applications enabled by this dataset, including neural scene reconstruction and prompted segmentation.

[Mckay Wrigley @mckaywrigley]

I fed an *entire* biology textbook into Gemini 1.5 Pro.

491,002 tokens.

I asked it 3 extremely specific questions, and it got each answer 100% correct.

1M token context windows are a gamechanger.

[Dreaming Tulpa 🥓👑 @dreamingtulpa]

TikTok presents Boximator!

This method can generate rich and controllable motions for image-to-video generations by drawing box constraints and motion paths onto an image and combining it with a prompt:

"A girl in red is covering her face with a skull."

10 crazy examples:

[AK @_akhaliq]

Apple presents Keyframer

Empowering Animation Design using Large Language Models

paper page: https://t.co/WXJpXFhty4

Large language models (LLMs) have the potential to impact a wide range of creative domains, but the application of LLMs to animation is underexplored and presents novel challenges such as how users might effectively describe motion in natural language. In this paper, we present Keyframer, a design tool for animating static images (SVGs) with natural language. Informed by interviews with professional animation designers and engineers, Keyframer supports exploration and refinement of animations through the combination of prompting and direct editing of generated output. The system also enables users to request design variants, supporting comparison and ideation. Through a user study with 13 participants, we contribute a characterization of user prompting strategies, including a taxonomy of semantic prompt types for describing motion and a 'decomposed' prompting style where users continually adapt their goals in response to generated output.We share how direct editing along with prompting enables iteration beyond one-shot prompting interfaces common in generative tools today. Through this work, we propose how LLMs might empower a range of audiences to engage with animation creation.

[LangChain @LangChain]

Self-Reflective RAG using LangGraph

Self-reflection can enhance RAG, enabling correction of poor quality retrieval or generations.

Several recent papers focus on this theme, but implementing the ideas can be tricky.

Here is a video and two cookbooks that show how to engineer self-reflective RAG flows in LangGraph.

Cookbooks implement ideas from two recent, interesting papers:
+ Self-RAG (@AkariAsai et al.): Grade / filter documents for relevance, grade / filter generations for hallucinations + accuracy.
+ C-RAG: Grade document for relevance, perform knowledge refinement, use web retrieval to gather more context.

Video:
https://t.co/rVy4Tpf5Eo

Blog:
https://t.co/Gai8mxe81x

Cookbooks:
https://t.co/3ThXzscI7g
https://t.co/FG0gJ9K0Xw

[AI Breakfast @AiBreakfast]

Awesome demo of a ChatGPT-powered NPC in a VR game.

LLMs will completely change how we interact in games going forward.

Credit to Tamulur on YouTube (link below)

[Henry Smith @ArtifIntersting]

@llama_index @ollama Cool stuff, looking forward to this being a standard OS feature in a few years.

[AK @_akhaliq]

Apple releases ml-mgie

demo: https://t.co/HQ9l3xInPZ

github (gradio): https://t.co/pGsPSEcCno

github: https://t.co/dgK1Hy3TRX

Guiding Instruction-based Image Editing via Multimodal Large Language Models

[AK @_akhaliq]

TravelPlanner

A Benchmark for Real-World Planning with Language Agents

paper page: https://t.co/Eujgifzguo

Planning has been part of the core pursuit for artificial intelligence since its conception, but earlier AI agents mostly focused on constrained settings because many of the cognitive substrates necessary for human-level planning have been lacking. Recently, language agents powered by large language models (LLMs) have shown interesting capabilities such as tool use and reasoning. Are these language agents capable of planning in more complex settings that are out of the reach of prior AI agents? To advance this investigation, we propose TravelPlanner, a new planning benchmark that focuses on travel planning, a common real-world planning scenario. It provides a rich sandbox environment, various tools for accessing nearly four million data records, and 1,225 meticulously curated planning intents and reference plans. Comprehensive evaluations show that the current language agents are not yet capable of handling such complex planning tasks-even GPT-4 only achieves a success rate of 0.6%. Language agents struggle to stay on task, use the right tools to collect information, or keep track of multiple constraints. However, we note that the mere possibility for language agents to tackle such a complex problem is in itself non-trivial progress. TravelPlanner provides a challenging yet meaningful testbed for future language agents.

[AK @_akhaliq]

Microsoft presents StrokeNUWA

Tokenizing Strokes for Vector Graphic Synthesis

paper page: https://t.co/SsRxahYZgD

To leverage LLMs for visual synthesis, traditional methods convert raster image information into discrete grid tokens through specialized visual modules, while disrupting the model's ability to capture the true semantic representation of visual scenes. This paper posits that an alternative representation of images, vector graphics, can effectively surmount this limitation by enabling a more natural and semantically coherent segmentation of the image information. Thus, we introduce StrokeNUWA, a pioneering work exploring a better visual representation ''stroke tokens'' on vector graphics, which is inherently visual semantics rich, naturally compatible with LLMs, and highly compressed. Equipped with stroke tokens, StrokeNUWA can significantly surpass traditional LLM-based and optimization-based methods across various metrics in the vector graphic generation task. Besides, StrokeNUWA achieves up to a 94x speedup in inference over the speed of prior methods with an exceptional SVG code compression ratio of 6.9%.

[AK @_akhaliq]

Tencent presents TIP-Editor

An Accurate 3D Editor Following Both Text-Prompts And Image-Prompts

paper page: https://t.co/MLlVujjKP1

Text-driven 3D scene editing has gained significant attention owing to its convenience and user-friendliness. However, existing methods still lack accurate control of the specified appearance and location of the editing result due to the inherent limitations of the text description. To this end, we propose a 3D scene editing framework, TIPEditor, that accepts both text and image prompts and a 3D bounding box to specify the editing region. With the image prompt, users can conveniently specify the detailed appearance/style of the target content in complement to the text description, enabling accurate control of the appearance. Specifically, TIP-Editor employs a stepwise 2D personalization strategy to better learn the representation of the existing scene and the reference image, in which a localization loss is proposed to encourage correct object placement as specified by the bounding box. Additionally, TIPEditor utilizes explicit and flexible 3D Gaussian splatting as the 3D representation to facilitate local editing while keeping the background unchanged. Extensive experiments have demonstrated that TIP-Editor conducts accurate editing following the text and image prompts in the specified bounding box region, consistently outperforming the baselines in editing quality, and the alignment to the prompts, qualitatively and quantitatively.

[AK @_akhaliq]

Microsoft presents SliceGPT

Compress Large Language Models by Deleting Rows and Columns

paper page: https://t.co/Mxf445SAnu

show that SliceGPT can remove up to 25% of the model parameters (including embeddings) for LLAMA2-70B, OPT 66B and Phi-2 models while maintaining 99%, 99% and 90% zero-shot task performance of the dense model respectively. Our sliced models run on fewer GPUs and run faster without any additional code optimization: on 24GB consumer GPUs we reduce the total compute for inference on LLAMA2-70B to 64% of that of the dense model; on 40GB A100 GPUs we reduce it to 66%. We offer a new insight, computational invariance in transformer networks, which enables SliceGPT and we hope it will inspire and enable future avenues to reduce memory and computation demands for pre-trained models.