max mover

The blessed

[max mover @maxmover1]

I would say this is awesome

[Withings @Withings]

Introducing BeamO, the world’s first multiscope for revolutionary at-home checkups, enabling you to observe not only your temperature, but also your heart and lungs. https://t.co/vwXn99byuU

#CESUnveiled #CES2024

[Withings @Withings]

Introducing BeamO, the world’s first multiscope for revolutionary at-home checkups, enabling you to observe not only your temperature, but also your heart and lungs. https://t.co/vwXn99byuU

#CESUnveiled #CES2024

[Yu Su @ysu_nlp]

Generalist web agents may get here sooner than we thought---introducing SeeAct, a multimodal web agent built on GPT-4V(ision).

What's this all about?
> Back in June 2023, when we released Mind2Web (https://t.co/eF4ZzVrP7S) and envisioned generalist web agent, a language agent that can work out of the box on any given website, my projection was that it would take at least several years to see such an agent that is anywhere near usable in practice.
> Why wouldn't I? The most powerful LLM at the time (perhaps still is today), GPT-4, was pretty terrible at this---its end-to-end success rate was around 2% (!!) HTML of modern websites are too long and noisy for LLMs. It's like finding a needle in a haystack. And a long-horizon task can take 10+ actions, so an LLM needs to successfully find 10+ "needles" in a row (!!!) to complete a task.

What's changed in just a few months?
> Large multimodal models. The end of 2023 marked a major milestone for LMMs, with GPT-4V, Gemini, and many good OSS LMMs released.
> Multimodal web agents. Websites are designed to be visually rendered and consumed. Visuals are much more clean and intuitive than HTML, 10x more efficient in terms of token counts. Plus, a pretty unique property of websites is that we have the correspondences between visual elements and HTML code! Such perfectly aligned multimodality is a gold mine for modeling.
> Online evaluation. The final piece of the secret recipe is online evaluation on live websites. Mind2Web initially only supported offline eval on cached websites. We developed a new tool to support running and evaluating web agents on live websites. Both LLMs and LMMs get a big boost, because now they don't have to follow exactly the reference plan in offline eval but are rather free to explore alternative plans to achieve the same goal.

SeeAct
> SeeAct is a generalist web agent built on LMMs like GPT-4V. Specifically, given a task on any website (e.g., “Compare iPhone 15 Pro Max with iPhone 13 Pro Max” on the Apple homepage), the agent first performs action generation to produce a textual description of the action at each step towards completing the task (e.g., “Navigate to the iPhone category”), and then performs action grounding to identify the corresponding HTML element (e.g., “[button] iPhone”) and operation (e.g., CLICK, TYPE, or SELECT) on the webpage.

Main results
> SeeAct can successfully complete up to 50% of tasks on live websites, substantially outperforming GPT-4
(20%) and FLAN-T5 (18%), if oracle action grounding is provided.
> However, grounding is still a major challenge. It turns out that GPT-4V can often accurately describe in text what action should be taken, but has trouble grounding the action to the exact HTML element and operation on the webpage. Existing grounding strategies like set-of-mark prompting turns out not very effective for web agents. Our best grounding strategy leverages the correspondences between visuals and HTML.
> SeeAct w/ GPT-4V shows many interesting capabilities such as speculative planning, world knowledge (e.g., airport codes), and some sort of "world model" (for websites at least), that it can correctly predict the state transitions on a website (e.g., what would happen if I click this button)

Fun fact
Initially we were hoping to show that even GPT-4V would still be insufficient for generalist web agents and we may still need fine-tuning, but we kept getting blown away by its incredible capability as a web agent. Such pleasant surprises are why I enjoy doing AI research so much these days. I also look forward to test Gemini Ultra and see whether its strong performance on MMMU would transfer.

Conclusion
Practically useful web agents could be coming soon. Buckle up and start thinking about what new applications will be enabled.

📌Website: https://t.co/r9v8eRSseY
📌Paper: https://t.co/SLESONX8rt
📌Code: https://t.co/79swHiS2J2

Work led by my amazing students @boyuan__zheng @BoyuGouNLP from @osunlp, joint with Jihyung Kil and @hhsun1. Hire them for internships!

[max mover @maxmover1]

To learn https://t.co/6Xc6DF0qmK

[max mover @maxmover1]

@lalru7 阳泉哪里

[Elon Musk @elonmusk]

All design & engineering of the original @Tesla Roadster is now fully open source.

Whatever we have, you now have.

https://t.co/5d10soAYr8

[max mover @maxmover1]

@0xkarasy @OpenAI @adamdangelo Agree

[max mover @maxmover1]

@satyanadella well done

[max mover @maxmover1]

@sama welcome back

[max mover @maxmover1]

@OpenAI Keep shipping. How can I get the latest app?

[max mover @maxmover1]

@gdb Keep shipping

[max mover @maxmover1]

@elonmusk Awesome

[Zain @ZainHasan6]

Why do large language models pay more attention to and reason better over the beginning and end of what you tell them in prompts?🤔

@nelsonfliu and Percy Liang's group at Stanford recently published a paper (https://t.co/kXaZKF3nC4) that discovered this "lost in the middle" effect.

@GregKamradt also ran great experiments and posted about how this very same pattern of underperformance exists in the new GPT-4 128K models from OpenAI.

The point of the paper was to establish "how well LLMs use longer context" and ran experiments conducting QnA and key-value retrieval tasks on models from Mosaic, Anthropic and OpenAI and varied input context size and the position of the relevant information in the context.

The main discovery was that attention followed a U-shaped pattern where more importance was given to the beginning and end of the context window as opposed to the middle portion.

This is such a great paper with a wealth of knowledge gems💎- here are some details and reasons why this happens:

1. Due to Model Architecture: LLMs are transformers that scale poorly to long sequences (O(d^2)). As a result, language models are typically trained with relatively small context windows and thus perform better on these.

2. Tasks during supervised instruction-tuning are commonly placed at the beginning of the input context, which might lead these LLMs to place more weight on the start of the input context.

3. Encoder-decoder models perform better than decoder-only models, by making better use of their context windows because their bidirectional encoder allows processing each document in the context of future documents.

4. You can improve the performance of decoder-only models(can only attend to prior tokens at each timestep) by placing the query before and after the data, enabling query-aware contextualization of documents.

6. Based on the key-value retrieval experiments, let alone attending less to the middle, many models struggle to simply retrieve matching tokens that occur in the middle of their input context.

7. Even base language models (i.e., without instruction fine-tuning) show a U-shaped performance curve.

8. For open-domain QnA tasks, where none or many of the top k documents may contain the answer, models fail to effectively use more than 20 retrieved documents.

[max mover @maxmover1]

@ilyasut https://t.co/8ovYrdt9bj

[Ilya Sutskever @ilyasut]

Ego is the enemy of growth

[max mover @maxmover1]

@gdb @merettm @sidorszymon @aleks_madry @sama Looking forward to it

[Andy Zou @andyzou_jiaming]

LLMs can hallucinate and lie. They can be jailbroken by weird suffixes. They memorize training data and exhibit biases.

🧠 We shed light on all of these phenomena with a new approach to AI transparency. 🧵

Website: https://t.co/J7ikNYmZP9
Paper: https://t.co/b84PXQOOsc

[max mover @maxmover1]

@ilyasut You just destroy it.

[max mover @maxmover1]

@ilyasut OpenAI is continue to be NGO, controlled by a board with meaningless act.

[max mover @maxmover1]

@satyanadella @sama Awesome solution

[max mover @maxmover1]

@elonmusk The grok father