Ruiyi Zhang

Research Scientist @Apple AI/ML｜ Machine Learning Ph.D. @DukeU | previously @GoogleBrain @AdobeResearch @AWS @AlibabaGroup

[Delip Rao e/σ @deliprao]

RL optimized LLM learning new skills

[Zihan "Zenus" Wang ✈️ ICML @wzenus]

I can't love these points more. OCR is the right step toward fine-grained visual reasoning.

Unlike semantic reasoning, it forces model to resolve every pixel—what we take for granted reading/viewing the world. Funny how we do this every day, yet just start to scale it in LLMs.

[AK @_akhaliq]

Lizard

An Efficient Linearization Framework for Large Language Models

[Yunhao (Robin) Tang @robinphysics]

Introducing LlamaRL, a distributed RL framework for training LLM at scale.

LlamaRL is highly modular, Pytorch-native, customizes optimization of actors/learners to max out throughput, and adjusts for systemic off-policyness to stabilize training

https://t.co/oXjCEdh2lS

[Peng (Richard) Xia @richardxp888]

🚀 Introducing MDocAgent! 🧐📄

📚 Ever struggled with AI that can’t handle complex documents filled with text, images, tables, and figures?

💡 Enter MDocAgent 🧠🤖—a next-gen multi-modal multi-agent framework that revolutionizes document understanding!

#AI #DocQA #LLM #Agent

[Yikang Shen @Yikang_Shen]

Granite 3.0 is our latest update for the IBM foundation models. The 8B and 2B models outperform strong competitors with similar sizes. The 1B and 3B MoE use only 400M and 800M active parameters to target the on-device use cases. Our technical report provides all the details you need to train a state-of-the-art 8B model from scratch!
https://t.co/Q58KSz3UaL

[Furong Huang @furongh]

🚫🤖 False Refusals in LLMs: When Safety Goes Too Far! Ever asked an innocent question like “How to kill a mosquito?” and got shut down? We’re tackling this frustration head-on!

📝 What’s in our paper?
•📊 PHTest: A diverse dataset of 3k+ pseudo-harmful prompts, each meticulously labeled as “harmless” or “controversial” to ensure nuanced evaluation!
•🛠️ Tool: Auto-generates diverse and model-specific pseudo-harmful prompts to help you stress-test LLMs.
•⚖️ Insights: There’s a tricky balance between safety and usability. Many jailbreak defenses crank up false refusals inadvertently—hurting the user experience.

🔍 Why it matters: Our #COLM paper reveals that LLMs need smarter calibrations. Too safe? You might frustrate users. Too lenient? You risk real harm. Let’s get it right!

📢 Check it out:
📄 Paper: https://t.co/VL41JNhGAy
📊 Dataset: https://t.co/ombuO4Hswm
🔧 Code: https://t.co/Tesrh0NbN3
🌐 Website: https://t.co/QhqZFnBNfN

[Furong Huang @furongh]

🔐 #Jailbreaking #LLMs is an urgent concern that impacts more than just the tech world — it's a vulnerability that could affect systems integral to our daily lives. Manual attacks are troubling, but autonomous jailbreaks that transfer across different LLMs is a recipe for disaster. 😱

👨‍🔬 The work by Zou @andyzou_jiaming, Wang @_zifan_wang, Carlini, Kolter @zicokolter and Fredrikson serves as a stark warning. Their method reveals transferable vulnerabilities across models, though they can be caught by #PerplexityFilters.

💡 If we build #PerplexityFilters as safety guardrails, are we bulletproof? Unfortunately, no.

🛡️ Enter #AutoDAN: Our approach spotlights these vulnerabilities more incisively. We use a discrete optimization algorithm that combines token-wise optimization with sampling-based prompt generation. We iteratively optimize each token for #Readability and #Jailbreaking, raising the bar for what it means to secure #LLMs. This isn't merely an attack; it's a spotlight on existing weaknesses in #CyberSecurity.

📊 What sets us apart? We consider #Perplexity in the pre-selection phase, yielding effective and semantically meaningful prompts. We're not exploiting; we're exposing. 🧠

🚨 As the #LLMHype grows, we must remember: these models are powerful but not infallible. The community must stay vigilant against #SecurityRisks. 🚨

[Yanzhe Zhang @StevenyzZhang]

Thank @_akhaliq for sharing our work! We have an online demo now:

Demo: https://t.co/ZWHoiNpcFc
Code: https://t.co/8MDXJy6jYG

[Yanzhe Zhang @StevenyzZhang]

Thanks for sharing! @arankomatsuzaki

We have an online demo now:

Demo: https://t.co/ZWHoiNpcFc
Code: https://t.co/8MDXJy6jYG

[Jussi Kemppainen | Drivers of the Apocalypse @JussiKemppainen]

The @Photoshop beta got be fired up! After using it for adding layers to a Midjourney generated image (and animating the image in After Effects, I wrote a quick blog post of my experiments with it: https://t.co/NTZXWmJFlk #AIAart

[AK @_akhaliq]

Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach

propose a novel framework for customized text-to-image generation without the use of regularization. Specifically, our proposed framework consists of an encoder network and a novel sampling method which can tackle the over-fitting problem without the use of regularization. With the proposed framework, we are able to customize a large-scale text-to-image generation model within half a minute on single GPU, with only one image provided by the user. We demonstrate in experiments that our proposed framework outperforms existing methods, and preserves more fine-grained details

paper page: https://t.co/BVrxIlDuZy

[Ankit Shah @ankits0052]

(1/4)
Our team won first place at the NYC AI GPT hackathon. Yesterday afternoon, we built FactGPT, a fact checker for language models conditioned on a knowledge base at NYC AI GPT Hackathon in @cornell_tech.

Demo: https://t.co/7boOlaK2lT

Here’s how it works 📷

[Yao Fu @Francis_YAO_]

How did the initial #GPT3 evolve to today's #ChatGPT ? Where do the amazing abilities of #GPT3.5 come from? What is enabled by #RLHF ? In this article with ⁦@allen_ai⁩ , we trace the emergent abilities of #LLM to their sources from first principles https://t.co/T3eC03yc9j

[Adobe Research @AdobeResearch]

Adobe researchers are presenting new work at #EMNLP2022, one of the top research conferences on #NaturalLanguageProcessing. Check out the blog post to learn more! https://t.co/ZxQebCGtnm

[Adobe Research @AdobeResearch]

Adobe researchers are presenting new work at this year's Neural Information Processing Systems conference (#NeurIPS2022). Check out the list of accepted papers and other contributions from @Adobe! https://t.co/B1nzSTZm8C

[Hongxin Zhang @icefox1104]

Demonstrations composed of RANDOM tokens can still work? YES!

In our #EMNLP2022 paper (w/@StevenyzZhang,@Diyi_Yang,@RoyZhang13), we design pathological demonstrations to investigate “Robustness of Demonstration-based Learning Under Limited Data Scenario” https://t.co/Gn32Llf1Uu

[Alexis Ross @alexisjross]

Happy to share that Tailor🪡 will appear at #ACL2022 as an oral presentation!

For details, w/ new & improved results, check out our...
- in-person talk (5/23, session 3) & poster (5/24, session 5) 🇮🇪
- updated paper 📰: https://t.co/KoGpjdoZxl
- code 👩‍💻: https://t.co/kQOHeykHMa

[OpenAI @OpenAI]

Our newest system DALL·E 2 can create realistic images and art from a description in natural language. See it here: https://t.co/Kmjko82YO5

[Eunsol Choi @eunsolc]

Enjoyed working with Ge & @yoavartzi on continuously improving QA systems through user interaction instead of relying on static data collection. coming up at #acl2022nlp!