Hailin Chen

Research Scientist at Salesforce AI Research

[wh @nrehiew_]

How to train a 670B parameter model.

Let's talk about the DeepSeek v3 report + some comparisons with what Meta did with Llama 405B

[Hailin Chen @HailinChen3]

Quick try on Claude computer use: definitely next-gen agent, though not super mature now

Pic 1: able to move pawn
Pic 2: tried to move knight, cursor moved to wrong place
Pic 3: able to put prime numbers to excel column
Pic 4: tried to draw line chart, but clicks insert image

[Hailin Chen @HailinChen3]

This is gonna be HUGE

[Anthropic @AnthropicAI]

The new Claude 3.5 Sonnet is the first frontier AI model to offer computer use in public beta.

While groundbreaking, computer use is still experimental—at times error-prone. We're releasing it early for feedback from developers.

[Marc Benioff @Benioff]

Salesforce Agentforce — driving the highest accuracy & lowest hallucination AI —- integrated🔥SFR-RAG 🔥 LLM specialized in RAG use cases, optimized to fully leverage contextual content for accurate and faithful response generation. This is what AI was meant to be. Congrats @SFResearch!

📘 SFR-RAG Paper: https://t.co/6qDHl1cg5S
🧪 ContextualBench benchmark: https://t.co/gZUI3BRIWO
🧠 Blog Post: https://t.co/cxUHRNlYnU

[Hailin Chen @HailinChen3]

Happy to be part of the effort

[Salesforce AI Research @SFResearch]

When it comes to CRM use cases, LLMs are not “one size fits all.” Announcing the world’s first LLM benchmark for CRM, allowing you to select the right model for the right task, across four key dimensions: Accuracy, Cost Speed, and Trust & Safety. Read the press release and explore the LLMs we’re measuring today—with more on the way. 🚀 @huggingface
https://t.co/FiQmJmJvVv #SalesforceAI #AI #LLM

[Caiming Xiong @CaimingXiong]

Are your LLMs highly accurate, or simply contaminated?
As the race to build the best LLM intensifies, clean evaluation is becoming more important than ever, yet contaminated LLMs and benchmarks obfuscate the real performance of models.
Checkout our new work (comprehensive survey + library) at NTU-NLP lab & Salesforce Research on the critical issue of contamination detection in LLMs, cc @ntunlp @MatRavox @D_Boss001 @HailinChen3 @XingxuanLi @RuochenZhao3 @FangkaiJiao @qcwntu @CaimingXiong @JotyShafiq
Paper:
https://t.co/16jXQ1hNXk
Library:
https://t.co/vNgXrto1XJ

[Hailin Chen @HailinChen3]

Modularized Chain-of-revisions works really well for code generation. Check our new ICLR'24 paper!

[Henry Hung Le @LHung1610]

Language models are well known for their strong performance in NLP. What about competitive programming problems e.g. Codeforces? Check out our work "CodeChain: Towards Modular Code Generation Through Chain of Self-revisions with Representative Sub-modules" accepted to #ICLR2024!

[Sophia Yang, Ph.D. @sophiamyang]

Open-Source LLMs vs. ChatGPT:

1. General Capabilities: Llama-2-chat-70B variant exhibits enhanced capabilities in general conversational tasks, surpassing the performance of GPT-3.5-turbo; UltraLlama matches GPT-3.5-turbo’s performance in its proposed benchmark.

2. Agent Capabilities (using tools, self-debugging, following natural language feedback, exploring environment): Lemur-70B-chat surpasses the performance of GPT-3.5-turbo when exploring the environment or following natural language feedback on coding tasks. AgentLlama-70B achieves comparable performance to GPT-3.5-turbo on unseen agent tasks. Gorilla outperforms GPT-4 on writing API calls.

3. Logical Reasoning Capabilities: fine-tuned models (e.g., WizardCoder, WizardMath) and pre-training on higher quality data models (e.g., Lemur-70B-chat, Phi-1, Phi-1.5) show stronger performance than GPT-3.5-turbo.

4. Modeling Long-Context Capabilities: Llama-2-long-chat-70B outperforms GPT-3.5-turbo-16k on ZeroSCROLLS.

5. Application-specific Capabilities:
- query-focused summarization (fine-tuning on training data is better)
- open-ended QA (InstructRetro shows improvement over GPT3)
- medical (MentalLlama-chat-13 and Radiology-Llama-2 outperform ChatGPT)
- generate structured responses (Struc-Bench outperforms ChatGPT)
- generate critiques (Shepherd is almost on-par with ChatGPT)

6. Trust-worthy AI:
- hallucination: during finetuning - improving data quality during fine-tuning; during inference - specific decoding strategies, external knowledge augmentation (Chain-of-Knowledge, LLM-AUGMENTER, Knowledge Solver, CRITIC, Prametric Knowlege Guiding), and multi-agent dialogue.
- safety: GPT-3.5-turbo and GPT-4 models remain at the top for safety evaluations. This is largely attributed to Reinforcement Learning with Human Feedback (RLHF). RL from AI Feedback (RLAIF) could help reduce costs for RLHF.

🔗https://t.co/AQ7UpsL2ev

Thanks to the authors for the great paper! @CaimingXiong @HailinChen3 @FangkaiJiao @qcwntu
@XingxuanLi @RuochenZhao3 @MatRavox @JotyShafiq

[Hailin Chen @HailinChen3]

@sophiamyang Great summary! Thanks for sharing it ❤️

[Shafiq Joty @JotyShafiq]

It has been exactly one year since the release of ChatGPT. How far are open-source LLMs? We provide an exhaustive review of open-source LLMs that claim to catch up with or surpass ChatGPT in various capabilities.

paper🔗: https://t.co/oPixlqjICe
🧵(1/5)

[Caiming Xiong @CaimingXiong]

It has been exactly one year since the release of #ChatGPT. How far are open-source LLMs? We provide an exhaustive review of open-source LLMs that claim to catch up with or surpass ChatGPT in various capabilities.

paper🔗: https://t.co/ARg0Jt0LjX

[Hailin Chen @HailinChen3]

@agihippo So you’re like research Merlion:)

[Hailin Chen @HailinChen3]

@kthai1618 Hhhhh, that’s a hilarious pun

[Roy Lee @SRoyLee]

Singapore has the highest Acceptance rate for EMNLP'22!!! Small country with inspiring NLP research #EMNLP22 #EMNLP2022livetweet

[Hailin Chen @HailinChen3]

Feel free to join our oral presentation session on 10th Dec at Collaboratorium (or catch up with me anytime). This work is collaborated with Amrita Saha, @JotyShafiq and @stevenhoi

Github: https://t.co/qM6k13p766
[6/6]

[Hailin Chen @HailinChen3]

Human can solve a new problem by extracting modular knowledge from previous learning and re-combining them, while deep models can't.
Inspired by this, we present "Learning Label Modular Prompts for Text Classification in-the-wild" [https://t.co/jiXRbhFXGh] at #emnlp2022 . [1/6]

[Hailin Chen @HailinChen3]

In qualitative case study, ModularPT shows in-context learning ability to generalize to unseen label combinations. E.g. ModularPT never sees person-xx and org-xx label prompts together during training, but it's able to integrate them (even make mistakes by doing so) [5/6]