![cwolferesearch's tweet photo. The capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! For example, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models...
Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we:
1. Perform self-supervised pretraining over a large amount of unlabeled data.
2. Perform supervised finetuning (SFT) over a smaller summarization dataset (i.e., human-written summaries of articles).
Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid. As such, training a summarization model to exactly match a human-written summary is a flawed approach! All references summaries—even those that are low quality—are equally emphasized during the training process, and we have no way to account for the diversity of valid summaries.
“We show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.” - from [1]
Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on preference data from humans. The LLM is first pretrained, then trained using supervised finetuning over human reference summaries, producing a supervised baseline. This model is then finetuned using reinforcement learning from human feedback (RLHF), which collects human feedback data by:
1. Taking a textual input (source document) from the training dataset.
2. Using several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input.
3. Picking two summaries from the set of samples responses.
4. Asking a human annotator to identify the better of the two summaries.
Human comparison data is collected in large batches and used to finetune the model—a decoder-only LLM—via RLHF in an offline fashion. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model.
Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training.
Is learning from feedback effective? In [1], the proposed approach for learning from human feedback is shown to yield models that are highly preferred by humans. After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone.
From summarization to ChatGPT: However, the approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2]-the sister model and predecessor to ChatGPT-to align foundation language models. In fact, the training strategy used in [2] is nearly identical to the training framework proposed in [1]! Eventually, this work was extended to yield the impressive foundation LLMs that we have today! In this way, work on abstractive summarization laid the foundation for the creation of modern LLMs.](https://pbs.twimg.com/media/GPO_fs7bAAAV0lH.jpg)
![cwolferesearch's tweet photo. The capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! For example, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models...
Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we:
1. Perform self-supervised pretraining over a large amount of unlabeled data.
2. Perform supervised finetuning (SFT) over a smaller summarization dataset (i.e., human-written summaries of articles).
Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid. As such, training a summarization model to exactly match a human-written summary is a flawed approach! All references summaries—even those that are low quality—are equally emphasized during the training process, and we have no way to account for the diversity of valid summaries.
“We show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.” - from [1]
Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on preference data from humans. The LLM is first pretrained, then trained using supervised finetuning over human reference summaries, producing a supervised baseline. This model is then finetuned using reinforcement learning from human feedback (RLHF), which collects human feedback data by:
1. Taking a textual input (source document) from the training dataset.
2. Using several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input.
3. Picking two summaries from the set of samples responses.
4. Asking a human annotator to identify the better of the two summaries.
Human comparison data is collected in large batches and used to finetune the model—a decoder-only LLM—via RLHF in an offline fashion. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model.
Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training.
Is learning from feedback effective? In [1], the proposed approach for learning from human feedback is shown to yield models that are highly preferred by humans. After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone.
From summarization to ChatGPT: However, the approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2]-the sister model and predecessor to ChatGPT-to align foundation language models. In fact, the training strategy used in [2] is nearly identical to the training framework proposed in [1]! Eventually, this work was extended to yield the impressive foundation LLMs that we have today! In this way, work on abstractive summarization laid the foundation for the creation of modern LLMs.](https://pbs.twimg.com/media/GPO-6KPaYAAsrcK.jpg)
![cwolferesearch's tweet photo. The capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! For example, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models...
Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we:
1. Perform self-supervised pretraining over a large amount of unlabeled data.
2. Perform supervised finetuning (SFT) over a smaller summarization dataset (i.e., human-written summaries of articles).
Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid. As such, training a summarization model to exactly match a human-written summary is a flawed approach! All references summaries—even those that are low quality—are equally emphasized during the training process, and we have no way to account for the diversity of valid summaries.
“We show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.” - from [1]
Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on preference data from humans. The LLM is first pretrained, then trained using supervised finetuning over human reference summaries, producing a supervised baseline. This model is then finetuned using reinforcement learning from human feedback (RLHF), which collects human feedback data by:
1. Taking a textual input (source document) from the training dataset.
2. Using several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input.
3. Picking two summaries from the set of samples responses.
4. Asking a human annotator to identify the better of the two summaries.
Human comparison data is collected in large batches and used to finetune the model—a decoder-only LLM—via RLHF in an offline fashion. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model.
Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training.
Is learning from feedback effective? In [1], the proposed approach for learning from human feedback is shown to yield models that are highly preferred by humans. After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone.
From summarization to ChatGPT: However, the approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2]-the sister model and predecessor to ChatGPT-to align foundation language models. In fact, the training strategy used in [2] is nearly identical to the training framework proposed in [1]! Eventually, this work was extended to yield the impressive foundation LLMs that we have today! In this way, work on abstractive summarization laid the foundation for the creation of modern LLMs.](https://pbs.twimg.com/media/GPO-bHPaEAAPRgF.jpg)
Olivia
Online
Owen Ingraham
@justowen
Technologist
British Columbia, Canada
Joined February 2009
447 Following
185 Followers
670 Posts
https://t.co/R9glP4V1Vu
What makes BC Tech’s C-Councils so valuable? Hear it straight from our members! From peer-to-peer learning to a trusted space for connection. C-Councils are where tech leaders grow together.
Learn more about the BC Tech C-Councils here:https://t.co/Fol3bRf4y8
🚀 Innovation is alive in BC!
👏 Congrats to our Gamechanger – Industry Innovation finalists:
@Intiveo, @SimplyAskAI, #thisfish & @VRIFYTechnology!
Thanks to @Innovate_BC for supporting visionary leaders!
🔗 Read all about the #2025TIAs Finalists Here: https://t.co/xM69ylRuJb
Who to follow
Fernando Cerimedo 
@FerCerimedo_ok
Marketing Phd. Consultor Político.
🇦🇷🇧🇴🇭🇳🇺🇲🇮🇱
Urban Development Institute
@udibc
BC's foremost voice for urban development and related professions since 1972.
Non-partisan, non-profit, and paving the way for tomorrow's cities.
Kwasi Akyeampong @TheBlackList
@theblacklist
The GOOD-NEWS, Concerns, Field-Of-View of Africa/Diaspora in Conversation for A World That Works for Everyone, no exceptions #SMYNews POV=#Kwasibility 1❤️🇯🇲
As the world’s highest IQ record holder, I believe that Jesus Christ is God, the way and the truth and the life.
https://t.co/6i6h3sKi3x
ML twitter this morning 😂
Congratulations to Geoffrey Hinton! 🇨🇦
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Physics to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”
https://t.co/lcfqjQUoGm interesting
🆓 "Today, we’re launching fine-tuning for GPT-4o, one of the most requested features from developers. We are also offering 1M training tokens per day for free for every organization through September 23." https://t.co/KRbNRVsTSb
@MatthewBerman can you please add political neutrality to your test suite? https://t.co/qiwCWawoTI
tf is grokking AI and why it's important?
https://t.co/ij8zwWpITg
Actually, really liked the Apple Intelligence announcement. It must be a very exciting time at Apple as they layer AI on top of the entire OS. A few of the major themes.
Step 1 Multimodal I/O. Enable text/audio/image/video capability, both read and write. These are the native human APIs, so to speak.
Step 2 Agentic. Allow all parts of the OS and apps to inter-operate via "function calling"; kernel process LLM that can schedule and coordinate work across them given user queries.
Step 3 Frictionless. Fully integrate these features in a highly frictionless, fast, "always on", and contextual way. No going around copy pasting information, prompt engineering, or etc. Adapt the UI accordingly.
Step 4 Initiative. Don't perform a task given a prompt, anticipate the prompt, suggest, initiate.
Step 5 Delegation hierarchy. Move as much intelligence as you can on device (Apple Silicon very helpful and well-suited), but allow optional dispatch of work to cloud.
Step 6 Modularity. Allow the OS to access and support an entire and growing ecosystem of LLMs (e.g. ChatGPT announcement).
Step 7 Privacy. <3
We're quickly heading into a world where you can open up your phone and just say stuff. It talks back and it knows you. And it just works. Super exciting and as a user, quite looking forward to it.
@andrewchen Would be a smart move for Apple as they have zero new market R&D left, but @ouraring (Finnish) isn’t willing to serve it on an M&A platter (remove Android’s Health Connect support, change manufacturing agreements, scrub Ubuntu from their values, etc)
@ThePrimeagen One PR to rule them all, One PR to find them, One PR to bring them all, and in the codebase bind them
@jessepollak Good luck with the wedding!
The capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! For example, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models...
Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we:
1. Perform self-supervised pretraining over a large amount of unlabeled data.
2. Perform supervised finetuning (SFT) over a smaller summarization dataset (i.e., human-written summaries of articles).
Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid. As such, training a summarization model to exactly match a human-written summary is a flawed approach! All references summaries—even those that are low quality—are equally emphasized during the training process, and we have no way to account for the diversity of valid summaries.
“We show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.” - from [1]
Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on preference data from humans. The LLM is first pretrained, then trained using supervised finetuning over human reference summaries, producing a supervised baseline. This model is then finetuned using reinforcement learning from human feedback (RLHF), which collects human feedback data by:
1. Taking a textual input (source document) from the training dataset.
2. Using several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input.
3. Picking two summaries from the set of samples responses.
4. Asking a human annotator to identify the better of the two summaries.
Human comparison data is collected in large batches and used to finetune the model—a decoder-only LLM—via RLHF in an offline fashion. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model.
Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training.
Is learning from feedback effective? In [1], the proposed approach for learning from human feedback is shown to yield models that are highly preferred by humans. After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone.
From summarization to ChatGPT: However, the approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2]-the sister model and predecessor to ChatGPT-to align foundation language models. In fact, the training strategy used in [2] is nearly identical to the training framework proposed in [1]! Eventually, this work was extended to yield the impressive foundation LLMs that we have today! In this way, work on abstractive summarization laid the foundation for the creation of modern LLMs.
![cwolferesearch's tweet photo. The capabilities of large language models (LLMs) have progressed rapidly, but the key ideas surrounding LLMs have their roots in research from years before! For example, the training pipeline used by LLMs (pretraining -> SFT -> RLHF) was originally proposed in 2020 for training abstractive summarization models...
Supervised learning for summarization: Prior to the popularization of LLMs, most summarization models were trained in a supervised fashion where we:
1. Perform self-supervised pretraining over a large amount of unlabeled data.
2. Perform supervised finetuning (SFT) over a smaller summarization dataset (i.e., human-written summaries of articles).
Humans don't always write perfect summaries and summarization is a very open-ended task (i.e., one document can be summarized in many different ways that are equally valid. As such, training a summarization model to exactly match a human-written summary is a flawed approach! All references summaries—even those that are low quality—are equally emphasized during the training process, and we have no way to account for the diversity of valid summaries.
“We show that it is possible to significantly improve summary quality by training a model to optimize for human preferences.” - from [1]
Learning from human feedback: Inspired by these issues, authors in [1] propose a three-part framework that enables LLMs to be finetuned based on preference data from humans. The LLM is first pretrained, then trained using supervised finetuning over human reference summaries, producing a supervised baseline. This model is then finetuned using reinforcement learning from human feedback (RLHF), which collects human feedback data by:
1. Taking a textual input (source document) from the training dataset.
2. Using several policies (e.g., pretrained model, supervised baseline, current model, or the human reference summary) to sample summaries of the input.
3. Picking two summaries from the set of samples responses.
4. Asking a human annotator to identify the better of the two summaries.
Human comparison data is collected in large batches and used to finetune the model—a decoder-only LLM—via RLHF in an offline fashion. Once the data has been collected, we use this comparison data to train a reward model (using a pairwise ranking loss) that accurately predicts a human preference score given a summary produced by the LLM. From here, we use PPO (an RL algorithm) to finetune the model based on preference scores outputted by the reward model.
Avoiding drift: The authors in [1] add a KL divergence term to the objective being optimized by PPO, which penalizes the policy from becoming too different from the supervised baseline policy during RLHF. Such an approach, which is now commonly used (e.g., see Eq. 4 in the LLaMA-2 report), encourages exploration without mode collapse and prevents summaries written by the LLM from becoming too different from those that are seen during training.
Is learning from feedback effective? In [1], the proposed approach for learning from human feedback is shown to yield models that are highly preferred by humans. After training a 1.3B and 6.7B GPT-style LLM with this approach, we see that 1.3B human feedback model outperforms a 10× larger model trained with supervised learning alone, and the 6.7B human feedback model performs even better than the 1.3B model. These models are consistently preferred to those trained via SFT alone.
From summarization to ChatGPT: However, the approach proposed in [1] is not just useful for summarization! This strategy for learning from human feedback was extended by InstructGPT [2]-the sister model and predecessor to ChatGPT-to align foundation language models. In fact, the training strategy used in [2] is nearly identical to the training framework proposed in [1]! Eventually, this work was extended to yield the impressive foundation LLMs that we have today! In this way, work on abstractive summarization laid the foundation for the creation of modern LLMs.](https://pbs.twimg.com/media/GPPANmSb0AAGxzL.jpg)
@JsonBasedman Gm and happy southward exportation
Last Seen Users on Sotwe
bielefeld can
Seen from Turkey
Miha stefe
Seen from United States
Agresif_panda
Ardi gigolo DKI Jakarta
Seen from Netherlands
Trans Porn/DM FOR FREE PROMO
Seen from Germany
Deshi Dangal
Seen from India
TÜRK Porno Dünyası
Seen from Turkey
Yaşanmış Seks Hikayeleri
Seen from Turkey
NİZİP
Seen from Turkey
KİMBUTOPRAK-J
Seen from Turkey
Most Popular Users
Elon Musk
@elonmusk
241.2M followers
Barack Obama
@barackobama
119.1M followers
Cristiano Ronaldo
@cristiano
112.9M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.3M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.5M followers
KATY PERRY
@katyperry
89M followers
Taylor Swift
@taylorswift13
82.9M followers
Lady Gaga
@ladygaga
74.4M followers
Virat Kohli
@imvkohli
71.9M followers
Kim Kardashian
@kimkardashian
70.5M followers
YouTube
@youtube
68.8M followers
Neymar Jr
@neymarjr
64.9M followers
Bill Gates
@billgates
64.6M followers
The Ellen Show
@theellenshow
62.4M followers
Selena Gomez
@selenagomez
62.1M followers
CNN
@cnn
61.8M followers
X
@x
60.8M followers