Guher Gorgun

👩‍💻Ph.D. in Measurement, Evaluation, and Data Science @UAlbertaMEDS @UAlberta #psychometrics #assessment #AIED #EDM #datamining #learninganalytics

[Dr. Okan Bulut @drokanbulut]

@guhergorgun and I will be presenting our research on automatic question generation, "Current Evaluation Methods are a Bottleneck in Automatic Question Generation," at @RealAAAI's #AAAI24 Workshop on #AI for #Education in Vancouver (Feb 2024). https://t.co/q4YNNkXv8r

[Cameron R. Wolfe, Ph.D. @cwolferesearch]

Due to the recent surge in popularity of AI and language models, one of the most common questions I hear is: How can we train a specialized LLM over our own data? The answer is actually pretty simple…

TL;DR: Training LLMs end-to-end is quite difficult due to the size of the model (unless you work for OpenAI and have a limitless supply of massive GPUs). But, LoRA—a parameter efficient finetuning technique—can easy finetune LLMs on more modest hardware and achieve comparable performance to a model that is fully trained.

Why is this difficult? Recent advances in generative AI are powered by massive models with many parameters, and training such an LLM requires expensive hardware (i.e., many expensive GPUs with a lot of memory) and fancy training techniques (e.g., fully-sharded data parallel training). For this reason, training these models end-to-end is quite difficult and expensive.

The training process. Luckily, LLMs are usually trained in two phases—pretraining and finetuning—where the former phase is (much) more expensive. Given that high-quality pretrained LLMs are readily available online, most AI practitioners can simply download a pretrained model and focus upon adapting this model (via finetuning) to their desired task. However, the size of the model does not change during finetuning, so this is still not easy.

Parameter-efficient finetuning (PEFT). As a solution to the issues outlined above, AI researchers have explored a variety of parameter-efficient finetuning techniques, such as prefix tuning and adapter layers. Instead of training the full model end-to-end, parameter-efficient finetuning leaves pretrained model weights fixed and only adapts a small number of task-specific parameters during finetuning, thus reducing memory overhead and allowing us to finetune LLMs with more reasonable hardware.

LoRA. One of the most widely used PEFT techniques is LoRA. The core idea behind LoRA is to model the finetuning update to the model’s parameters with a low-rank decomposition. LoRA leaves the pretrained layers of the LLM fixed and injects a trainable rank decomposition matrix, implemented in practice as a pair of linear projections, into each layer of the model. The output of these two linear projections is added to the output of the pretrained model.

Practical utility. LoRa has countless practical benefits, such as:
- A single pretrained model can be shared by several task-specific LoRA modules.
- It has no added inference latency, as we can combine the weight update from LoRA with pretrained model weights.
- We only train a small number of parameters, which drastically reduces memory overhead during finetuning.
- Finetuning with LoRa is faster than full finetuning (25% faster in the case of GPT-3).
- The resulting model still performs comparably to full finetuning.

“Full finetuning … would be 30-40x more expensive than the parameter-efficient LLaMA-Adapter or LoRA alternatives.” - from Sebastian Raschka

Training specialized LLMs. LoRA lowers the barrier to entry for training specialized LLMs. Finetuning with LoRa is faster and has a low memory overhead relative to training the full model, so we need fewer/smaller GPUs. As such, LoRA is a great (and accessible) first step in attempting to train a specialized LLM. If this doesn’t meet your needs, attempting full finetuning (or even pretraining from scratch) could yield better results at the cost of added complexity.

[Eiko Fried @EikoFried]

A lot of reforms in psych science in the last decade have focused on improving statistical practices (and that's great). But, as the many of us who have re-analyzed a lot of data can tell you: statistics really cannot help you much when your measures are problematic or invalid.

[Kız Kafası @cigdemerken]

Şu hayatta içimdeki hüzünü besleyen en büyük müzisyenlerden biriydi Astrud Gilberto. Sadece bir kişi gibi şarkı söyleyeceksin deseler onu seçerdim. Işıklar içinde uyusun. 🙏🏻♥️ #AstrudGilberto

[Ömer Zülfü Livaneli @LivaneliZulfu]

Zülfü Livaneli I deutschlandfunk Klassik Pop-et cetera programını sundu. Tamamı, 6 Temmuz 2023’de yayında. “Klassik-Pop-et cetera” Cumartesi sabahları Alman ulusal halk radyosunda 40 yılı aşkın bir süredir yayınlanıyor. (Yön.).

[Ece Temelkuran @ETemelkuran]

“How will you celebrate the day?” might sound too vague a question for many who aren’t from Turkey, but a no-brainer for us. “The day” refers to “when we get rid of Erdogan’s regime.” That’s why the video is more than a raki advert.:)

[Andrew Lampinen @AndrewLampinen]

The recent discussions of what language models can and can't accomplish highlight some important issues in how cognitive science, linguistics, etc. think about human capabilities or competencies, and thus how to test them in models. Thread: 1/

[The Nobel Prize @NobelPrize]

Irish novelist and playwright Samuel Beckett with some words to live by.

Beckett was awarded the 1969 Nobel Prize in Literature for his novels and plays.

[Sebastian Raschka @rasbt]

Are LLMs making it easier to write good essays? I’d say NO, they are just shifting the goalpost.

LLMs will help us to write "correctly". But crafting an original and engaging essay that someone wants to read will become more challenging in the future.

https://t.co/V8LWhQeXdk

[Adam Grant @AdamMGrant]

It's a mistake to stop saying "um" and "uh" altogether.

Evidence: filler words signal that new information is coming, making it easier for listeners to understand and remember what comes next.

Hesitations don't make you sound weak. They help you... uh... communicate clearly.

[Kareem Carr, Ph.D. @kareem_carr]

Introduction to Statistics courses are often extremely badly taught in my opinion.

They are full of unnecessary math.

This is unfortunate because statistical math can be tedious and kind of ugly which turns off a lot of people.

[Ethan Mollick @emollick]

I think we haven't fully absorbed the fact that careful academic papers have found ChatGPT clearly passes some of the most challenging American professional exams:
🩺United States Medical Licensing Exam
🎓MBA-level Operations exam
🧑‍⚖️The Bar Exam (based on typical exam questions)

[Naveen Rao @NaveenGRao]

LLMs pass/will pass most professional cert exams (Bar, USMLE, CFA, etc). You can interpret that to mean we're close to AGI.

...or that those exams have gaping holes in ascertaining readiness for a field.

I'm thinking the latter.

[Guher Gorgun @guhergorgun]

It was a wonderful experience to present our paper at #EDM2022 with @ErbasliSeyma and @cdemmansepp and our poster at #AIED2022 with @drokanbulut. I had great time, received many insightful comments, and meet with fantastic scholars. See you next year!

[Dashun Wang @dashunwang]

Steven Weinberg, one of the most preeminent physicists in history, passed away.

Here's his four lessons, published in Nature back to 2003:

☑️ No one knows everything, and you don't have to
☑️ Aim for rough water
☑️ Forgive yourself for wasting time
☑️ learn history of science

[Mark Tenenholtz @marktenenholtz]

What data scientists think they need for a good solution:

• Bigger servers
• Bigger data
• Bigger models

What they SHOULD be using for a good solution:

• A smaller model
• A deep knowledge of the data
• A debugger

Rapid iteration + solid theory + solid coding = success

[University of Alberta Faculty of Education @UAlbertaEd]

Congratulations to #UAlberta Education researchers Guher Gorgun and Gehan Hamad, whose respective work on improving online assessments earned them each an Alberta Innovates Graduate Student Scholarship!

[Tim Gill @timgill924]

I’m seeing so many academics hung up on manuscript rejections and how to cope. For starters, don’t take yourself too seriously. There’s a whole world out there: hiking, kayaking, organizing. Your research isn’t that important. Much of it isn’t even generative. Have perspective.