Ayman

[Ayman @aymantarig0]

الحمدلله حمدًا كثيرًا طيبًا
يوصى بمنحه درجة البكلاريوس في الهندسة الكهربائية والإلكترونية
Straight out of UofK 🥳

[Ayman @aymantarig0]

One should always seek truth, even if it challenges one's beliefs, for only in truth can genuine understanding and progress be found.

[Ayman @aymantarig0]

صباح آخر امتحان في جامعة الخرطوم 🥳

[Cameron Hudson @_hudsonc]

In Senator Rubio's confirmation this week to be the next Secretary of State was this very important exchange on #Sudan that gives, perhaps, some window into how he might approach the genocide there, including speaking frankly and openly about the role of outside powers. Listen!

[Ayman @aymantarig0]

ما عارف الحياة قبل ChatGPT كانت عامله كيف

[مختار @MukhtarIsmaeil]

تجربتي الجامعية مع الجبت بقت افضل سبعة الف مرة
شكراً @OpenAI

[Ayman @aymantarig0]

أي زول عندو حاجه واقفه ليو، تطبيق، موقع، أي فهم يمشي للهكر ده بس

[قاسم @mohdahmed202]

كملت 100 days of coding challenge 🥳🥳

[Ayman @aymantarig0]

@MAbdulkariim @Masy_11 check this

[Andrew Ng @AndrewYNg]

A barrier to faster progress in generative AI is evaluations (evals), particularly of custom AI applications that generate free-form text. Let’s say you have a multi-agent research system that includes a researcher agent and a writer agent. Would adding a fact-checking agent improve the results? If we can’t efficiently evaluate the impact of such changes, it’s hard to know which changes to keep.

For evaluating general-purpose foundation models such as large language models (LLMs) — which are trained to respond to a large variety of prompts — we have standardized tests like MMLU (multiple-choice questions that cover 57 disciplines like math, philosophy, and medicine) and HumanEval (testing code generation); the LMSYS Chatbot arena, which pits two LLMs’ responses against each other and asks a human to judge which response is superior; and large-scale benchmarking like HELM. These evaluation tools took considerable effort to build, and they are invaluable for giving LLM users a sense of different models' relative performance. Nonetheless, they have limitations: For example, leakage of benchmarks datasets’ questions and answers into training data is a constant worry, and human preference for certain answers does not mean those answers are more accurate.

In contrast, our current options for evaluating specific applications built using LLMs are far more limited. Here, I see two major types of applications.
- For applications designed to deliver unambiguous, right-or-wrong responses, we have reasonable options. Let’s say we want an LLM to read a resume and extract the candidate's most recent job title, or read a customer email and route it to the right department. We can create a test set that comprises ground-truth labeled examples with the right responses, and measure the percentage of times the LLM generates the right output. The main bottleneck is creating the labeled test set, which is expensive but surmountable.
- But many LLM-based applications generate free-text output with no single right response. For example, if we ask an LLM to summarize customer emails, there’s a multitude of possible good (and bad) responses. The same holds for an agentic system to do web research and write an article about a topic, or a RAG system for answering questions. It’s impractical to hire an army of human experts to read the LLM’s outputs every time we tweak the algorithm and evaluate if the answers have improved — we need an automated way to test the outputs. Thus, many teams use an advanced language model to evaluate outputs. In the customer email summarization example, we might design an evaluation rubric (scoring criteria) for what makes a good summary. Given an email summary generated by our system, we might prompt an advanced LLM to read it and score it according to our rubric. I’ve found that the results of such a procedure, while better than nothing, can also be noisy — sometimes too noisy to reliably tell me if the way I’ve tweaked an algorithm is good or bad.

The cost of running evals poses an additional challenge. Let’s say you’re using an LLM that costs $10 per million input tokens, and a typical query has 1000 tokens. Each user query therefore costs only $0.01. However, if you iteratively work to improve your algorithm based on 1000 test examples, and if in a single day you evaluate 20 ideas, then your cost will be 20*1000*0.01 = $200. For many projects I’ve worked on, the development costs were fairly negligible until we started doing evals, whereupon the costs suddenly increased. (If the product turned out to be successful, then costs increased even more at deployment, but that was something we were happy to see!)

In addition to the dollar cost, evals also have a significant time cost. Running evals on 1000 examples might take tens of minutes or even hours. Time spent waiting for eval jobs to finish also slows down the speed with which we can experiment and iterate over new ideas. Previously I wrote that fast, inexpensive token generation is critical for agentic workflows. This will also be useful for evals, which involve nested for-loops that iterate over a test set and different model/hyperparameter/prompt choices and therefore consume large numbers of tokens.

Despite the limitations of today's eval methodologies, I’m optimistic that our community will invent better techniques (maybe involving agentic workflows like reflection?) for getting LLMs to evaluate such output.

If you’re a developer or researcher and have ideas along these lines, I hope you’ll keep working on them and consider open sourcing or publishing your findings!

[Original text: https://t.co/HXtzJH7eP8 ]

[Medo @m7med_3dil]

إلى كل المعارف و الأصدقاء و من بإمكانه مساعدتنا في هذا الشأن

ابن عمي أبّي الحسن الخضر طالب بجامعة السودان تم اعتقاله من قبل الدعم السريع بتاريخ 14 أبريل 2024 وهو يسكن بحي الصحافة مربع (29). ابي يافع صغير لا علاقة له بتنظيم سياسي أو جماعة.

#اطلقوا_سراح_المعتقلين

[Ayman @aymantarig0]

هو نحن بجد "حنعمل شنو الليلة؟"

[آلاء. @allaalmouiz]

I miss reading that MTN advertisement saying "حنعمل شنو الليلة؟" while passing by this street. :(

[Ayman @aymantarig0]

@yassir_aladdin ف "نحن" نتيجة حاجات عشوائيه حاصله بس 😂

[Ayman @aymantarig0]

@yassir_aladdin Thoughts are fundamentally the result of random movements of electrons and ions in our brain.

[Ayman @aymantarig0]

@yassir_aladdin Congratulations ياخ 🔥🔥

[Ayman @aymantarig0]

"Happiness is a prison, Evey. Happiness is the most insidious prison of all."
أكتر عبارة حقيقية سمعتها في حياتي.

[F @KUR3LY]

I deeply hate phone calls

[Ayman @aymantarig0]

@9aaaba أتفق، مافيش فايدة اصلًا

[ريّان إنسان حر طليق ذ.م.م @just_shosho]

I’m only 25 and so far I’ve witnessed a revolution, a pandemic, military coup, and a war. And I’m expected to navigate my future and build a successful career at this age

كان جنيتوا قولوا نحن جنينا.