Tim Courtenay

VP Sales with key focus on operationalizing AI in the workplace

[Tim Courtenay @courtenay_tim]

@TheAIColony What is https://t.co/wP8BsOEQ2x

[Tim Courtenay @courtenay_tim]

This is VERY significant. It makes sense to have a 'workforce' of 'compact', fast, cost-effective agents that are fit-for-purpose and have a low consumption footprint. They can escalate to the smartypants AI when they need to!

[Andrej Karpathy @karpathy]

LLM model size competition is intensifying… backwards!

My bet is that we'll see models that "think" very well and reliably that are very very small. There is most likely a setting even of GPT-2 parameters for which most people will consider GPT-2 "smart". The reason current models are so large is because we're still being very wasteful during training - we're asking them to memorize the internet and, remarkably, they do and can e.g. recite SHA hashes of common numbers, or recall really esoteric facts. (Actually LLMs are really good at memorization, qualitatively a lot better than humans, sometimes needing just a single update to remember a lot of detail for a long time). But imagine if you were going to be tested, closed book, on reciting arbitrary passages of the internet given the first few words. This is the standard (pre)training objective for models today. The reason doing better is hard is because demonstrations of thinking are "entangled" with knowledge, in the training data.

Therefore, the models have to first get larger before they can get smaller, because we need their (automated) help to refactor and mold the training data into ideal, synthetic formats.

It's a staircase of improvement - of one model helping to generate the training data for next, until we're left with "perfect training set". When you train GPT-2 on it, it will be a really strong / smart model by today's standards. Maybe the MMLU will be a bit lower because it won't remember all of its chemistry perfectly. Maybe it needs to look something up once in a while to make sure.

[Tim Courtenay @courtenay_tim]

This is so true. These 'little' ad-hoc tasks create a virtuous circle. Using LLMs more -> becoming more creative in using them-> using them more....

[Ethan Mollick @emollick]

I often find a half dozen little one-off uses for LLMs every day, saving me a lot of time. A Swiss army knife for the mind.

[Tim Courtenay @courtenay_tim]

We are seeing this in #AIForWork - limit the prompt or RAG to things the LLM doesn't already know, and get a better output. Keep it focused and tight.

[Philipp Schmid @_philschmid]

How faithful are LLMs in RAG applications? 🤔 A recent paper from @Stanford explores how well LLMs respect retrieved information or fallback to their internal knowledge. The goal is to quantify the tension between LLMs’ internal knowledge and the retrieved information

Experiments:
1️⃣ Create a dataset from six domains using web content with questions, answers, and k references
2️⃣ Manipulate samples and introduce errors in reference documents (e.g. 20 → 60)
3️⃣ Create a Baseline using LLM without context
4️⃣ Iterate over dataset with manipulated documents (0 errors, 1 error, 2 …)
5️⃣ Determine if LLMs prefer internal knowledge or RAG data.

Insights
🔼 Using correct retrieved information increases accuracy from 35.7% to 94%.
🛑 Manipulated retrieved information led to LLM citing the wrong information.
💪 Stronger LLMs (GPT-4) resist incorrect retrieved information better.
👮 "strict" prompts (YOU MUST) helped respect retrieved information.
🔄 RAG most improvements for up-to-date information and questions.

Paper: https://t.co/zGEruMgi5n

[Tim Courtenay @courtenay_tim]

How much work is going into clever AI workarounds that will be obsolete in the next 6-12 months?

[Tim Courtenay @courtenay_tim]

Read this #AIForWork #SAIles people. Or get AI to summarise the 300 pages for you :).
- LLMs are getting very good, quickly
- Learn top-down to exploit AI in your business, or become a victim
AI Index: State of AI in 13 Charts https://t.co/MAkBpGVvQt

[Tim Courtenay @courtenay_tim]

#Forbes 2024 AI 50 List - Top Artificial Intelligence Startups: By spring of 2023, the massive popula... via @forbes https://t.co/ru6Kw4sTo1

[Tim Courtenay @courtenay_tim]

Great interview.

[Chris Madden @chrismadden_]

Eric Vaughan is doing something incredible at his companies...

A year ago he institutionalized an AI-First culture after the reveal of ChatGPT 3.5 — asking his 100+ employees to experiment and teach each other AI tools.

Then he started “AI Mondays” where everything employees did had to be learning about AI, or AI tangental activities.

At @ImagineAILive he announced his company’s new product - Jive Personas, the ability to create AI Clones of your key employees (and their knowledge base) that everyone in the company can interact with, saving everyone time.

Amazing to sit down with Eric @TheGenAICEO and so clearly see the efficiencies AI is already providing businesses!

Recapping Imagine AI Live conference
Eric Vaughan and his two companies @IgniteTech and @GFISoftware
Eric’s AI-First culture explained
How to hire for an AI-First culture
AI Clones in the workplace
Their product Jive Personas explained
Why create AI Clones?
Technology behind the AI Clones
Eric’s urgent message about AI

Eric’s full presentation is available in our membership area on the Imagine AI Live website and will be released on YouTube in the coming month.

If you found this conversation fascinating, please share it. Thank you!

[Tim Courtenay @courtenay_tim]

Two interesting talks from SXSW 2024. Ray Kurzweil (Singularity) https://t.co/iO3SeHzqkM - Scary? Hyped? .... and Billion Dollar Teams: The Future of an AI Powered Workforcehttps://youtu.be/1MZoxuxAfTM?si=-STEziItJJMmKgay. - some great specifics on AI at work

[Tim Courtenay @courtenay_tim]

No doubt (and hopefully) the lead will change continuously....important to abstract work from the choice of LLM as much as possible.

[Min Choi @minchoi]

Claude 3 Opus just dethroned GPT-4 on Arena Leaderboard.

And people are finding wild ways to use Claude to enhance their workflows.

10 examples:

[Tim Courtenay @courtenay_tim]

I must say, I often use the 'Pause and wait for my instruction before continuing' approach, when using a multi-step prompt, and it absolutely makes a difference. Tricky when using an API though.

[Ethan Mollick @emollick]

I love that the low hanging fruit for improving performance for LLMs are things like forcing them to pause before replying or giving them an inner monologue or providing a backspace key.

On one hand, it is bad to anthropomorphize AI, on the other, LLMs are pretty weird so 🤷‍♂️

[Tim Courtenay @courtenay_tim]

I think this is hugely important!! Even if you are non-technical. We need to think about how we need to think about using AI..

[Andrew Ng @AndrewYNg]

I think AI agentic workflows will drive massive AI progress this year — perhaps even more than the next generation of foundation models. This is an important trend, and I urge everyone who works in AI to pay attention to it.

Today, we mostly use LLMs in zero-shot mode, prompting a model to generate final output token by token without revising its work. This is akin to asking someone to compose an essay from start to finish, typing straight through with no backspacing allowed, and expecting a high-quality result. Despite the difficulty, LLMs do amazingly well at this task!

With an agentic workflow, however, we can ask the LLM to iterate over a document many times. For example, it might take a sequence of steps such as:
- Plan an outline.
- Decide what, if any, web searches are needed to gather more information.
- Write a first draft.
- Read over the first draft to spot unjustified arguments or extraneous information.
- Revise the draft taking into account any weaknesses spotted.
- And so on.

This iterative process is critical for most human writers to write good text. With AI, such an iterative workflow yields much better results than writing in a single pass.

Devin’s splashy demo recently received a lot of social media buzz. My team has been closely following the evolution of AI that writes code. We analyzed results from a number of research teams, focusing on an algorithm’s ability to do well on the widely used HumanEval coding benchmark. You can see our findings in the diagram below.

GPT-3.5 (zero shot) was 48.1% correct. GPT-4 (zero shot) does better at 67.0%. However, the improvement from GPT-3.5 to GPT-4 is dwarfed by incorporating an iterative agent workflow. Indeed, wrapped in an agent loop, GPT-3.5 achieves up to 95.1%.

Open source agent tools and the academic literature on agents are proliferating, making this an exciting time but also a confusing one. To help put this work into perspective, I’d like to share a framework for categorizing design patterns for building agents. My team AI Fund is successfully using these patterns in many applications, and I hope you find them useful.

- Reflection: The LLM examines its own work to come up with ways to improve it.
- Tool use: The LLM is given tools such as web search, code execution, or any other function to help it gather information, take action, or process data.
- Planning: The LLM comes up with, and executes, a multistep plan to achieve a goal (for example, writing an outline for an essay, then doing online research, then writing a draft, and so on).
- Multi-agent collaboration: More than one AI agent work together, splitting up tasks and discussing and debating ideas, to come up with better solutions than a single agent would.

I’ll elaborate on these design patterns and offer suggested readings for each next week.

[Original text: https://t.co/y4McIAjD2m]

[Tim Courtenay @courtenay_tim]

The rate of technological advancement in AI is unprecedented. -ve anecdotes about AI seem to exaggerate lack of trust, compared to +ve stories building trust. Is there a gap for an independent trust verification agency?

[Bernard Marr @BernardMarr]

This article explores the decline in public #trust towards #AI, emphasizing the importance of #ethical implementation and transparent, #secure, and #accountable AI practices to regain trust and fully realize AI's potential for societal good.

https://t.co/NQs8p2x1P0

[Tim Courtenay @courtenay_tim]

Another great article from Ethan Mollick. Awareness of when to be an AI 'Centaur' or 'Cyborg' is important.

I, Cyborg: Using Co-Intelligence, by @emollick https://t.co/shs5SN2Mu6

[Tim Courtenay @courtenay_tim]

Hey this is super useful thanks Dmitry!

[Dmitry Bakaev @bakaevd]

If you are playing around with the new Claude 3, check out this prompt library from Anthropic for examples. About 60+ prompts you can use as is or as inspiration: https://t.co/FclMMwe4vv

[Tim Courtenay @courtenay_tim]

Perplexity said to be close to unicorn status ($1bn valuation)...from just over $500m in January.

https://t.co/3k1N4Yzo9A

[Tim Courtenay @courtenay_tim]

@emollick Is this not similar to the 'testing effect' that humans experience? https://t.co/gtGGougdN2

[Tim Courtenay @courtenay_tim]

This interesting. I'm not familiar with the benchmarks, but would the models not inherently improve their performance over time anyway on the same benchmark as they 'learn' from the tests?

[Ethan Mollick @emollick]

And then there were three...

I got access to the new Anthropic Claude 3 AI a few days ago, so not enough time for a full review, but it was obvious it was GPT-4 class even before they released the testing stats. At the same time, like Gemini Advanced, it doesn't blow GPT-4 away.

[Tim Courtenay @courtenay_tim]

@_nerdai_ @VectorInst @llama_index Great, simple explainer of RAG. Whatever you level of involvement with AI, you should know this.

[Tim Courtenay @courtenay_tim]

Recommending anyone who is at MWC, GO TO THIS KEYNOTE!! :) You will not be disappointed!

[Totogi @totogi]

It's Fri-yay and we're off to Barcelona! 🇪🇸 Totogi's CEO, Danielle Rios Royston, will be giving a talk at the MVNO Summit on Feb 28th. Expect a live demo of telco's first AI-generated custom BSS! 💥

Register here: https://t.co/v7mDMPjMWb

#GSMA #MWC24 #BSS #AI @TelcoDR @MWCHub