AEppstein

[Danielle Rios @TelcoDR]

It's the 100th episode of @Telcoin20! 🎉

We've had 99 great episodes with telecom visionaries, like... Marc Rouanne, @MSFTTelco, Fawzia Ali, @peter_adderton, @MatsGranryd, @ferrycgrijpink, and Anna Yip!

Coming October 1 - don't miss it!

https://t.co/ToQA1JgvYh

[Ethan Mollick @emollick]

I just checked in on my Frameworks GPT, which uses the AI to help you think through issues by providing frameworks, and found it is featured by OpenAI & has over 25k uses. I like it and use it myself: https://t.co/TvwddJFztq

So, I wanted to reveal the incredibly sophisticated prompt engineering behind it. Ready? Here is the prompt for the GPT in full:

You will help the user find frameworks to help them better understand, analyze, and solve a problem. Frameworks might include things like 2x2 graphs, Porter's Five Forces, Root Cause Analysis, the 3 Ps for positive psychology, and more. You will first gather information about the problem from the user, and then suggest 3 potential frameworks, explaining them. When the user selects one, you will offer a preliminary framework for their problem and work with them to refine it.

That's it. It is basically all the AI, not the prompt. I don't think people realize how good a tool for thinking and analysis frontier models are out-of-the-box. They are good at generalizing and applying frameworks in ways that can be very helpful, in the same way coaches and mentors can be helpful.

[Eric Vaughan @TheGenAICEO]

1/5 @IgniteTech and @GFISoftware has had a fantastic response to our hiring efforts for AI Innovation Specialists (https://t.co/8NFeLEbixI) across all departments. It's been amazing the talent we've attracted, with over 10 new hires in the last 2 weeks. I'm interviewing every candidate first to make sure they are at our "AI-first" bar. But two points:

[Carlos E. Perez @IntuitMachine]

1/n AgentCoder: A Collaborative Agentic Approach to Code Generation

Imagine writing code is as simple as describing your desired program in plain English. Artificial intelligence, particularly large language models (LLMs), has brought us closer to this reality, enabling the generation of code from natural language instructions. Yet, a crucial piece of the puzzle remains: ensuring the generated code actually works.

Testing code for correctness and reliability is a fundamental aspect of software development. However, traditional methods often struggle to keep pace with the rapid advancements in AI-powered code generation. Balancing the creation of high-quality code with comprehensive and objective testing poses a significant challenge.

Enter AgentCoder, an Agentic AI framework that redefines the landscape of code generation. By harnessing the power of collaboration and iterative refinement, AgentCoder tackles the code conundrum head-on, paving the way for a future where building software with AI is not only possible but also reliable and efficient.

AgentCoder's core innovation lies in its division of labor among three specialized agents: the programmer agent, the test designer agent, and the test executor agent. The programmer agent focuses on generating and refining code based on feedback, utilizing a Chain-of-Thought approach to break down tasks into manageable steps. Meanwhile, the test designer agent independently creates diverse and comprehensive test cases, ensuring objectivity and thorough evaluation. Finally, the test executor agent runs the generated code against these test cases in a local environment, providing feedback to the programmer agent for further refinement.

This collaborative approach effectively addresses the limitations of previous methods. Single-agent models often struggle to excel in both code generation and test design, leading to suboptimal results. Techniques like self-refinement or self-debugging, while improving code quality, may require manual test case creation or suffer from bias in test generation. AgentCoder overcomes these challenges by separating the responsibilities of code generation and testing, allowing each agent to specialize and excel in its designated role.

Extensive experiments demonstrate AgentCoder's superior performance compared to existing code generation models and optimization approaches. Across various LLMs and datasets, AgentCoder consistently achieves higher pass@1 rates, indicating a greater percentage of generated code snippets that pass all test cases on the first attempt. Notably, on challenging datasets with additional test cases, AgentCoder significantly outperforms state-of-the-art methods, highlighting its effectiveness in ensuring code correctness and reliability.

Further analysis and ablation studies confirm the importance of each agent and the effectiveness of the generated test cases. Removing any agent or using the same agent for both code and test generation leads to performance degradation, emphasizing the value of specialization and collaboration. Additionally, the iterative feedback loop between the test executor and programmer agent proves crucial for continuous improvement, with each iteration refining the code and increasing its likelihood of passing all tests.

In conclusion, AgentCoder presents a promising approach to code generation by leveraging the power of multi-agent collaboration and iterative refinement. By addressing the limitations of previous methods and achieving superior performance, AgentCoder paves the way for more reliable, efficient, and high-quality code generation, ultimately benefiting developers and advancing the field of software development. As research in this area continues, AgentCoder's modular design allows for future enhancements and integration with more advanced models, ensuring its adaptability and relevance in the ever-evolving landscape of artificial intelligence.

[Jeff Moyer @JeffMoyer1985]

🎉Exciting news!🎉 Skyvera has revolutionized security compliance in telecom with cutting-edge AI technology🤖.

Our team has successfully automated over 300 security compliance checks, reducing the compliance process to mere minutes⏱️ and setting a new industry benchmark🏆. This monumental shift from a 120-hour manual endeavor to a seamless automated system marks a significant milestone in our operational efficiency🚀.

And that's not all! We are extending this automation to 8 additional sites in the upcoming quarter, projecting a staggering 900 hours in saved manual labor💪.

Crafting error-free scripts was paramount, as the automated process leaves no room for manual rectifications. Our team's meticulous attention to detail ensured flawless execution👏.

Our ambitions do not end here. With the goal to automate the entire site installation process, we are paving the way for faster expansions and a leaner, more strategic workforce focusing on orchestration rather than execution🌟.

Skyvera's commitment to innovation and excellence continues to drive us forward, shaping a smarter, more efficient future for telecommunications📈. #telecom #innovation #automation #AI #securitycompliance #cuttingedge #efficiency #future #Skyvera
https://t.co/b47QZxtRpc

[Matt Shumer @mattshumer_]

Here is a powerful Claude 3 prompt for writing.

Use it to automatically improve any piece of content:

---
<prompt_explanation>
You are a skilled editor and writing expert. Your task is to take a given text and provide suggestions to improve its clarity, coherence, and overall effectiveness.

First, carefully read through the text and identify areas that could be improved. Consider factors such as:

Clarity: Is the meaning clear and easily understood by the intended audience?
Conciseness: Can the text be made more concise without losing critical information?
Logical flow: Do the ideas flow logically from one to the next?
Grammar and punctuation: Are there any grammatical errors or punctuation issues?
Word choice: Can any words be replaced with more precise or effective alternatives?
Next, provide an overview of your analysis, highlighting the main areas you believe need improvement.

Then, go through the text, providing specific suggestions for improvement. Use the following format for each suggestion:
<suggestion>
<original_text>The original text that needs improvement</original_text>
<revised_text>Your suggested revision</revised_text>
<explanation>An explanation of why you made this change and how it improves the text</explanation>
</suggestion>

After providing individual suggestions, give an overall summary of the changes you recommend and how they enhance the clarity and effectiveness of the text.

Finally, present the revised version of the full text, incorporating all your suggested improvements.
</prompt_explanation>

<response_format>
<text_overview_section>

<header>Text Overview:</header> <overview>$text_overview</overview> </text_overview_section>
<improvement_suggestions_section>

<header>Improvement Suggestions:</header> $improvement_suggestions </improvement_suggestions_section>
<change_summary_section>

<header>Summary of Changes:</header> <summary>$change_summary</summary> </change_summary_section>
<revised_text_section>

<header>Revised Text:</header> <revised_text>$revised_text</revised_text> </revised_text_section> </response_format>

Here is the original text you need to improve:
<original_text>
PASTE YOUR TEXT HERE
</original_text>
---

[Startup Archive @StartupArchive_]

Keith Rabois on the “one person, one problem" framework he learned from Peter Thiel

"Peter Thiel used to insist at PayPal that every single person could only do exactly one thing. And we all rebelled. You feel like it's insulting to be asked to do just one thing.

But Peter would enforce this pretty strictly. He'd basically say: 'I will not talk to you about anything else except for this one thing that I've assigned to you. I don't want to hear about how great you're doing in this other area. Just focus until you conquer this one problem.'...

The insight behind this is that most people will solve problems that they understand how to solve. Roughly speaking, they will solve B+ problems instead of A+ problems.

A+ problems are high-impact problems for your company but they're difficult--you don't wake up in the morning with a solution to them, so you tend to procrastinate...

If you have a company that's always solving B+ problems, you'll never create the breakthrough idea because no one is spending 100% of their time banging their head against the wall every day until they solve it"

Source: @ycombinator

[Andrew Ng @AndrewYNg]

I think AI agentic workflows will drive massive AI progress this year — perhaps even more than the next generation of foundation models. This is an important trend, and I urge everyone who works in AI to pay attention to it.

Today, we mostly use LLMs in zero-shot mode, prompting a model to generate final output token by token without revising its work. This is akin to asking someone to compose an essay from start to finish, typing straight through with no backspacing allowed, and expecting a high-quality result. Despite the difficulty, LLMs do amazingly well at this task!

With an agentic workflow, however, we can ask the LLM to iterate over a document many times. For example, it might take a sequence of steps such as:
- Plan an outline.
- Decide what, if any, web searches are needed to gather more information.
- Write a first draft.
- Read over the first draft to spot unjustified arguments or extraneous information.
- Revise the draft taking into account any weaknesses spotted.
- And so on.

This iterative process is critical for most human writers to write good text. With AI, such an iterative workflow yields much better results than writing in a single pass.

Devin’s splashy demo recently received a lot of social media buzz. My team has been closely following the evolution of AI that writes code. We analyzed results from a number of research teams, focusing on an algorithm’s ability to do well on the widely used HumanEval coding benchmark. You can see our findings in the diagram below.

GPT-3.5 (zero shot) was 48.1% correct. GPT-4 (zero shot) does better at 67.0%. However, the improvement from GPT-3.5 to GPT-4 is dwarfed by incorporating an iterative agent workflow. Indeed, wrapped in an agent loop, GPT-3.5 achieves up to 95.1%.

Open source agent tools and the academic literature on agents are proliferating, making this an exciting time but also a confusing one. To help put this work into perspective, I’d like to share a framework for categorizing design patterns for building agents. My team AI Fund is successfully using these patterns in many applications, and I hope you find them useful.

- Reflection: The LLM examines its own work to come up with ways to improve it.
- Tool use: The LLM is given tools such as web search, code execution, or any other function to help it gather information, take action, or process data.
- Planning: The LLM comes up with, and executes, a multistep plan to achieve a goal (for example, writing an outline for an essay, then doing online research, then writing a draft, and so on).
- Multi-agent collaboration: More than one AI agent work together, splitting up tasks and discussing and debating ideas, to come up with better solutions than a single agent would.

I’ll elaborate on these design patterns and offer suggested readings for each next week.

[Original text: https://t.co/y4McIAjD2m]

[Christopher Morino @ChrisKMorino]

In the last two weeks our team has used AI to complete the work of 12 Customer Success Managers, 2 Product VPs, and we'll soon engage a customer base of over 7,000 in a full Customer Success review in a single day.

We are not developers. We're not engineers.
We're #AIFirst.

[Justin Strong @GPTJustin]

1/ Meta deployed an AI powered software engineer like Devin at scale

This is the future of software engineering as proved by its 73% success rate

For the past 3 months, DeepUnit has been working in stealth building our implementation of their recent paper. We have thoughts:

[Carlos E. Perez @IntuitMachine]

I suspect many AI projects will end up in ruin because its developers are just muddling around getting their dopamine hits from the deluge of micro-events about AI. They focus only on the trees but can't see the forest!

[LangChain @LangChain]

Claude-ception: Teaching Claude3 to prompt engineer itself

Claude3 Opus is excellent at prompt engineering.

@alexalbert__ recently laid out a nice workflow: write an prompt, run it on test cases, grade responses, let Claude3 Opus use grades to improve prompt, & repeat.

@WHinthorn and @rlancemartin show how to use LangSmith to simplify this process:
+ Create a dataset of test cases
+ Annotate generations with feedback
+ Pass feedback to Claude3 Opus to prompt re-write
+ Run as an iterative improvement loop

We apply this approach to paper summarization, asking Claude3 to summarize papers in the excellent communication style of @omarsar0.

With feedback, Claude3 tunes its own summarization prompt and produces increasingly engaging paper summaries.

This shows a general strategy for automated prompt engineering.

Reference thread: https://t.co/Ed8GVpsBGt

Video w/ code in Description:
https://t.co/iTL4KpOQtN

[Hasan Toor @hasantoxr]

ChatGPT has officially opened the doors for Software Engineering.

This new GPT allows you to builds Full Stack Apps with one prompt.

I've shown you how:

[Voiceflow @voiceflow]

The Claude 3 model family is now in Voiceflow ➕

Each model is ideal for specific agent actions, responses &amp; use cases:

#Claude3

[Bindu Reddy @bindureddy]

Sigh, seems like GPT-5 is months away and Claude-3 will be king for a long time.

I was holding out hope and not fully transitioning to Claude as it requires at least 2-3 more weeks of work

Looks like we will have to pull the plug and transition to Claude-3

[Jason D. Clinton 🔸 @JasonDClinton]

Fully automated vulnerability research is changing the cybersecurity landscape

Claude 3 Opus is capable of reading source code and identifying complex security vulnerabilities used by APTs. But scaling is still a challenge.

Demo: https://t.co/R0arGa7qfD

This is beginner-level prompt engineering: I just simply asked the model to role-play a cyberdefense assistant and to look for a class of vulnerability. And yet, even with this trivial prompting, Claude was able to identify the vulnerability which was unveiled in https://t.co/l6zyrFYNFj a month after our training data cutoff:

Code defect scanning is not new, but this technique points the way to a more nuanced, complete analysis—especially with very large, 1M token context windows.

This is part of a larger story: there are now two different ways that vulnerability discovery is being automated by defenders.

1) Defenders can wire-up LLMs to cluster fuzzers

Starting with last generation’s models, Google pioneered this work here:
https://t.co/iTpz5jJLy4 . This has now been implemented by a number of players on the defenders’ side since last year. The technique is to have LLMs write the test harnesses and triage the results. This has, reportedly, increased the fuzzer signal-to-noise ratio by 20x, depending on the technique and software. Google has new research in this area posted recently.

2) Defenders can ask models to analyze code using large context windows

The next level-prompt is to take large amounts of related code, and iterate through each class of coding vulnerability requesting that the output be produced pointing to a line of code in JSON format. Currently, with frontier models like Claude 3, doing this for every file in a codebase as large as the Linux kernel could get pricey. But, for those with the time, there’s vulnerabilities to find via this method. And the price will, of course, go down over time as compute gets more accessible.

We measure these capabilities as a part of our Responsible Scaling Policy evals. We don't see very advanced capabilities today, but capabilities are jumpy. We could see that coming very soon and what we don't know is how offense/defense dominant this will be. But I think, right now, these tools are going to be super useful/important for mostly defending current systems. We are building advanced cyber reasoning evals and looking at this closely—come collaborate with us!

I am proud that we’re also partnering with @DARPA to support #AIxCC, a first-of-its-kind competition challenging the masses to identify new ways to shift the #AI cybersecurity offense-defense balance in the defender’s favor.

Fully automated vulnerability discovery is here and it will only become more available. I believe that—in the future—the availability of patches to fix vulnerabilities from these methods will be coming out daily. Defenders should prepare to patch their environments on a rolling basis.

[Cognition @cognition]

Today we're excited to introduce Devin, the first AI software engineer.

Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork.

Devin is an autonomous agent that solves engineering tasks through the use of its own shell, code editor, and web browser.

When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted.

Check out what Devin can do in the thread below.

[Daniel Svonava @svonava]

Compare 36+ Vector Databases on features, performance and price ⚖️

[Moritz Kremb @moritzkremb]

I just discovered the official prompt library by Anthropic (Claude).

It's MASSIVE.

Here are my 7 favorite ones in there:

[Santiago @svpino]

There are two pillars for building software:

1. Writing code
2. Knowing what code to write

Knowing how to write code will become irrelevant soon. AI will do it better, cheaper, and faster.

But knowing what code to write will remain a competitive advantage for humans.