Eric Halverson

Ruby/Rails Engineer | Interests: coding, chess, hiking, biking, paddling, birding, art, movies, writing 🙂

[Eric Halverson @elhalvers]

Thank you @AndrewYNg @DeepLearningAI
I appreciate your perspective, care, energy and teaching style. Thank you for all of the good work! : )

[Eric Halverson @elhalvers]

Very good article! Fundamental and straight forward : )

[Akshay 🚀 @akshay_pachaar]

https://t.co/VoQByJUrBU

[Eric Halverson @elhalvers]

@_avichawla @DailyDoseOfDS_ Excellent! Thanks! : )

[Eric Halverson @elhalvers]

This is really helpful! : )

[Avi Chawla @_avichawla]

RAG vs. CAG, clearly explained!

RAG is great, but it has a major problem:

Every query hits the vector DB. Even for static information that hasn't changed in months.

This is expensive, slow, and unnecessary.

Cache-Augmented Generation (CAG) addresses this issue by enabling the model to "remember" static information directly in its key-value (KV) memory.

In fact, you can combine RAG and CAG for the best of both worlds.

Here's how it works:

RAG + CAG splits your knowledge into two layers:

↳ Static data (policies, documentation) gets cached once in the model's KV memory

↳ Dynamic data (recent updates, live documents) gets fetched via retrieval

This gives faster inference, lower costs, and less redundancy.

The trick is being selective about what you cache.

Only cache static, high-value knowledge that rarely changes. If you cache everything, you'll hit context limits. Separating "cold" (cacheable) and "hot" (retrievable) data keeps this system reliable.

You can start today. OpenAI and Anthropic already support prompt caching in their APIs.

I have shared my recent article on prompt caching below if you want to dive deeper.

👉 Over to you: Have you tried CAG in production yet?

[Eric Halverson @elhalvers]

@excid3 Congrats Chris!! 👏🥳

[Eric Halverson @elhalvers]

@rhiannon_io Where next? : )

[Eric Halverson @elhalvers]

Pretty great deal! Pay what you want : ) <em>LLM Engineer's Handbook</em> & help charity! https://t.co/zhm7HHfwbR via @humble

[Eric Halverson @elhalvers]

This is a big world we’re in. I’m happy to want to play and be a part of the conversation! : )

[Andrew Ng @AndrewYNg]

Separate reports by the publicity firm Edelman and Pew Research show that Americans, and more broadly large parts of Europe and the western world, do not trust AI and are not excited about it. (Links in original text, below.) Despite the AI community’s optimism about the tremendous benefits AI will bring, we should take this seriously and not dismiss it. The public’s concerns about AI can be a significant drag on progress, and we can do a lot to address them.

According to Edelman’s survey, in the U.S., 49% of people reject the growing use of AI, and 17% embrace it. In China, 10% reject it and 54% embrace it. Pew’s data also shows many other nations much more enthusiastic than the U.S. about AI adoption.

Positive sentiment toward AI is a huge national advantage. On the other hand, widespread distrust of AI means:
- Individuals will be slow to adopt it. For example, Edelman’s data shows that, in the U.S., those who rarely use AI cite Trust (70%) more than lack of Motivation and Access (55%) or Intimidation by the technology (12%) as an issue.
- Valuable projects that need societal support will be stymied. For example, local protests in Indiana brought down Google’s plan to build a data center there. Hampering construction of data centers will hurt AI’s growth. Communities do have concerns about data centers beyond the general dislike of AI; I will address this in a later letter.
- Populist anger against AI raises the risk that laws will be passed that hamper AI development.

To be clear, all of us working in AI should look carefully at both the benefits and harmful effects of AI (such as deepfakes polluting social media and biased or inaccurate AI outputs misleading users), speak truthfully about both benefits and harms, and work to ameliorate problems even as we work to grow the benefits. But hype about AI’s danger has done real damage to trust in our field. Much of this hype has come from leading AI companies that aim to make their technology seem extraordinarily powerful by, say, comparing it to nuclear weapons. Unfortunately, a significant fraction of the public has taken this seriously and thinks AI could bring about the end of the world. The AI community has to stop self-inflicting these wounds and work to win back society’s trust.

Where do we go from here?

First, to win people’s trust, we have a lot of work ahead to make sure AI broadly benefits everyone. “Higher productivity” is often viewed by general audiences as a codeword for “my boss will make more money,” or worse, layoffs. As amazing as ChatGPT is, we still have a lot of work to do to build applications that make an even bigger positive impact on people’s lives. I believe providing training to people will be a key piece of the puzzle. https://t.co/zpIxRSuky4 will continue to lead the charge on AI training, but we will need more than this.

Second, we have to be genuinely worthy of trust. This means every one of us has to avoid hyping things up or fear mongering, despite the occasional temptation to do so for publicity or to lobby governments to pass laws that stymie competing products (such as open source).

I hope our community can also call out journalism that spreads hype. For example, Nirit Weiss-Blatt wrote a remarkable article about how 60 Minutes’ coverage of an Anthropic study in which Claude, threatened with being shut down, resorted to “blackmail,” was highly misleading. The study carried out a red-teaming exercise in which skilled researchers, after a lot of determined work, finally pushed an AI system into a corner so it demonstrated “blackmailing” behavior. Unfortunately, news reports distorted this and led many to think the “blackmail” behavior occurred naturally rather than only because skilled researchers engineered it to happen. The reports left many with a wildly exaggerated picture of how often AI actually “schemes.” Red-teaming exercises are important to test vulnerabilities of systems, but this particular piece of hype, which was widely circulated, will hurt AI for a long time.

Living in Silicon Valley, I realize I live in a bubble of AI enthusiasts, which is great for exchanging ideas and encouraging each other to build! At the same time, I recognize that AI does have problems, and the AI community needs to address them. I frequently speak with people from many different walks of life. I’ve spoken with artists concerned about AI devaluing their work, college seniors worried about the tough job market and whether AI is exacerbating their challenges, and parents worried about their kids being addicted to, and receiving harmful advice from, chatbots.

I don’t know how to solve all of these problems, but I will work hard to solve as many as I can. And I hope you will too. It will only be through all of us doing this that we can win back society’s trust.

[Original text, with links: https://t.co/oi29S8uu6C ]

[Eric Halverson @elhalvers]

@zicokolter Cool! Looking forward to working through the online material when available! : )

[Eric Halverson @elhalvers]

Loving Andrew Ng’s courses! : )

[Andrew Ng @AndrewYNg]

https://t.co/zpIxRStMIw Pro is now generally available -- this is the one membership that keeps you at the forefront of AI. Please join!

There has never been a moment when the distance between having an idea and building it has been smaller. Things that required months of work for teams can now be built by individuals using AI, in days. This is why we built https://t.co/zpIxRStMIw Pro. I'm personally working hard on this membership program to help you to build applications that can launch or accelerate your career, and shape the future of AI.

https://t.co/zpIxRStMIw Pro gives you full access to 150+ programs, including my recently launched Agentic AI course, the new Post-Training and PyTorch courses by Sharon Zhou and Laurence Moroney (just released this week), and all of https://t.co/zpIxRStMIw's top courses and professional certificates.

All course videos remain free. Pro membership adds hands-on learning: labs to build working systems, practice questions to hone your understanding, and certificates to share your skills.

I'm also building new tools to help you create AI applications and grow your career (and have fun doing so!). Many will be available first to Pro members.

Try out https://t.co/zpIxRStMIw Pro free, and let me know what you build!

Join here: https://t.co/SmHogm1Whn

[rubyAndPolitics @caseyprovost]

Who needs a staff level rails developer for ongoing contract work?

Looking for 10-20hrs a week (nights and weekends). Prefer Hotwire/fullstack projects that are long term whether it’s a brand new project or staff aug on existing.

Expertise: fintech, security, hotwire, pairing.

[Eric Halverson @elhalvers]

@schwad_rb Have fun!! 🥳

[Eric Halverson @elhalvers]

@Python_Dv 45

[Eric Halverson @elhalvers]

@excid3 I like the Golden Ratio tee! : )

[Eric Halverson @elhalvers]

@anjaneyasevak @TivadarDanka Khan Academy : )

[Eric Halverson @elhalvers]

@TivadarDanka Thank you! That was fun! : )

[Eric Halverson @elhalvers]

@JackUnicyclist @launchschool Congrats Jack!! 🎉🥳

[Eric Halverson @elhalvers]

@rhiannon_io Happy Birthday!! 🎉🥳

[Eric Halverson @elhalvers]

@_avichawla Thank you! : )

[Eric Halverson @elhalvers]

@swapnakpanda 🙏🏼😊