Murali Krishna

Startup Guy, Algorithms, AI, Machine Learning, AdTech, MarTech, Management, ToC, Open Source

[Murali Krishna @mk44214]

As time went on, young Elara and Leo kept nurturing the garden, recounting its tale, and welcoming travelers from across the globe. They aged side by side, encircled by their kin, the garden’s splendor, and the deep affection that had once united them.

[Murali Krishna @mk44214]

A dog jogs effortlessly along the road during a winter stroll, wrapped in a cozy scarf.

[Murali Krishna @mk44214]

⚡I am free to be myself.230243Tea with honey and rose Tea, drunk

[Murali Krishna @mk44214]

🐳Balcony cleaning Balcony, continuing to be cleaned641632Picking Blueberries The blueberries, the basket

[Murali Krishna @mk44214]

Upon returning to her apartment, she removes her jacket and sneakers before making her way to the bathroom for a brisk shower. The soothing warmth of the water caresses her skin, rinsing off all the perspiration from her stroll. A practical little shoulder bag hangs comfortably

[Murali Krishna @mk44214]

🦪Stay truthful with yourself and those around you.620082True courage isn't about being fearless in life; it's about moving forward calmly even while anxiety remains. Embrace your hesitations and stride boldly toward the goals that matter most to you.

[Murali Krishna @mk44214]

🐄Little actions can bring about significant changes. Stroll through narrow lanes lined with jasmine in ancient mountain villages, where a gentle, sweet scent accompanies every step, soothing the sharp, anxious thoughts that weigh down your tired, fragile inner spirit day after

[Murali Krishna @mk44214]

Your unwavering, soothing love wraps around every difficulty in my life like a protective layer, soothing the fiercest storms of negative emotions through your dependable and constant devotion. Embrace love in every fleeting second.

[Murali Krishna @mk44214]

Lola gazed up at him and said, 'I’m truly grateful we crossed paths. You’ve transformed my life in countless ways. I once believed selfies were merely for vanity, but now I understand they’re about seizing moments and embracing who you are.'💶Allow yourself enough compassion

[Murali Krishna @mk44214]

Brew chamomile tea. Calms366756 Every difficult training session builds our patience and perseverance, allowing us to remain calmer and more powerful when confronting various pressures and challenges.😫

[Andrej Karpathy @karpathy]

+1 for "context engineering" over "prompt engineering".

People associate prompts with short task descriptions you'd give an LLM in your day-to-day use. When in every industrial-strength LLM app, context engineering is the delicate art and science of filling the context window with just the right information for the next step. Science because doing this right involves task descriptions and explanations, few shot examples, RAG, related (possibly multimodal) data, tools, state and history, compacting... Too little or of the wrong form and the LLM doesn't have the right context for optimal performance. Too much or too irrelevant and the LLM costs might go up and performance might come down. Doing this well is highly non-trivial. And art because of the guiding intuition around LLM psychology of people spirits.

On top of context engineering itself, an LLM app has to:
- break up problems just right into control flows
- pack the context windows just right
- dispatch calls to LLMs of the right kind and capability
- handle generation-verification UIUX flows
- a lot more - guardrails, security, evals, parallelism, prefetching, ...

So context engineering is just one small piece of an emerging thick layer of non-trivial software that coordinates individual LLM calls (and a lot more) into full LLM apps. The term "ChatGPT wrapper" is tired and really, really wrong.

[Murali Krishna @mk44214]

I have 3 new followers from India 🇮🇳, and more last week. See https://t.co/6TaswRoLwF

[Murali Krishna @mk44214]

Wordle 1,078 4/6

🟨⬛🟨🟨⬛
⬛⬛⬛🟨⬛
🟨⬛🟨🟩🟩
🟩🟩🟩🟩🟩

#Wordle

[Murali Krishna @mk44214]

Special thank you to my 2 new followers from India last week. https://t.co/6TaswRoLwF

[Tomasz Łakomy @tlakomy]

The only person who should interview Putin

[Murali Krishna @mk44214]

Wordle 966 5/6
#Wordle

[Murali Krishna @mk44214]

@thepurpledoe Feeling incomplete.. definitely no...

I think whenever I end up in a relationship, I'd feel I am something more than what I am now ..

So.. if that does not happen.. I'm happy with who and what I am ..

[Jim Fan @DrJimFan]

In my decade spent on AI, I've never seen an algorithm that so many people fantasize about. Just from a name, no paper, no stats, no product. So let's reverse engineer the Q* fantasy. VERY LONG READ:

To understand the powerful marriage between Search and Learning, we need to go back to 2016 and revisit AlphaGo, a glorious moment in the AI history.
It's got 4 key ingredients:

1. Policy NN (Learning): responsible for selecting good moves. It estimates the probability of each move leading to a win.

2. Value NN (Learning): evaluates the board and predicts the winner from any given legal position in Go.

3. MCTS (Search): stands for "Monte Carlo Tree Search". It simulates many possible sequences of moves from the current position using the policy NN, and then aggregates the results of these simulations to decide on the most promising move. This is the "slow thinking" component that contrasts with the fast token sampling of LLMs.

4. A groundtruth signal to drive the whole system. In Go, it's as simple as the binary label "who wins", which is decided by an established set of game rules. You can think of it as a source of energy that *sustains* the learning progress.

How do the components above work together?

AlphaGo does self-play, i.e. playing against its own older checkpoints. As self-play continues, both Policy NN and Value NN are improved iteratively: as the policy gets better at selecting moves, the value NN obtains better data to learn from, and in turn it provides better feedback to the policy. A stronger policy also helps MCTS explore better strategies.

That completes an ingenious "perpetual motion machine". In this way, AlphaGo was able to bootstrap its own capabilities and beat the human world champion, Lee Sedol, 4-1 in 2016. An AI can never become super-human just by imitating human data alone.

-----
Now let's talk about Q*. What are the corresponding 4 components?

1. Policy NN: this will be OAI's most powerful internal GPT, responsible for actually implementing the thought traces that solve a math problem.

2. Value NN: another GPT that scores how likely each intermediate reasoning step is correct.
OAI published a paper in May 2023 called "Let's Verify Step by Step", coauthored by big names like @ilyasut @johnschulman2 @janleike: https://t.co/iAvXNjjhcK
It's much lesser known than DALL-E or Whipser, but gives us quite a lot of hints.

This paper proposes "Process-supervised Reward Models", or PRMs, that gives feedback for each step in the chain-of-thought. In contrast, "Outcome-supervised reward models", or ORMs, only judge the entire output at the end.

ORMs are the original reward model formulation for RLHF, but it's too coarse-grained to properly judge the sub-parts of a long response. In other words, ORMs are not great for credit assignment. In RL literature, we call ORMs "sparse reward" (only given once at the end), and PRMs "dense reward" that smoothly shapes the LLM to our desired behavior.

3. Search: unlike AlphaGo's discrete states and actions, LLMs operate on a much more sophisticated space of "all reasonable strings". So we need new search procedures.

Expanding on Chain of Thought (CoT), the research community has developed a few nonlinear CoTs:
- Tree of Thought: literally combining CoT and tree search: https://t.co/KM1P2ZJrjG @ShunyuYao12
- Graph of Thought: yeah you guessed it already. Turn the tree into a graph and Voilà! You get an even more sophisticated search operator: https://t.co/5ncT5tuTOY

4. Groundtruth signal: a few possibilities:
(a) Each math problem comes with a known answer. OAI may have collected a huge corpus from existing math exams or competitions.
(b) The ORM itself can be used as a groundtruth signal, but then it could be exploited and "loses energy" to sustain learning.
(c) A formal verification system, such as Lean Theorem Prover, can turn math into a coding problem and provide compiler feedbacks: https://t.co/vpOBOI2FR5

And just like AlphaGo, the Policy LLM and Value LLM can improve each other iteratively, as well as learn from human expert annotations whenever available. A better Policy LLM will help the Tree of Thought Search explore better strategies, which in turn collect better data for the next round.

@demishassabis said a while back that DeepMind Gemini will use "AlphaGo-style algorithms" to boost reasoning. Even if Q* is not what we think, Google will certainly catch up with their own. If I can think of the above, they surely can.

Note that what I described is just about reasoning. Nothing says Q* will be more creative in writing poetry, telling jokes @grok, or role playing. Improving creativity is a fundamentally human thing, so I believe natural data will still outperform synthetic ones.

I welcome any thoughts or feedback!!

[Murali Krishna @mk44214]

Special thank you to my 2 new followers from Vietnam last week. https://t.co/6TaswRoLwF

[Murali Krishna @mk44214]

@noworriesbehpy Why Compare ??