Spencer Lyon

Econ PhD out of NYU Stern. CEO and founder @ValorumData

[Spencer Lyon @spencer_lyon]

And I thought I couldn't love #ClaudeCode ultrathink more...

[Spencer Lyon @spencer_lyon]

MIT report states 95% of generative AI pilots at companies are failing 🤯. https://t.co/pzZULMgCnU

It doesn’t have to be this way! Based on personal projects, consulting engagements, and hours of study/research/reading, I believe there is a way to reliably employ this awesome tech!

To that end, I’m writing a book for leadership at midsize companies to make sense of AI. I’m planning on including

- A framework for how to understand AI’s capabilities
- A scoring system for identifying AI opportunities that are most likely to succeed and provide value
- How to assemble the right team
- An implementation roadmap
- ... more

Send me a message if you are interested in seeing some pre-release chapters, have ideas or experiences to share, or are curious to learn more!

[Spencer Lyon @spencer_lyon]

I’m working a lot with LLM based coding agents like Claude code (my personal preference), Gemini cli (so fast!), and cursor (nice integration). Two main issues I keep hitting (1) keeping them on task and (2) keeping them busy!

Anyone else facing these same issues? If not, what are you seeing?

I'm building a new project to help maximize human-LLM efficiency.

It builds on solid foundations of (1) planning + prompting (2) context management and (3) tool calling models and ties them together in a developer optimized flow.

Reach out if you are interested in a sneak peek

[Spencer Lyon @spencer_lyon]

AI coding assistants like Claude Code, Cursor & Aider are stateless—they send your entire chat history with each request.

This “limitation” is actually a superpower: sessions are fully resumable!

Start coding on your phone, continue on your laptop. Computer crashes? Just `claude --resume`

https://t.co/odnqxdVLeR

#AI #coding #productivity #TIL

[Spencer Lyon @spencer_lyon]

Call for Speakers for ElixirConf® US 2025 is open on Sessionize and I've just submitted a session! https://t.co/TpHAcf813S

[Spencer Lyon @spencer_lyon]

Working with @cursor_ai today on an #elixir @elixirphoenix project. A few things I have learned.

1. Use the `.cursor/rules` directory. It is really awesome!
2. I tried o3-mini-high, gemini 2.0 thinking, gemini 2.0 pro for creating a PRD for a new feature. o3-mini-high was best

[Spencer Lyon @spencer_lyon]

@marclou It will inevitably end ;)

Jokes aside, thanks for the recommendation! I’ll check it out

[Andrej Karpathy @karpathy]

New 3h31m video on YouTube:
"Deep Dive into LLMs like ChatGPT"

This is a general audience deep dive into the Large Language Model (LLM) AI technology that powers ChatGPT and related products. It is covers the full training stack of how the models are developed, along with mental models of how to think about their "psychology", and how to get the best use them in practical applications.

We cover all the major stages:
1. pretraining: data, tokenization, Transformer neural network I/O and internals, inference, GPT-2 training example, Llama 3.1 base inference examples
2. supervised finetuning: conversations data, "LLM Psychology": hallucinations, tool use, knowledge/working memory, knowledge of self, models need tokens to think, spelling, jagged intelligence
3. reinforcement learning: practice makes perfect, DeepSeek-R1, AlphaGo, RLHF.

I designed this video for the "general audience" track of my videos, which I believe are accessible to most people, even without technical background. It should give you an intuitive understanding of the full training pipeline of LLMs like ChatGPT, with many examples along the way, and maybe some ways of thinking around current capabilities, where we are, and what's coming.

(Also, I have one "Intro to LLMs" video already from ~year ago, but that is just a re-recording of a random talk, so I wanted to loop around and do a lot more comprehensive version of this topic. They can still be combined, as the talk goes a lot deeper into other topics, e.g. LLM OS and LLM Security)

Hope it's fun & useful!
https://t.co/75mXcUBI8L

[Spencer Lyon @spencer_lyon]

Yesterday I participated in a Q/A panel for an audience of MBA students on the importance of technical vs soft skills in today's workplace. Of course the use of "AI" came up and I had a realization based on my personal experience over the last few months...

LLMs are becoming good enough now that individuals who know how to communicate effectively with them (a real skill that takes practice!) act more like managers of a full team than individual contributors. This shift includes a change in the total output achievable and the day to day details of the job.

Thoughts?

[Spencer Lyon @spencer_lyon]

@Tim_Denning @readwise save thread

[Spencer Lyon @spencer_lyon]

Another win for the @huggingface
ecosystem! I encourage my students to become familiar with huggingface and things like this are a big reason why. Across models, datasets, libraries, and spaces; huggingface allows everyone to plug into AI research and applications.

[Daily Dose of Data Science @DailyDoseOfDS_]

Hugging Face released its own low-code library to build Agents ⚡️

smolagents is a 100% open-source library that enables you to run powerful agents in as low as three lines of code.

↳ Import the necessary modules.
↳ Define the agent
↳ Specifying the LLM and tools it can access.
↳ Run the agent!

Done!

Here’s what makes it special:

- Supports 40+ LLMs
- Share and load tools to/from the HF Hub.
- CodeAgent, that writes its actions in code.

[Spencer Lyon @spencer_lyon]

@pyquantnews Awesome resource! Will add to my list of books I recommend to students

[Spencer Lyon @spencer_lyon]

Development of @jupyteach coming along nicely!

Check out https://t.co/iXpWT2Km6P today to create an account and enroll in a free #python fundamentals course.

More courses coming soon!

Drop a reply to help us know what you’d like to learn next to help us prioritize

[Spencer Lyon @spencer_lyon]

I love @duckdb. Just the other day at @ValorumData I had a very simple reverse ETL job. I needed to move processed data from a warehouse (built with @getdbt and @AirbyteHQ ) back upstream to prod. A simple duckdb sql script with a Postgres connection did the trick!

[Spencer Lyon @spencer_lyon]

Just published a new website for @ValorumData

https://t.co/rfRhfBP25q

If you or your company are looking for training or a partner to assist with any #DataScience, #machine_learning, or #AI projects get in touch!

[Valorum Data @ValorumData]

Updated website https://t.co/ZcNzRwxwoo

If your company could use some #DataScience, #AI, #Python , #DataEngineering help or training get in touch!

[Andrej Karpathy @karpathy]

Thinking a lot about centralization and decentralization these few days.