Oliver

ML Engineer, Swiss AI Platform

[Andrej Karpathy @karpathy]

It is hard to communicate how much programming has changed due to AI in the last 2 months: not gradually and over time in the "progress as usual" way, but specifically this last December. There are a number of asterisks but imo coding agents basically didn’t work before December and basically work since - the models have significantly higher quality, long-term coherence and tenacity and they can power through large and long tasks, well past enough that it is extremely disruptive to the default programming workflow.

Just to give an example, over the weekend I was building a local video analysis dashboard for the cameras of my home so I wrote: “Here is the local IP and username/password of my DGX Spark. Log in, set up ssh keys, set up vLLM, download and bench Qwen3-VL, set up a server endpoint to inference videos, a basic web ui dashboard, test everything, set it up with systemd, record memory notes for yourself and write up a markdown report for me”. The agent went off for ~30 minutes, ran into multiple issues, researched solutions online, resolved them one by one, wrote the code, tested it, debugged it, set up the services, and came back with the report and it was just done. I didn’t touch anything. All of this could easily have been a weekend project just 3 months ago but today it’s something you kick off and forget about for 30 minutes.

As a result, programming is becoming unrecognizable. You’re not typing computer code into an editor like the way things were since computers were invented, that era is over. You're spinning up AI agents, giving them tasks *in English* and managing and reviewing their work in parallel. The biggest prize is in figuring out how you can keep ascending the layers of abstraction to set up long-running orchestrator Claws with all of the right tools, memory and instructions that productively manage multiple parallel Code instances for you. The leverage achievable via top tier "agentic engineering" feels very high right now.

It’s not perfect, it needs high-level direction, judgement, taste, oversight, iteration and hints and ideas. It works a lot better in some scenarios than others (e.g. especially for tasks that are well-specified and where you can verify/test functionality). The key is to build intuition to decompose the task just right to hand off the parts that work and help out around the edges. But imo, this is nowhere near "business as usual" time in software.

[chansung @algo_diver]

@NeurIPSConf 2024 reimagined with AI !!
- summaries for instant insights 🧠
- easy-to-understand audio podcasts 🎙️
- quick links to NeurIPS Proc., @huggingface & more 🌐
- Full papers, topic & affiliation filters 📂

All your research needs, in one hub. Dive in now! 👇
https://t.co/cunILFdlMn

Currently, Oral and Spotlight papers (~400) are included in the blog, and other poster papers (~3600) will be added by tomorrow.

All blog contents are generated by the following technologies:
- @GoogleDeepMind #Gemini models for generating summaries, in-depth insights, better captions for Fig. and tables, and podcast script.
- @googlecloud Vertex AI's TTS service for voice synthesis which is used in @notebooklm
- @OpenDataLab_AI's PDF Extract Kit for extracting figures and tables from papers

If you are curious about the pipeline, and if you think this project is helpful for tech & non-tech people, please consider visiting the GitHub repository and give a ⭐️
: https://t.co/TfmFlVIcit

[Oliver @iamOliT]

Great demo! Really impressed by how full o1 solved the prime numbers problem and how it suggested a reasonable solution for the agentic workflow using only some handdrawn sketches.
Exciting times!

[elvis @omarsar0]

ChatGPT o1 is the most "intelligent" AI model and it's not even close!

Full o1 generates thinking steps ~50 faster than preview.

It's more accurate, reliable, and got better on harder tasks that require advanced reasoning and knowledge.

I ran a few tests on it already. Here are my observations:

Full video with examples & explanations: https://t.co/oVcBdyZDxB

Strengths - impressive at math, code, and knowledge-intensive tasks.

Weakness - it only failed on a cross-word puzzle but I think it might be solvable when a web search becomes available. In the end, while very efficient with complex knowledge use, it's still constrained by data it's trained on.

Speed - the thinking steps are generated a lot faster! Not a fair comparison with the open alternatives but I think this improves the overall user experience.

"Knowledgeable and highly intelligent" - as mentioned in the demo by OpenAI researchers, o1 is great at dealing with ambiguity and filling in knowledge gaps. I was impressed by how it implemented an agentic solution (with lots of details) from a basic diagram of architecture (with minimal details). Check out the sample video.

Better Task Coverage - Due to the speed and the ability to make sense of instructions and intent (i.e., know when to response fast and when to "think" deeply) much better, it feels like it might be more useful for a broader range of tasks.

Image understanding - the image understanding capability is mysterious (often leads to faster responses but no thinking) but impressive.

More experiments and notes soon. Stay tuned!

[elvis @omarsar0]

Structured output is all you need!

Structured outputs can help to improve the performance and reliability of your LLM application.

OpenAI just announced that they are now supporting structured outputs in the API.

You can use it with function calling and defining your own JSON schema.

Here is the FULL video if you are interested: https://t.co/M05DshftF0

The idea is to constrain the outputs based on defined schemas. There are many tools that can perform this for you such as instructor, outlines, and guidance.

Very excited to use this in many of our use cases. I am working on a part 2 tutorial to dive deep into this feature and apply it to other common and creative use cases.

[Oliver @iamOliT]

@jpaulpoliquit @todoist @NotionHQ ✅