Products, startups, history, philosophy. From first principles. Building agentic AI at AWS, ex-CTO Hitwicket. At home in contradiction and the unknown.
This is good, but my honest take on any such forced mechanisms is that it comes with tradeoffs on happy path quality. Things added to global or repo level prompt keep adding up and sway model towards those rather than the core work. Hooks before/after git commit doesn't trip quality but there also if we keep piling-up cases like this it makes the flow a but cumbersome. But for now a git hook seems the way to go.
I find myself asking agents "what terms did you pick?" way more than I'd like. 5.5 series has come a long way on human readability (RIP em-dash), but agents quietly deciding what things are called is still very much there. Maybe you mentioned a word once in passing, maybe never, and it just becomes a thing.
And these compound. Next change reads that word and uses it, the one after builds on it, and a month or two later it's all over the repo. Matters more as we move more and more towards solely relying on agents (at least for pet projects) to read the code, tell us what's going on & execute. But they tell you in those same drifting words, and the project slowly gets away from you.
@AzFlin Ha, define "fine".
Auto mode on can reach the wallets or anything your user account can - the repo limit is a convention. On my Mac it just read a Documents file w/ auto-approve.
Thankfully I found TCC - an actual wall at macOS level - https://t.co/358wa7UNd0
Claude code told me, in an earlier session, that it couldn't reach outside the repo - not Desktop, not Documents, not Downloads. Today, when I asked it to compare a file sitting in my Documents, it just went and read it (w/ auto-approve).
Turns out the fix is quite simple, and claude itself suggested it - macOS TCC. System Settings -> Privacy and Security -> Files and Folders -> Turn off Desktop, Documents, and Downloads for your terminal and your IDE. Two minutes, and the agents can't touch those folders even if they try (also check Full Disk Access, which overrides Files & Folders).
Many of us run these agents on a personal laptop, not in containers or cloud dev boxes. With auto-approve on, it can reach anything your user account can. I'm a security freak for things like bank statements and tax files, and want access to them not left to an inference decision. The July incident where an OpenAI eval agent escaped its sandbox and broke into Hugging Face was a different failure, but similar blind spot: trusting the wall. There, agent jumped a real wall. On laptops the working directory restriction is just a convention (claude code's words), and auto-approve means nobody's watching.
FWIW I too hated those persistent permission prompts that after a while you just start approving blindly - but I feel guardrails like TCC should be more widely known.
Things have improved a lot, but one thing still bothering me is naming. Agent coins terms I never picked, or said once in passing. And they compound - next change reads the word & builds on it. Month later the repo's in words used without context and don' t make much sense. https://t.co/EjHkxlN7iG
I find myself asking agents "what terms did you pick?" way more than I'd like. 5.5 series has come a long way on human readability (RIP em-dash), but agents quietly deciding what things are called is still very much there. Maybe you mentioned a word once in passing, maybe never, and it just becomes a thing.
And these compound. Next change reads that word and uses it, the one after builds on it, and a month or two later it's all over the repo. Matters more as we move more and more towards solely relying on agents (at least for pet projects) to read the code, tell us what's going on & execute. But they tell you in those same drifting words, and the project slowly gets away from you.
@redp314 Same duck, different loop. My daughter drew a butterfly, Opus 5.5 designed the rewards, and the duck traced it with its feet. 24 training runs.
Moving off the eye candy for a second - opus 5.5 surprised me today in a much more ordinary way. Asked https://t.co/m8uUcphTcl to translate my kidโs bespoke piano-class notation so I could import it in another app. It built a working player w/ speed, left/right hand modes, and highlighting current note. I honestly thought the artifact was a pdf until I hit play.
https://t.co/ksRndD4WR8
Opus 5.5 had me in shock today - not w/ a demo, w/ my daughter's piano homework. It built a full song player when I only asked for help understanding a bespoke piano notation, in https://t.co/m8uUcphTcl (not claude code).
Daughter gets these specialized music notations from her class, and was confused on one. I asked claude if it can understand and translate into standard notation that I can import on some apps.
After 1 clarifying question, it generated a beautiful player with controls for start/stop, speed change, and hand selection (left only, right only or both), along with highlighting corresponding notes being played.
I was honestly pissed at first - how'd I import this rendered pdf (which I thought it was). Didn't even think the button was clickable, much less that it's a full-blown player. Then clicked it, and it just played.
Opus 5.5 had me in shock today - not w/ a demo, w/ my daughter's piano homework. It built a full song player when I only asked for help understanding a bespoke piano notation, in https://t.co/m8uUcphTcl (not claude code).
Daughter gets these specialized music notations from her class, and was confused on one. I asked claude if it can understand and translate into standard notation that I can import on some apps.
After 1 clarifying question, it generated a beautiful player with controls for start/stop, speed change, and hand selection (left only, right only or both), along with highlighting corresponding notes being played.
I was honestly pissed at first - how'd I import this rendered pdf (which I thought it was). Didn't even think the button was clickable, much less that it's a full-blown player. Then clicked it, and it just played.
This matches what happened to me today, and I use these models a lot. Asked https://t.co/m8uUcphTcl to translate my kidโs bespoke piano-class notation so I could import it in another app. I didnโt think to ask for a practice player. It built one anyway. The gap isnโt the model. Itโs that I still asked for the step I knew it could do yesterday.
https://t.co/ksRndD4WR8
Opus 5.5 had me in shock today - not w/ a demo, w/ my daughter's piano homework. It built a full song player when I only asked for help understanding a bespoke piano notation, in https://t.co/m8uUcphTcl (not claude code).
Daughter gets these specialized music notations from her class, and was confused on one. I asked claude if it can understand and translate into standard notation that I can import on some apps.
After 1 clarifying question, it generated a beautiful player with controls for start/stop, speed change, and hand selection (left only, right only or both), along with highlighting corresponding notes being played.
I was honestly pissed at first - how'd I import this rendered pdf (which I thought it was). Didn't even think the button was clickable, much less that it's a full-blown player. Then clicked it, and it just played.
This is closer to the version I needed this morning. Asked claude to translate my daughterโs bespoke piano-class notation so I could import it in another app. Surprisingly and amazingly, it built a working player w/ speed + left/right hand modes instead. I thought the artifact was a pdf until I hit play.
https://t.co/ksRndD4WR8
Opus 5.5 had me in shock today - not w/ a demo, w/ my daughter's piano homework. It built a full song player when I only asked for help understanding a bespoke piano notation, in https://t.co/m8uUcphTcl (not claude code).
Daughter gets these specialized music notations from her class, and was confused on one. I asked claude if it can understand and translate into standard notation that I can import on some apps.
After 1 clarifying question, it generated a beautiful player with controls for start/stop, speed change, and hand selection (left only, right only or both), along with highlighting corresponding notes being played.
I was honestly pissed at first - how'd I import this rendered pdf (which I thought it was). Didn't even think the button was clickable, much less that it's a full-blown player. Then clicked it, and it just played.
With daughter's piano homework. Asked it to translate my kid's bespoke piano-class notation so i could import it in another app. Surprisingly and amazingly, it built a working player w/ speed + left/right hand modes. I thought the artifact was a pdf until I hit play. ๐คจ
https://t.co/ksRndD4WR8
Opus 5.5 had me in shock today - not w/ a demo, w/ my daughter's piano homework. It built a full song player when I only asked for help understanding a bespoke piano notation, in https://t.co/m8uUcphTcl (not claude code).
Daughter gets these specialized music notations from her class, and was confused on one. I asked claude if it can understand and translate into standard notation that I can import on some apps.
After 1 clarifying question, it generated a beautiful player with controls for start/stop, speed change, and hand selection (left only, right only or both), along with highlighting corresponding notes being played.
I was honestly pissed at first - how'd I import this rendered pdf (which I thought it was). Didn't even think the button was clickable, much less that it's a full-blown player. Then clicked it, and it just played.
@snr_boost No JS. Physics is MuJoCo (Python), running Microduck's existing walking policy. Opus tuned the steering with rewards that compare the duck's footprints to the hand drawing. The recorded motion was then replayed in Blender (EEVEE) to render, and edited with ffmpeg.
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@addyosmani Test on my side was one prompt: design the rewards so Microduck traces my daughterโs drawing, run the training loop, record it in Blender. 24 automated iterations. Opus 5 took 96 on a simpler apple.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@claudeai@kevin_t_ngo Robot duck + her drawing. 5.5 designed the rewards, ran 24 training iterations, and recorded it in Blender. Opus 5 took 96 on a simpler apple.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@alexalbert__ Same Blender jump, different loop. One prompt: design rewards so Microduck traces my daughterโs drawing, train it, record the run. 24 automated iterations vs 96 on Opus 5.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@MengTo We pointed it at a robot instead of a scene. One prompt: trace her drawing, train the duck, record it in Blender. 24 iterations. Opus 5 took 96 on a simpler apple.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@doodlestein That agency showed up as: design the reward, run the loop, record the video. I didnโt start a second session. 24 iterations vs 96 on Opus 5 for a simpler shape.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@haider1 The fewer-iterations thing showed up cleanly here. One prompt. 24 automated training iterations vs 96 on Opus 5. It also handled the Blender capture without another pass from me.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.
@mattpocockuk Same โone promptโ shape, but the work after the prompt was a training loop. Opus 5.5 designed the rewards, ran 24 automated iterations, and also stood up the Blender recording. Opus 5 needed 96 for a simpler apple.
https://t.co/6NmJ9YakU9
Building games used to be why kids learned to code.
Teaching a robot toy might be how they learn AI.
Explored Microduck with my daughter, and simulated the duck to draw/trace with its feet.
She drew the shapes. Opus stood up the sim, helped define rewards, and fielded her infinite questions. The duck already has a trained walking policy. We gave it the outlines, and used rewards to tune its steering over 96 simulated attempts.
Next time the activity is her choosing the reward and watching the curve move.