Bob O'Loughlin

Intellectual Property attorney. AI Developer. Patents, metal, and intellectual honesty. Tweets are my own. Tweets/RTs are NOT legal advice or endorsements.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 At least with respect to reasoning over technical content, I have had some decent results by having the agent derive an abstraction of the content and re-play the reasoning with the abstraction. Tends to amplify otherwise subtly poor reasoning turns.

[Bob O'Loughlin @rcoIPLaw]

@gabenotgave @AnthropicAI It’s a reasonable idea but I can’t fathom how it could be enforceable abroad. And that seems significantly more dangerous than just barreling ahead.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Invaluable context. Thanks!

[elvis @omarsar0]

Great technical paper from Harvard and MIT.

It's on role drift in compound LLM systems.

(bookmark it)

End-to-end RL improves the accuracy of a multi-module LLM pipeline without constraining how the modules divide labor internally. Researchers name that failure mode and measure it.

Role drift happens when a module preserves or improves end-task performance while abandoning its assigned role through shortcuts that system-level evaluation cannot see.

Two instances across two pipelines:

- A decomposer meant to split a question into sub-questions for a separate solver instead plants the answer inside them.

- A reader meant to answer from retrieved passages instead falls back on parametric memory.

An interesting finding is that if your hold the decomposer to its role and 86% of the RL improvement disappears.

Role Anchor is the proposed control. It preserves how the role prompt shifts a module's next-token predictions relative to a neutral prompt, using that as a proxy for the role's intended effect during training. Gradient analysis indicates it reduces alignment with the drift direction rather than simply suppressing learning.

Paper: https://t.co/kKvtowNXhP

Learn to build effective AI agents in our academy: https://t.co/1e8RZKs4uX

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 I think that's the opposite of crazy. I'm definitely building a pipeline but wrestling with which models to use - but I know the answer: test a number of them and see what works best for my use cases. Thanks.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 any opinions/recs on best pipelines or models for processing pdf (with text, images, and tables) into text? I've been using a combination of paddle and gemini for image-extraction and summarization, but results are mixed.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Amen to this. Structured output has its place, as does unstructured output. Like anything else, picking the right tool for the job is paramount to optimizing efficiency and accuracy.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 I'm working on an open source solution here, but not as my primary career, so it'll be a couple of months until I'm happy with it. But there are no doubt a whole host of people doing so, as well. Exciting stuff, for sure!

[Bob O'Loughlin @rcoIPLaw]

@vral @tryramp How do I reserve space to give this a go?

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 It really does. I’ve gotten to where I’m pretty much using it for only for complex, nuanced design planning and modifying prompts and context for use by other models. And asking it to keep its output focused.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 I would love to have that if you don’t mind sharing.

[Bob O'Loughlin @rcoIPLaw]

RT @omarsar0: All these AI harnesses should just be open-sourced, period!

This is the way! Good stuff from @SpaceXAI!

Don't trust it or o…

[Bob O'Loughlin @rcoIPLaw]

@thsottiaux @OpenAI I am loving the thoroughness and rationale of ChatGPT 5.6 Sol. I haven’t switched per se, but I use Sol to clean up and fix all of the implementation plans I get from Fable. The quality result is significantly improved with them both on the case.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Variety in application is meaningful too. I recently stumbled onto an interesting path. I ask ChatGPT a lot of questions about music and that lead to discussions about my intellectual approach. Applying that knowledge to my AI’s interactions with me has been transformative.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Even when just running a planning session in a browser chat, I always bounce ideas and draft specs off of multiple models across providers. No one model ever sees all the angles.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Cool. Are you using the front end design skill already available or is this a custom skill?

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 Awesome. I look forward to it. Have been giving serious thought to building my own harness as well - the upside makes it almost a no-brainer I think.

[Bob O'Loughlin @rcoIPLaw]

@omarsar0 The importance of this cannot be overstated. Being able to recognize the subtle features of failure surfaces and success markers from nested steps nested in steps away from the goal enables much more efficient pathfinding as a team with the agents, and generally opens horizons

[Bob O'Loughlin @rcoIPLaw]

@forallcurious