How can we make LLMs more modular?
Why can't an agent have an intuitive sense of repo contents?
My research gives an LLM an additional input modality for compressed text
https://t.co/9D7QD0wTC1
Just like images in multimodal LLMs, we treat text input as a differentiable promptable input port. The decoder LLM gets access via tool calls to a promptable encoder which compresses the text and splices its compressed positional output back into the decoder.
I demonstrate gain from targeted compression prompts and show that the architecture works in principle. This was done with relatively small networks as base LFM 350m encoder and decoder.
@FrancoisChauba1 With the transformer we went for an architecture that to some degree optimizes for gradient descent and not falling into vanishing gradients. Does this make long c vision recurrent models more feasible / attractive?
@thsottiaux I believe this happened to me on the 17th -- I was due for a plan downgrade on the 18th -- but still on $200 plan for the 17th reset, which drained within like 2 hours. Haven't been credited -- perhaps fell through the cracks due to my next day plan switch?
VibeCamp Episode 1: Stan James, Gabriel Pickard and Jimmy (an AI agent on the call) build the show itself, live. YouTube channel, GitHub org, logo, theme song.
Watching people vibe conference turns out to be fun.
https://t.co/pDrvwcobGf
@policytensor It changes so often -- a month ago that would have been the absolute best choice, now it's not a bad choice. -- Astra is great if you want to go somewhere wacky -- it's much less conservative in my experience and will ride with wilder ideas, but it's also not careful
I have been thinking about how to make 'personal software' actually non-boring. -- In the age of AI there's no reason to build apps that aren't infused with some intelligence, every step of the way. -- Actually, it is still quite hard to do that right. -- That's why we need means for sharing and collaboratively iterating on our intuitions.
Adding javascript to the browser made web-apps possible -- we don't yet have a good equivalent for AI enabled web pages.
See the video below for my reasoning and a demo of what I've built to move in that direction -- a browser with a batteries included vibe-coding system and agentic environment: https://t.co/DkAImAmaw8 -- a secure enough environment that you can share and personalize 3rd party agentic enabled apps.
@deok_filho Exactly what I was looking for today -- all my gpus are already busy with my other project and I couldn't sleep because of a new idea I had that want to train for modular SLMs can't wait to check it out -- werg-org
OpenAI are shedding load surreptitiously it seems. (a) I'm seeing a 'bug' where even in auto mode it keeps asking for permission. (b) Every once in a while all ongoing sessions have their connection reset and the model stops generating. After that reset, auto mode breaks as well.
@ysegmond I have been thinking about this for a while. A number of different angles to come at this, for the functional programming part I think I have a decent approach. For the LP-inspired part I'm really not sure yet.
@simonw "I want to sail across the Atlantic from East to West, where should I start?" -- less capable models reliably get confused by the fact that the American East Coast is on the Western side of the Atlantic.
I’m rounding off the PL features, to make it usable, but already looking for input on application ideas. Certainly it could be great for email processing, maybe data analysis and also for game logic. If you have specific thoughts in these or other areas I’d be delighted to hear!