Thomas Han

Making LLM & agents local | ex @AWSCloud

[Thomas Han @Thomas__Han]

RustConf Seattle!!!

[Thomas Han @Thomas__Han]

@econoar There are open source LLM models out there that people can host locally. I’m working on a solution to make it super simple with great UX.

Get ready to upgrade your GPU, lads

[Thomas Han @Thomas__Han]

I am actively working on this at the moment. Follow me on my journey to make this a reality!

[Thomas Han @Thomas__Han]

One thing I want to build is a software that runs models efficiently and beautifully on your machine. LM Studio uses Llama.cpp, but I think I can do a much better job on the UX but with vLLM.

[Thomas Han @Thomas__Han]

vLLM is backed by a16z, Sequoia, etc. It’s faster but supports fewer models atm. I am looking very closely at this, and I think its local server style runtime will become more popular.

[Thomas Han @Thomas__Han]

Meanwhile, vLLM can only be run as a localhost server that a Swift app can talk to (through APIs), which is similar to how you run nvidia’s TensorRT-LLM.

[Thomas Han @Thomas__Han]

Right now, Llama.cpp is more mature and easier to work with. For example, on Mac, it can be compiled directly into a Swift app with a bridge header, because it’s written in C++.

[Thomas Han @Thomas__Han]

The battle between open-source inference framework is on! And I think the winner will be vLLM.

[Thomas Han @Thomas__Han]

Just saw RFK Jr.’s autism registry, I’m so fucked lol.

[Thomas Han @Thomas__Han]

What o1 Pro thinks when provided current season's data:

[Thomas Han @Thomas__Han]

I asked OpenAI's 4o to generate my March Madness bracket this year. Which model would actually have the best result on predicting? My bet is on o1 pro w/ Deep Research for obvious reasons...

[Thomas Han @Thomas__Han]

Actually 4o thinks Auburn is not going to make it to the Final Four and St John's is going play Duke.

[Thomas Han @Thomas__Han]

@davefontenot @Daw_g5 Awesome to know. Got the rejection last week, but will definitely have a hockey stick or two by next batch 🦾

[Thomas Han @Thomas__Han]

really cool stuff

[Ibelick @Ibelick]

Introducing prompt-kit

A set of high-quality, customizable components for building AI interfaces.

Built on top of shadcn/ui, starting with PromptInput, a flexible input field for chatting with AI models.

New components coming soon!

[Thomas Han @Thomas__Han]

Getting accepted or not, I'm just vibing and staring at this cover photo they generated for Argonaut AI. Apply everyone!

[Thomas Han @Thomas__Han]

@HF0Residency

Getting accepted or not, I'm just vibing and staring at this cover photo they generated for Argonaut AI. What a piece! Apply, everybody!

[Thomas Han @Thomas__Han]

@FedericoNoemie Been on this exact groove recently. Classical for docs and designs, techno / running music for 7+ hour coding streaks.

[Thomas Han @Thomas__Han]

the world is filled with simple ideas. the simpler the idea, the harder to get to perfection.

say, building the best espresso machine: extremely simple concept, very hard implementation to perfection.

this is the right hill to die on.