Tushar Sheth

co-Founder @SuperfocusAI, x-Obama WH, @umich

[Tushar Sheth @TJSheth]

If what you say is true, the VISION and the LANGUAGE on-device could be dangerous

First came AIs that could see. Like the AI in your self-driving-ish car and your phone’s face ID. But these vision-only models had no ability to use language or think.

Language and “thinking” ability came with the LLM explosion. But, at first, language models couldn’t see.

Then vision and language merged. Your AI has been able to see, think about what it sees, AND use language for at least a couple of years.

These “vision-language models” are typically huge models in the cloud running on massive GPUs.

But what about the many cases when you can’t, or don’t want to, use AI in the cloud?

At @SuperfocusAI, we’re building on-device AI systems with VLMs for these situations.

Our demo video below uses the case of AI on a drone to show what is possible.

The challenge of on-device AI is that the chips used in most devices have only a tiny fraction of the firepower that cloud GPUs have. So you can only use a very small VLM. But you still have to pack all of the intelligence that you need into this small VLM.

Imagine a scenario where the drone has to deliver supplies to a “medic truck”.

The video shows what a vision-only model (top) would do in this scenario versus a VLM (bottom).

Both models are running together live on a chip that is commonly used in drones. We can also do this on even cheaper, lower firepower chips.

The vision-only model does a good job of identifying trucks. But it can’t think about what it is seeing, so it can’t distinguish between the trucks without further training. It is impractical to train in advance for every nuance that might arise in the moment.

In comparison, the VLM just needs to be told that the medic truck is the one with the red cross on it. With that information alone, the VLM reasons about what it is seeing and correctly picks out the medic truck from the group of similar looking trucks.

We’re using vision and language together in on-device AI systems to add contextual intelligence like this in all sorts of use cases. More examples of what we’ve built to come in future posts…

En garde!

@hsu_steve @reindsummit

[steve hsu @hsu_steve]

UAV autonomous action via edge VL model

This video shows a Vision-Language model Superfocus has running on inexpensive (~$50 chipset) hardware, interfaced with the control system on a drone.

The text at the top is the LLM's interpretation of what is happening in the scene, and the green text describes its actions. It can differentiate between different vehicle types, different uniforms, orient itself to capture face-recognition data, track a vehicle or person, etc.

None of this requires access to the cloud - the intelligence runs locally on the drone itself. The drone does not require a remote operator or controller. It can't be jammed.

[Tushar Sheth @TJSheth]

I don’t understand Abdul’s thinking here.

Maybe, just maybe, the country enabling Gaza genocide and that made an unlawful and immoral attack on Iran might be selling a misleading righteous war narrative in Ukraine?

[Jolani Peterson 🇺🇦🟣 @andymikhailovic]

Abdul El-Sayed today in the Detroit Free Press on aid: “So, when it comes to Ukraine, I think we are involved in a righteous struggle to protect a sovereign nation from being invaded by their neighbor. That's completely consistent with international law.” https://t.co/zs8at9bwOs

[steve hsu @hsu_steve]

This is a small Vision-Language model running on an inexpensive phone chipset. Executes reasoning task while controlling UAV (drone) in sim world.

The task is to find the human doctor near the medic truck. If desired, the drone could be instructed to hit the doctor or the truck.

@SuperfocusAI

[Tushar Sheth @TJSheth]

Did it take a foreigner to make the team more American?

It’s hard to escape the feeling that lots of things in the country suffer from the same problems that plagued the USMNT.

[Bruce McGuire @dunord]

Fascinating reading. One of the most honest things to ever come out of the USMNT system ever.
By @JeffreyCarlisle

https://t.co/MDfPKI6Typ

[steve hsu @hsu_steve]

Vision-Language models can run on *very* inexpensive chips now. Look for fully autonomous but inexpensive drone swarms in the near future!

[Tushar Sheth @TJSheth]

Space, bitches!

[Alumni Association of the University of Michigan @michiganalumni]

.@UMengineering alum Andre Douglas has been named to NASA's Artemis III mission and will fly to space in 2027. 🚀 https://t.co/lGwKZkmT74

[Tushar Sheth @TJSheth]

My point lumps together a handful of chips from China & Taiwan. Their prices fall into a range, and the low end of that range is ~$25 (and possibly cheaper).

The lower firepower MTK options are near that $25 end of the spectrum.

You are correct to point out that the better MTK options, the ones that look more like their QC & NVIDIA counterparts, are a little further up the cost range.

They are still far cheaper though than the QC & NVIDIA options.

[Tushar Sheth @TJSheth]

It ain’t no fun if the Americans can’t make none.

There is not a single low-cost American AI chip for running genAI models on-device.

That’s one of the eye-opening things we’ve learned at Superfocus dot ai investigating on-device AI, and a point I made in my @EmbVisionSummit talk last week in the Valley.

These are the chips in phones, tablets, smart glasses, wearables, drones, robots etc.

The cheapest American on-device AI chips cost $100+ at volume, likely more. The lowest NVIDIA GPU chip and the top Qualcomm smartphone chips.

A $100 chip means a ~$500 product. At least.

Chinese and Taiwanese on-device AI chips that can do the exact same thing cost $25, maybe less, at volume.

A $25 chip means a ~$125 product.

There are lots of use cases where you can’t, or wouldn’t want to, use AI in the cloud.

For those situations, why isn’t there a low-cost American on-device AI chip?

Does it matter? The cheaper options from China and Taiwan are good (but not perfect) chips from solid companies.

@edgeaivision @phil_lapsley @JeffBierBDTI @Rakshit_Ag @vghadiok @daveselinger @hsu_steve #EVS26

[steve hsu @hsu_steve]

This is the situation for on-device intelligence. Can be done inexpensively, but the chips are all from Taiwan or China.

Think massive drone swarms...

[Tushar Sheth @TJSheth]

Wow - clash of the (debate) titans

[Quincy Institute @QuincyInst]

🚨Next Wednesday: John Mearsheimer and Stephen Walt debate Mike Pompeo and Victoria Nuland on US foreign wars.

The motion: "Be it resolved, don't go hunting monsters"

This one matters.

Livestream available. Link below.

[Tushar Sheth @TJSheth]

You are what you do, not what you say.

In the same briefing, right before this, Rubio was lying through his teeth about the state of the disastrous war with Iran that he and his Trumpista buddies are waging.

His actions are the opposite of the spirit of the words he is using.

[Tushar Sheth @TJSheth]

@policytensor Generally agree, but should not going along with the sanctions on Russia count for something? And maybe the resistance to be on America’s “team” re China? Perhaps those are too subtle and obvious to qualify as bold?

[Tushar Sheth @TJSheth]

@farhaad_riyaz This is something we are exploring!

[Tushar Sheth @TJSheth]

Two lesser-noted points in the Jan '25 DeepSeek paper caught our attn at Superfocus dot ai.

1) They produced AI models small enough to run directly on consumer electronics devices but with surprisingly strong reasoning ability. 2) They did it with relatively fewer data samples.

We wanted to know if these smarter “on-device” size models are actually good enough to be useful? And how cheap of a chip could we run these models on?

Why does this matter? There are lots of situations where you can’t, or wouldn’t want to, use AI in the cloud via the internet.

Personal & family privacy (ex: your home). Child safety. Security. Data/IP privacy (factories). No or unreliable signal (commercial/defense drones). Low latency (robot/vehicle safety). No or outdated physical/cloud infra (industrial, warehouses).

Plus, you don’t have to pay compute costs for on-device AI!

We’re sussing out many of these use cases, but because of our shared interest in education, my co-Founder @hsu_steve & I immediately thought about kids & learning.

Below is a video of my daughter demo’ing our prototype AI reading buddy. We’ve rigged our little AI puck up to a pair of smart glasses from our homies at @MentraGlass . (h/t @caydengineer )

This may not be a big deal for huge models in the cloud running on GPUs. What’s significant is that we’re doing it with small models and completely on-device. No internet connection.

The AI sees what my daughter sees. It listens to her and follows along as she reads. And helps her when she needs it by speaking to her. All from small AI models that we finetuned, engineered and got running fast and accurately on a chip that costs less than $25.

It’s not quite a reading tutor yet, but perhaps you can see how it could become one? Or how on-device AI like this could solve other problems?

Would love to know what you think, good, bad, or other!

[Josh Wolfe @wolfejosh]

Arooj Aftab incredible Pakistani heir to Sade

Was with director/video producer of this last night…sick.

“Raat Ki Rani”

[Tushar Sheth @TJSheth]

It's also not fair to the many staffers who don't come from wealth, which disproportionally is the case for black and Latino staffers.

Most of the people who spend their entire careers in non-prof & govt and who create and enforce this weird moral standard against those that go into private sector have rich parents and family money. (Which there's nothing wrong with either!)

[Tushar Sheth @TJSheth]

A legend. My parents Sunday morning soundtrack in our house growing up always had a brimful of Asha in it.

[The Times Of India @timesofindia]

Asha Bhosle has passed away at the age of 92. The legendary singer had been admitted to Breach Candy #Hospital in #Mumbai.

#AshaBhosle’s last rites will be held with full state honours on Monday. The legendary #singer passed away on Sunday, at the age of 92, in Mumbai Hospital.