Kamil Kaczmarczyk

Software Engineer @swmansion

[Jakub Chmura @_jakubchmura]

Week 2 of Inference knowledge 🧵👇

You often see those terms when talking about AI models, but how does it work, and when is a model memory or compute bound?

[Norbert Klockiewicz @Nklockiewicz]

I've run the same depth model on the NPU and the CPU of a Galaxy S25: 27 ms per frame vs 550 ms :D

If the phone has an NPU, use it. Kudos to @Qualcomm for this chip, the 8 Elite even beats the iPhone 16 Pro's Neural Engine (~90 ms). In the next react-native-executorch release most of our models run on the NPU on Snapdragon devices. Android isn't a second-class citizen anymore.

[Norbert Klockiewicz @Nklockiewicz]

Finally I can share with you a thing I was working on for a past few months.

It's called Velonos and in a few words, it's a privacy gateway that filters the traffic that goes to chatGPT, Claude and Gemini, scans the documents and prompts and hides a sensitive information and restores it in the answer when it comes back.

Most important part is that everything happens on the device.

Here's a link and I will share some more details in thread.

https://t.co/6htN0mXbXR

[Kamil Kaczmarczyk @_kkaczmar_]

ruNNtime is open source:
docs: https://t.co/rHEdLqRikK
github: https://t.co/LGzkDjlSpl

[Kamil Kaczmarczyk @_kkaczmar_]

Searching a 54-min podcast by meaning, in any language, fully local in the browser.

EmbeddingGemma 2 matches your question straight to the audio, no transcription. Ask in Italian or Chinese, it still finds the right moment.

Runs on your GPU with ruNNtime (WebGPU)

[Kamil Kaczmarczyk @_kkaczmar_]

Cool things happening. Check it out!

[Jakub Chmura @_jakubchmura]

ruNNtime v0.2.0 just got a huge performance boost 🚀
We integrated a lot of findings from @huggingface kernels library into ruNNtime, speeding inference up significantly. huge thanks for your help on this @nicodotdev @xenovacom 🤗

[Jakub Chmura @_jakubchmura]

If you want to check out how fast it goes in runntime, it's going to be just an npm install this week, stay tuneed!!!

[Jakub Chmura @_jakubchmura]

LFM3-Audio-2B running fully local on an M4 Pro and it's pretty bonkers.
I talk → the model hears the raw audio (no speech-to-text step) → it picks a tool and writes the call → the Mac acts → it talks back.
can't wait to make this thing go brr on webgpu
soon in runntime 👀

[Jakub Chmura @_jakubchmura]

Week 1 of inference knowledge 🧵 👇

Ever heard of operator fusion?
It's one of the simplest tricks that makes AI models run faster.
But what does it even mean?

[Jakub Chmura @_jakubchmura]

Just ported Phonon by @GradiumAI to runntime 🔊

31M-param open TTS, running fully in the browser on WebGPU at half-precision.

💻 M4 Pro: ~47× real time, first audio in ~20 ms
📱 Phone 13 Pro: 7× real time, first audio in 87 ms

expect some cool things next week 👀 👇

[Kamil Kaczmarczyk @_kkaczmar_]

ain't that fast?

[Jakub Chmura @_jakubchmura]

47 minutes since the release of d1 omni by @liquidai
me and my friend Claude already made it run in the browser using ruNNtime.
this is one of our main goals - to make porting AI models to browsers hassle-free and agent-friendly

[Jakub Chmura @_jakubchmura]

EmbeddingGemma 2 by @googlegemma was released yesterday, and I already got it running in WebGPU via ruNNtime.

The model lets you capture multimodal data as vectors, such as voice, photos, or text. It will soon be released in the runntime zoo package.

links 👇🏻

[Kamil Kaczmarczyk @_kkaczmar_]

ruNNtime on GitHub: https://t.co/LGzkDjmqeT

[Kamil Kaczmarczyk @_kkaczmar_]

Tried porting Laya to ruNNtime. Took about an hour, all from existing core ops, no new kernels. Works in the browser in real time.
We want porting models like this to be that simple.
Live chat moderation, fully on-device. Core package coming soon.

[雨夹雪❄️ @mizorewww]

介绍比Jev快50倍，在你设备上跑的laya-mlx!

只在你的设备上占用最高1G内存

Laya是一个开源的类似于Jev的,基于文本输出概率的分类系统

我将其移植到MLX,并且做了一些性能优化!

视频中就是这个模型在我的本地M3Max上玩贪吃蛇

这个模型能够以每秒决策60次的速度玩贪吃蛇！

https://t.co/31KGUiNunb

[Jakub Chmura @_jakubchmura]

a simple football analysis tool running in the browser via runntime
this demo is using 2 separate vision models, which are running together at over 30fps.
this is the kind of performance you'd want for offline-first coaching apps, for pretty much any sport

[Kamil Kaczmarczyk @_kkaczmar_]

Really cool to be able to work on this especially with this guy @_jakubchmura

[Jakub Chmura @_jakubchmura]

we just shipped ruNNtime, a library for running neural networks in browsers, built on top of TypeGPU

this demo show’s how fast it is. it is speech-to-text + a privacy filter model running together on an M4 Pro. i say my email and it's blacked out before it hits the screen
🧵👇

[Norbert Klockiewicz @Nklockiewicz]

I've built Apple's Spatial Scenes in React Native, running on iPhone and Android :D

Depth Anything V2 + segmentation + MI-GAN inpainting run once (~1s on-device), then a TypeGPU shader does the parallax from the gyroscope at 60fps. The person's background is fully hallucinated, those pixels don't exist in the photo.

stack: @swmansion's react-native-executorch + typegpu

[Jakub Chmura @_jakubchmura]

i was wondering what would it take to reimplement the same model Konrad uses here in our TypeGPU based kernel library and it took around an hour
it still needs a bit polishing, but already works in real time
we're working on porting such models as easy as possible

[Jakub Chmura @_jakubchmura]

Talking to the Moonshine v2 speech-to-text model running on-device in the browser

It works at 350+ tokens/sec, which is so fast that you can transcribe an hour-long podcast in under a minute! Currently working on open-sourcing this project, so expect news soon 👀
More context 👇🏻