Leo

Just another person on the internet

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Leo @leowei31]

@budapp Can I get some credits to replace myself at work

[sky @notsky]

the time picker on iphones alarm app isn’t actually circular it’s just a really long list

[Leo @leowei31]

Cost to benefit ratio: I bet hard prompting is probably way cheaper than training soft prompts even if you are optimizing your hard prompts manually. Soft prompting requires hyperparameter selection, training the soft prompt meaning you have to host the model yourself.

[Leo @leowei31]

I wondered about this a lot this past week on why soft prompts are not more popular, why not everybody is using them. I was able to deduce a few reasons:

[Leo @leowei31]

Not very future proof if you want to use SOTA models: since soft prompts are model dependent, and the pace of models that are coming out is so fast, optimizing soft prompts are model specific, so this gets hard because nowadays a new models comes out like every week.

[Leo @leowei31]

Black box nature: soft prompts are just trainable embeddings treated like a black box, hard to see what they are actually doing. Meanwhile compared to hard prompting, you can see exactly what the prompt is and what the system user is asking the LLM to do.

[Leo @leowei31]

Lack of open sourced models: training soft prompts requires gradients updates, with all these SOTA models behind APIs, it's hard to train soft prompts. And on top of that, training soft prompts for these extremely large models is going to take forever.

[Leo @leowei31]

Explored soft prompts this past week, trained on the Twitter sentiment eval dataset, included my own research in there as well.
https://t.co/7UnQXedIk8

[Leo @leowei31]

Actually very true, I interviewed with YC a while ago and this is pretty much exactly what they reiterated to us.

[Leo @leowei31]

Been playing around with soft prompts past couple days. Curious why this isn’t more popular. I understand that smaller models don’t have as much latent knowledge, but there’s gotta be some application for it right? Maybe bigger models for different modalities? What am I missing?

[Leo @leowei31]

For this week, I will be working on soft prompts. I aim to adapt smaller models to perform way better on new tasks.

[Leo @leowei31]

LLMs have become way more expensive to train, with the increase in return on investment being pretty minimal. Smaller multimodal systems are the future, at least application-wise. I don't need my GPT to be an expert at everything

[Leo @leowei31]

Also I think federated learning could be interesting to try out here since training across device is important when it comes to real world applications.

[Leo @leowei31]

Built a simple VAD system this past week, trained on synthetic noisy mixes, lightweight DS-CNN. Aimed at being efficient and noise resilient, not SOTA but I learn a lot about VAD building it. https://t.co/NgzNHVQ0RK

[Leo @leowei31]

human speech vs background noise, natural pauses between sentences, different people have different rhymes, latency is also a huge problem because ideally a VAD should be under 15-70ms or else the voice AI doesn't feel natural

[Leo @leowei31]

This week, I am building a Voice Activity Detection(VAD) system. I recently learned how interesting this problem is after speaking with some folks at Boson AI (they work on voice ai), and whether or not to continue listening to the user is not trivial

[Leo @leowei31]

literally me at all my hackathons