Today I polished the prompt I give to the AI and added some guardrails so users wonโt get their homework done with the help of my app ๐
Also thinking about giving the AI context about the user, so it could give more personalised responses
Finished the implementation of hybrid approach. Settled on gemini-3.1-flash-lite, seems to be capable enough and dirt cheap.
Funny thing is, Iโm actually throttling the stream on the UI to make it look like itโs running at 110 t/s, because the actual response is near instant
Finished the implementation of hybrid approach. Settled on gemini-3.1-flash-lite, seems to be capable enough and dirt cheap.
Funny thing is, Iโm actually throttling the stream on the UI to make it look like itโs running at 110 t/s, because the actual response is near instant
Been implementing a generic AI chatbot for my iOS app. First I wanted to go with a privacy-focused approach with local LLMs, but Apple Intelligence is only available on iPhone 15 or up. Then I tried SpeziLLM - a library that lets you run models on device via XMLโฆ
The trade-off is that users have to download a 2-4GB model, basically killing user retention. So I decided to go with a hybrid approach: check if an Apple Intelligence model is available, if not I switch users to an API-based one, while saving no metadata or analytics whatsoever