Project #2: LLM Visualization
So I created a web-page to visualize a small LLM, of the sort that's behind ChatGPT. Rendered in 3D, it shows all the steps to run a single token inference. (link in bio)
So, @vercel reverted all edge rendering back to Node.js 😬
Wanted to correct the record here as it's something I've advocated for in the past, and share what I've learned since then.
Also, the "edge" naming has been a bit confusing, so let's clear that up here as well.
What is edge, anyway?
We have too many products with Edge 😅
First off, Vercel has an Edge Network. A request comes in for your site. The edge region can go grab some HTML or JS for your site and quickly return it. Great.
Then, sometimes, you have routing rules. When that request comes in, redirect it, proxy it somewhere else, or add headers before continuing (e.g. Middleware). You don't need to go all the way back to the origin. This is also great.
Sometimes those routing or configuration rules need to be read from a file. We have "Edge Config" which is like reading from a JSON file in every region. Again, still good.
Now, what about rendering my application with edge compute?
Not as great.
But Lee, you said...
Yeah, I gotta admit, this one fooled me. It sounded great at the start – running close to the visitor makes sense.
If I constrain my site to use this limited runtime, and make the tradeoff for access to the Node.js package ecosystem, I can (hypothetically) get better performance.
This hasn't worked for two reasons.
First, and most obvious, is that your compute needs to be close to your database. Most data is not globally replicated. So running compute in many regions, which all connect to a us-east database, made no sense.
I believe many folks understood this. But then, maybe this limited runtime was better even if I only run it close to my database?
Well, that was wrong too. We tested it extensively with @v0. Using the Node.js runtime with 1 vCPU (our "Standard" performance) option for Vercel Functions had faster startups 🤯
Are there other reasons for edge rendering?
If Node.js functions can be faster, especially with Node v20, what about cost?
I expected an edge compute pricing model, which is based on only paying for when your compute runs (and never I/O), would always be cheaper.
Turns out that was also wrong. It can be cheaper sometimes! But not for all workloads, so some nuance applied here.
Same story for security. I wanted edge compute to work, but customers kept telling me they wanted a private connection to other services (i.e. Vercel Secure Compute), or had concerns (or no interest) in doing global data replication (e.g. data sovereignty).
So now what?
It's a good lesson for me to be careful with naming / nuance going forward.
Also, networks have gotten really fast. We saw with @v0 that it was faster to do SSR + streaming with Node.js than edge rendering.
Now we're running @v0 with Partial Prerendering to see if this is better or worse (it's still experimental).
The summary of PPR is: user makes a request, the edge region starts streaming back the HTML for the fast initial visual, and then your compute still runs in us-east or wherever close to your data.
So far this has been about 50% faster time to first byte than SSR + streaming. Pretty amazing.
I'll be sharing more on Vercel Functions improvements soon, as well 🦀
🧑🚀⭐ Calling all aspiring astronauts—Astro 3.0 touches down next week. We're giving away 10 of these official Astro hats to celebrate!
Retweet for your chance to win.
Trying something new... I would love your support to dedicate more time and effort to my side projects!
Please consider sponsoring me on GitHub
https://t.co/CreVK2vLsU
It's @scrimba course launch day 🚀 My new React Router 6 course is now available 100% FOR FREE on Scrimba!
Because we're excited, we're giving away a free year of Scrimba Pro to a ☘️ lucky ☘️ someone who retweets THIS initial tweet. Go ahead, I'll wait ⏱️🙂
About the course 👇
On vous présente Le DayBrief 🎉🤩
Votre rendez-vous actus web à retrouver tous les mardis !
Vous y retrouverez :
- Un résumé des infos web de la semaine
- Des outils et ressources
- Des tips et astuces
Hésite pas à claquer ton like et commenter 😉
#ledaybrief N°1
Thread 👇
📣 Changement important 📣
A partir de demain, nos émissions matinales sur Twitch commenceront à 08h30 et non plus 08h00 afin qu'un maximum d'entre vous puisse suivre les actus en direct. Merci à toutes celles et ceux qui nous ont fait remonté l'info 🙏🙏
@BastiUi en interview live sur @TwitchFR pour débattre de l'utilisation de l'#IntelligenceArtificielle dans le #design 🤖🤖
RDV le vendredi 16 déc. à 13h sur notre chaîne Twitch pour une interview live animée par @antho_parle_web
🫶 Likes et RT appréciés
🔥 @BastiUi EN INTERVIEW 🔥
RDV le vendredi 16 déc. à 13h sur notre chaîne Twitch pour une interview live animée par @antho_parle_web
Basti reviendra sur son parcours de designer/influenceur et nous débattrons sur l'utilisation des IA dans le #design
🫶 Likes et RT appréciés
❄️ Retour du biathlon demain pour les courses de pré-saison ❄️
⏰ 9h15 sur @lachainelequipe !
⛷️ Pour l'occasion, @AliciaDauby3 vous fait gagner la toute nouvelle tenue de l'Équipe de France de biathlon !
@FedFranceSki
RT + follow
#EDG
On lance Dayclic, le nouveau média des métiers du #web ! On détaille tout dans cette vidéo👇
📌Liens vers nos plateformes en commentaire
Merci à @BastiUi, grand gourou du DASAGN, pour l'accompagnement au début de ce projet 🫶
#webdev#Webdesign#ecommerce#seo#jamstack
🎟TICKET GIVEAWAY 🎟 https://t.co/VhIHJANDaz
6 tickets for @cityjsconf Athens 🇬🇷 are ready to be sent out to you! 🤩
What do you need to do to get one?
✅ follow @storyblok
🔄 retweet this tweet
Good luck everyone!🍀