Juliette Estelle | oooo

Optimization Obsessed. Solar powered. Run, yoga, meditate. Proud wife to Jason.

[John Collison @collision]

Marc Andreessen on Cheeky Pint. Please do not attempt at 2x speed.

[Ryan Petersen @typesfast]

In the 3 weeks since the tariffs took effect, ocean container bookings from China to the United States are down over 60% industry wide. 🧵

[Juliette Estelle | oooo @JulietteSurette]

Sith order youngest recruit?

[Juliette Estelle | oooo @JulietteSurette]

Deflationary trend

[Andrew Ng @AndrewYNg]

After a recent price reduction by OpenAI, GPT-4o tokens now cost $4 per million tokens (using a blended rate that assumes 80% input and 20% output tokens). GPT-4 cost $36 per million tokens at its initial release in March 2023. This price reduction over 17 months corresponds to about a 79% drop in price per year. (4/36 = (1 - p)^{17/12})

As you can see, token prices are falling rapidly! One force that’s driving prices down is the release of open weights models such as Llama 3.1. If API providers, including startups Anyscale, Fireworks, Together AI, and some large cloud companies, do not have to worry about recouping the cost of developing a model, they can compete directly on price and a few other factors such as speed.

Further, hardware innovations by companies such as Groq (a leading player in fast token generation), Samba Nova (which serves Llama 3.1 405B tokens at an impressive 114 tokens per second), and wafer-scale computation startup Cerebras (which just announced a new offering this week), as well as the semiconductor giants NVIDIA, AMD, Intel, and Qualcomm, will drive further price cuts.

When building applications, I find it useful to design to where the technology is going rather than only where it has been. Based on the technology roadmaps of multiple software and hardware companies — which include improved semiconductors, smaller models, and algorithmic innovation in inference architectures — I’m confident that token prices will continue to fall rapidly.

This means that even if you build an agentic workload that isn’t entirely economical, falling token prices might make it economical at some point. As I wrote previously, being able to process many tokens is particularly important for agentic workloads, which must call a model many times before generating a result. Further, even agentic workloads are already quite affordable for many applications. Let's say you build an application to assist a human worker, and it uses 100 tokens per second continuously: At $4/million tokens, you'd be spending only $1.44/hour – which is significantly lower than the minimum wage in the U.S. and many other countries.

So how can AI companies prepare?
- First, I continue to hear from teams that are surprised to find out how cheap LLM usage is when they actually work through cost calculations. For many applications, it isn’t worth too much effort to optimize the cost. So first and foremost, I advise teams to focus on building a useful application rather than on optimizing LLM costs.
- Second, even if an application is marginally too expensive to run today, it may be worth deploying in anticipation of lower prices.
- Finally, as new models get released, it might be worthwhile to periodically examine an application to decide whether to switch to a new model either from the same provider (such as switching from GPT-4 to the latest GPT-4o-2024-08-06) or a different provider, to take advantage of falling prices and/or increased capabilities.

Because multiple providers now host Llama 3.1 and other open-weight models, if you use one of these models, it might be possible to switch between providers without too much testing (though implementation details — specifically quantization, does mean that different offerings of the model do differ in performance). When switching between models, unfortunately, a major barrier is still the difficulty of implementing evals, so carrying out regression testing to make sure your application will still perform after you swap in a new model can be challenging. However, as the science of carrying out evals improves, I’m optimistic that this will become easier.

[Original text (with links): https://t.co/txk7q32EXn ]

[Juliette Estelle | oooo @JulietteSurette]

“We are all attempting to give away a fortune that was enabled by systems in need of change.” She then donates $2.74B to 286 orgs with no strings. @mackenziescott

[Juliette Estelle | oooo @JulietteSurette]

Giving Back. Force multiplier <3

[The New York Times @nytimes]

MacKenzie Scott, one of the richest women in the world, announced on Tuesday a third multibillion-dollar round of donations in less than a year. Scott who was married to Jeff Bezos, Amazon's founder, gave away nearly $6 billion in 2020. https://t.co/NVJDb2PnxH

[Juliette Estelle | oooo @JulietteSurette]

Grateful for my husband. I admire his determination, discipline and drive. I am thankful for every moment we share together. He is my hero. It is the smallest simplest moments that count most.

[Juliette Estelle | oooo @JulietteSurette]

I’m human.

[Juliette Estelle | oooo @JulietteSurette]

https://t.co/DqYkRvvK4X

[Juliette Estelle | oooo @JulietteSurette]

New Oceana Series https://t.co/W0huslWeCj

[Juliette Estelle | oooo @JulietteSurette]

@wiseconnector Gratitude. Love. Creation.

[CBC Gem @cbcgem]

New immigrants and Indigenous peoples meeting for the first time and discovering they share many of the same traditional dance moves is one of our favourites moments of this doc.

(📺 : Behind the Bhangra Boys | @cbcdocs)

[Juliette Estelle | oooo @JulietteSurette]

QRCodes as proof of vaccination.
https://t.co/DEzhTipj2h

[Juliette Estelle | oooo @JulietteSurette]

🚴‍♀️🚴🚴‍♂️

[United Nations @UN]

🚴‍♀️🚴‍♂️ Thursday is #WorldBicycleDay!

Learn more about the benefits of bikes, which are used around the world as simple, affordable & efficient modes of sustainable transport. https://t.co/4QSs2moNuR #ClimateAction

[CBC News Alerts @CBCAlerts]

UPDATE: The National Advisory Committee on Immunization has updated its guidance to recommend that people who received one dose of AstraZeneca receive either the same vaccine or an mRNA vaccine for their 2nd dose.

[Juliette Estelle | oooo @JulietteSurette]

I hope we see more of this. https://t.co/6XCEZDHOlP

[Juliette Estelle | oooo @JulietteSurette]

Speech is not free.

[Juliette Estelle | oooo @JulietteSurette]

Respect your elders.

[Juliette Estelle | oooo @JulietteSurette]

Psst, let’s talk.

Mental health is health.

May is Mental Health awareness month.

[Juliette Estelle | oooo @JulietteSurette]

A few things I hope we keep after the pandemic: opening streets to restaurants and pedestrians (car free), hand washing & wearing masks to curb flu season, remote work. What else did I miss ?