aterikob

Make Things People Want

[aterikob @aterikob]

#MyTwitterAnniversary #MyXAnniversary

[1X @1x_tech]

Introducing NEO Gamma.
Another step closer to home.

[AK @_akhaliq]

This is HUGE

The AI App store is here

Ask anything you want to do with AI

With ~400k Apps, this is the best place to find the AI apps you need

developers can build apps, users can try them out and find new apps with AI search

[Kimberly Tan @kimberlywtan]

📢 New @a16z thesis: AI will unbundle the BPO and disrupt the $300b outsourcing market.

Enterprises often outsource important, but high-volume and repetitive work to BPOs. These BPOs rely heavily on human labor, often leading to slow turnaround times, human error, and unsatisfactory results.

With modern AI, there's a clear opportunity to productize the BPO and unbundle the use cases they have dominated for decades.

[Mckay Wrigley @mckaywrigley]

The future of work is just talking to an AI whether it be via voice or text and it doing what you ask.

Btw - you’ll notice the project is called “o3-mini-jarvis.”

Needs quite a bit of work, this is like a 20min hacky test, but I’ll open-source it once I get it working well.

[Austin Scholar @AustinScholar]

Harvard says AI tutors are better than Harvard professors.

The students who used an AI tutor in a Harvard physics class (instead of the professor’s teaching) learned more than 2X as much.

The best part?

They did it in less time and were more motivated and engaged.

[Leonie @helloiamleonie]

I’m trying to wrap my head around multi-agent system architectures.

Here are some patterns I’m seeing so far:

1. Type of collaboration:
Network vs. hierarchical
(also sometimes called horizontal vs. vertical)

2. Type of information flow:
Sequential vs. parallel vs. loop

3. Type of functionality:
Routing vs. aggregating

Since I’m new to this whole topic, I probably missed a bunch.

Please tell me, which ones you are also seeing.

[aterikob @aterikob]

Tbh...

[Leonie @helloiamleonie]

Still catching up on ModernBERT after the holidays?

(Its embedding model is already here!)

On December 19th:
A collaboration between Hugging Face, AnswerAI, and LightOn brought us ModernBERT.

ModernBERT is a new encoder model family that outperforms BERT in many aspects while maintaining full backward compatibility:
• Better: Outperforms on downstream tasks
• Faster: 2-4x faster inference speeds
• Longer: 16x larger sequence length

On December 30th:
Nomic AI brought us ModernBERT embed.

ModernBERT embed is a new embedding model built on ModernBERT-base, with all its advances:
• Better: Outperforms nomic-embed-text-v1 and nomic-embed-text-v1.5 on MTEB
• Faster: Immediate FA2 and unpacking support for efficient inference
• Longer: Embeds longer passages (up to 8,192 tokens)

Both ModernBERT and ModernBERT embed are Apache 2.0 licensed (fully commercially permissible).

The recipe notebook by my colleague Mary shows you how to use ModernBERT embed with @weaviate_io for vector search.
(Fully open source. No registration and no API key are required)

🔗 Try it out: https://t.co/7ynjb6wB5d

[rohit @seatedro]

I built a new thing

arxival [dot] xyz

it's a search engine for machine learning papers. ask any question related to ml and it will answer with citations and figures!

[Logan Kilpatrick @OfficialLoganK]

Straight shot to ASI is looking more and more probable by the month… this is what Ilya saw

[Eric @EricFlaningam]

The inference market is one of the more interesting ones in AI right now.

With OpenAI’s o1 release (more compute at inference) and increasingly complex AI applications on the market (Anthropic’s Computer Use for example), we’re starting to see the potential for the inference market.

Jensen’s said things like inference will be a billion times larger than training.

Landscape mapped out below (not an exhaustive list of companies, just a mental model for the industry):

[shyamal @shyamalanadkat]

orchestrating agents: routines + handoffs code example https://t.co/gAXtNwPpsg

[shyamal @shyamalanadkat]

introducing swarm: an experimental framework for building, orchestrating, and deploying multi-agent systems. 🐝
https://t.co/97n4fehmtM

[Karina @karinanguyen]

For the first time we are fundamentally changing how humans can collaborate with ChatGPT since it launched two years ago.

We’re introducing canvas, a new interface for working with ChatGPT on writing and coding projects that go beyond simple chat.

Product and model features:

1/ Ask for in-line feedback. With canvas, ChatGPT can better understand the context of what you’re trying to accomplish. You can highlight specific sections to indicate exactly what you want ChatGPT to focus on. Like a copy editor or code reviewer, it can give in-line feedback and suggestions with the entire project in mind.

2/ Directly edit the model's output and select a specific area for targeted editing. You control your creative work on canvas. You can directly edit text or code.

3/ Menu of shortcuts. There’s a menu of shortcuts for you to ask ChatGPT to adjust writing length, debug your code, and quickly perform other useful actions. You can also restore previous versions of your work by using the back button in canvas.

4/ Use search with canvas for research writing! As we are moving towards the new paradigm of reasoning we are fundamentally evolving the chat interface into a more collaborative human-AI interaction. Today you can say “browse / use browsing to find XYZ on the internet and write a report in canvas”

[Sam Altman @sama]

realtime api (speech-to-speech): https://t.co/cfUkb9hTsq

vision in the fine-tuning api: https://t.co/fDFazE7xvZ

prompt caching (50% discounts and faster processing for recently-seen input tokens):
https://t.co/97mz86YVKR

model distillation (!!):
https://t.co/4wH1bmP7rb

[Greg Kamradt @GregKamradt]

Scaling LLM apps - OpenAI break out session

Going from 1K > 10M users
Making apps better, faster, and cheaper

Accuracy
* Start by optimizing for accuracy, use the most intelligent model you have
* Build evals > Set Target > Optimize
* People skip the "set target" but then it's an emotional decision. Make it objective and define what is good enough
* Eval-driven development: Don't ship a component until you have a eval to test it. 1) Component evals "Did this question get routed to the right intent" 2) End-to-end evals "Did the customer achieve their objective?"
* Every time you raise a PR, run your eval tests to see if performance degraded
* How accurate is good enough? Set an accuracy target. You can set this by defining the dollar value of the positive and failure cases.
* How to make your apps more accurate: Prompt engineering, RAG, Fine-tuning, do it all
* Lean into meta-prompting. One of the use cases that o1 is best at. o1 makes the prompt for you.
* Extend evals to retrieval as well
* Fine-tuning: You only need about 50 examples, not many
* Techniques to try in your LLM stack (best worst): Retrieval with cosine similarity, hyde, fine tuning, chunking, reranking, classification step, prompt engineering, tool use, query expansion

Latency
* 3 types: Network latency (how long it takes to get to the GPU once they get the request), input tokens latency (time to first token), output tokens latency
* Network Latency: 200ms, this is on the OpenAI side. It's getting improved. They're going global.
* Time-to-first-token: How to make it better, use shorter prompts, smaller models, prompt caching and cache hit
* Time-between-tokens: Supply/demand is a factor. Weekends are the fastest, during the week days, the mornings PT is the most demand.
* Basically, ask the model to give you the bare minimum information that you actually need. Prompt length matters. Longer prompts = higher latency

Cost
* More inference for the same dollar budget. More requests, less money.
* Prompt caching - Reuse commonly used prompts. Put the static stuff at the beginning. If 1 token is different, you'll get a cache miss
* 5-10 minutes is the cache life. The cache always clears every hour.
* Save 50% on cached tokens. No extra work is required. You just do it and your bills should go down
* Use batch API by running your requests async. Create a batch files to create a large number of requests. It goes quicker at non-peak times
* EchoAI plans on saving $1M with batch for their call classification tasks

[AK @_akhaliq]

Microsoft releases GRIN😁 MoE

GRadient-INformed MoE

demo: https://t.co/DW48a5cc7D
model: https://t.co/O6nD3xreir
github: https://t.co/mcd2QF3DgA

With only 6.6B activate parameters, GRIN MoE achieves exceptionally good performance across a diverse set of tasks, particularly in coding and mathematics tasks.

[Tsarathustra @tsarnick]

Jensen Huang says technology has now reached a positive feedback loop where AI is designing new AI and is now advancing at the pace of "Moore's Law squared", meaning that the progress we will see in the next year or two will be "spectacular and surprising"

[1X @1x_tech]

hello, world model

https://t.co/pSAhdFZGdV