Skyler Thomas

Passion for #AIops #QuantifiedSelf #DeepLearning Co-founder/CTO Cake AI. HPE, MAPR, IBM alumni. All opinions my own

[Skyler Thomas @skythomas]

@NBAonNBC @TheBAsnyder Awww

[Skyler Thomas @skythomas]

Such a great discussion! Thanks for the wonderful questions

[Eylul Kayin @EylulKayin]

Talked GPUs, OSS, and the future of inference. No latency spikes, no memory leaks. Just hot🔥takes.
@JasonLGelman @CentML_Inc @skythomas

[Gradient @GradientVC]

See y’all tomorrow

[Skyler Thomas @skythomas]

Invited to speak on a panel discussing AI in Private Markets hosted by UNC’s Institute for Private Capital and Fidelity in Boston next week. https://t.co/A6i3Zfhtrm So excited to hear from some of the other speakers!

[Cake AI @cake_ai_ml]

New case study: Learn how Cake “cut a year off the product development lifecycle” for a leading R&D platform in the materials science industry.
https://t.co/2CzhWBm0fx

[Cake AI @cake_ai_ml]

DeepSeek's launch represented more than an incremental improvement - it unlocked entirely new possibilities.

Check out our latest post on how DeepSeek provides the missing piece in the architectural puzzle for truly real-time RAG applications: https://t.co/1sN3eSXwib

[Skyler Thomas @skythomas]

I have a new post up on Deepseek and Real-Time Rag... https://t.co/BC1SlSgkzd

[Cake AI @cake_ai_ml]

[New Case Study] How Ping Manages MLOps with Cake

Ping, a data intelligence company, uses Cake to process unstructured data for more accurate property insurance quotes.

Learn how their small team saved 3 FTE engineers and doubled operational efficiency: https://t.co/c36kvpJ4z8

[Cake AI @cake_ai_ml]

New on the blog, @skythomas outlines the 6 principles of Cake’s platform that drive our goal - to bridge the gap between cutting-edge AI and those who stand to gain the most from it.

https://t.co/g6aa3D1BaB

[Skyler Thomas @skythomas]

Wrote a new blog post on the architecture of Cake's platform. Thanks! https://t.co/3klE4AoVM6

[Darian Shirazi @darian314]

Excited to lead Cake AI’s $13MM seed and partner with @MishaHerscu as he builds the RedHat of AI! @GradientVC @WKortschak @PrimaryVC

[TechCrunch @TechCrunch]

Google’s Gradient backs Cake, a managed open-source AI infrastructure platform https://t.co/RPhCt1Qbbb

[Skyler Thomas @skythomas]

Cake is officially out of stealth! We just announced our $13 million seed led by Google's @gradientvc Thank you @darian314 and @vigsachi ! Thank you @primaryvc and @bschech for believing in us early! And thank you to my rockstar co-founder @MishaHerscu. https://t.co/seSzDs2ofL

[Andrej Karpathy @karpathy]

Claude 3 takes on the Tokenization book chapter challenge :) context: https://t.co/h1IH5cuPIh

Definitely looks quite nice, stylistically!

If you look closer there are a number of subtle issues / hallucinations. One example there is a claim that "hello world" tokenizes into 3 tokens "hello" (token 31373), " " space (token 318), and "world" (token 984). Which is actually a pretty bad mistake because the unintuitive crux of the issue here is that whitespaces are prefixes in GPT tokens, so it should be "hello" and " world" (note space in front). Understanding this detail and its ramifications is important e.g. later leading to the "trailing whitespace" error message, to unstable tokens, to the need/desire for a "add_dummy_prefix" setting in sentencepiece, etc.

Anyway, it's still really impressive that this close to works almost off the shelf!

I'm looking forward to playing with Claude 3 more, it looks like a strong model. If there is anything related that I have to get off my chest it's that people should be *extremely* careful with evaluation comparisons, not only because the evals themselves are worse than you think, but also because many of them are getting overfit in undefined ways, and also because the comparisons made are frankly misleading. GPT-4 is not 67% on coding (HumanEval). Whenever I see this comparison made to stand in for coding performance, the corner of my eye starts twitching.

[Yann LeCun @ylecun]

Modeling the world for action by generating pixel is as wasteful and doomed to failure as the largely-abandoned idea of "analysis by synthesis".

Decades ago, there was a big debate in ML about the relative advantages of generative methods vs discriminative methods for classification.
Learning theorists, such as Vapnik, argued against generative methods, pointing out that training a generative modeling was a way more difficult than classification (from the sample complexity standpoint).

Regardless, a whole community in computer vision was arguing that recognition should work by generating pixels from explanatory latent variables. At inference time, one would infer the configuration of latent variables that generated the observed pixels.
The inference method would use optimization: e.g. use a 3D model of an object and try to find the pose parameters that reproduce the image.
This never quite worked, and it was very slow.

Later, some people converted to the Bayesian religion and tried to use Bayesian inference for the latent (e.g. using variational approximations and/or sampling).
At some point, when Non-Parametric Bayes and Latent Dirichlet Allocation became the rage in text modeling, some folks heroically attempted to apply that to object recognition from images.
>>> THIS WAS A COMPLETE AND UTTER FAILURE <<<

If your goal is to train a world model for recognition or planning, using pixel-level prediction is a terrible idea.

Generation happens to work for text because text is discrete with a finite number of symbols. Dealing with uncertainty in the prediction is easy in such settings. Dealing with prediction uncertainty in high-dimension continuous sensory inputs is simply intractable.
That's why generative models for sensory inputs are doomed to failure.

[Ankur A. Patel @aapatel09]

- Everyone's talking about AI
- Everyone's asking for stuff to happen
- Everyone's investigating it

But VERY few are actually getting any real value out of it right now.

The next few years are going to be incredible.

[Sumanth @Sumanth_077]

Pandas is a powerful data analysis and manipulation library for Python!

NVIDIA just made Pandas 150x faster with zero code changes🔥

All you have to add is just a couple of lines of code:

%load_ext cudf.pandas
import pandas as pd

Thread🧵👇

[Andrej Karpathy @karpathy]

I touched on the idea of sleeper agent LLMs at the end of my recent video, as a likely major security challenge for LLMs (perhaps more devious than prompt injection).

The concern I described is that an attacker might be able to craft special kind of text (e.g. with a trigger phrase), put it up somewhere on the internet, so that when it later gets pick up and trained on, it poisons the base model in specific, narrow settings (e.g. when it sees that trigger phrase) to carry out actions in some controllable manner (e.g. jailbreak, or data exfiltration). Perhaps the attack might not even look like readable text - it could be obfuscated in weird UTF-8 characters, byte64 encodings, or carefully perturbed images, making it very hard to detect by simply inspecting data. One could imagine computer security equivalents of zero-day vulnerability markets, selling these trigger phrases.

To my knowledge the above attack hasn't been convincingly demonstrated yet. This paper studies a similar (slightly weaker?) setting, showing that given some (potentially poisoned) model, you can't "make it safe" just by applying the current/standard safety finetuning. The model doesn't learn to become safe across the board and can continue to misbehave in narrow ways that potentially only the attacker knows how to exploit. Here, the attack hides in the model weights instead of hiding in some data, so the more direct attack here looks like someone releasing a (secretly poisoned) open weights model, which others pick up, finetune and deploy, only to become secretly vulnerable.

Well-worth studying directions in LLM security and expecting a lot more to follow.

[Yam Peleg @Yampeleg]

Microsoft breaks the SOTA for embedding models!

Using quick & simple training run synthetic data.

---
Paper: https://t.co/33g4aGD0Gx
Model: https://t.co/C7gmdlTEv0
Dataset (by @andersonbcdefg): https://t.co/kRlDH494gn
---

Semantic search models are very hard to get "right".

Oftentimes they require complex multi-stage training:

1. First train on general language data.
2. Then on smaller a labeled dataset.

This is because obtaining large dataset for "fetching tasks" is very hard. (and expensive)

But.

What if we could just ask an LLM to:

1. "Generate text retrieval tasks"
2. "Generate relevant document for the task"
3. "Generate hard negative document for the task"

Then simply train the LLM with contrastive loss.

In this paper the authors fine tuned Mistral-7B.

They trained both on synthetic data alone and on a mixture of synthetic and labeled data.

Result: World record!

Top of Huggingface Leaderboard.

An absolutely amazing idea.
Simple and powerful.

---
My own opinion:
This might be a pivotal moment for RAGs!

From my experience:
RAGS in real life are VERY hard to get right.

They are VERY sensitive to the performance of the embedding model.

It it fetch the wrong context, the generator model has nothing to do that can fix it.

So far, no "off the shelf" model worked good enough for me and always required further fine-tuning.

So this might be the pivotal moment for RAGs.

Now everyone can generate their own dataset with ease and train a powerful embeddings models.

Again,
Amazing Amazing Amazing!

---
Thank you to @andersonbcdefg for pointing out!
This is an important paper.
---

[Skyler Thomas @skythomas]

@aronchick @_msw_ Also recommend this for your usecase. We’ve had a good experience so far with it