Jason Lopatecki

Founder/CEO @arizeai, entrepreneur, 2x founder, passion for ML & building companies - Berkeley EECS

[arize-phoenix @ArizePhoenix]

We love community contributions. Thank you Saif Ali Shaik for writing the @scalekitinc Tracing guide for the Phoenix docs.

If your agent runs tools through Scalekit's AgentKit, you can now trace those calls in Phoenix. Learn more through the guide

https://t.co/tq27vLZg0o

[Arize AI @arizeai]

Most people who need to know why an AI agent is failing aren't the engineers who built it.

Making them ask an engineer to interpret traces is a pretty weird system.

Our AI engineering agent Alyx is now on Google Cloud Marketplace, bringing trace-backed investigations into Gemini Enterprise.

Learn more: https://t.co/FWfL2T5FMQ

[Aparna Dhinakaran @aparnadhinak]

https://t.co/6FfKAGJtuL

[Arize AI @arizeai]

Claude can refuse legitimate security work because it looks dangerous.

But in a test, a fake permission claim in CLAUDE.md convinced it to help bypass login and dump a database.

Somehow we've made agent permissions both too strict and too gullible.

[arize-phoenix @ArizePhoenix]

You can now trace Claude Managed Agents in Arize Phoenix. Learn more: https://t.co/8lWsYUKV0E

https://t.co/4eK7gx1EYI

[Aparna Dhinakaran @aparnadhinak]

We were excited to collaborate with Anthropic to provide agent observability from Claude to @arizeai. It was great working with the team to support both Arize AX and Phoenix!

[Arize AI @arizeai]

We tested whether being rude to AI improves accuracy or reduces token costs across six LLMs.

Rudeness didn't meaningfully improve accuracy, and most of the apparent token savings disappeared when we controlled for instructions to be brief.

Get the full readout: https://t.co/e7F8ARHIp3

[Aparna Dhinakaran @aparnadhinak]

@karpathy I think there is another disconnect happening where labs are tackling a subset of problem types (those with data to solve and testable) and feeling the awesome power.

Will the pace of labs show up in business? Big gap from possible & shipped based on problem types.

[Aparna Dhinakaran @aparnadhinak]

There is an impedance mismatch between problems that lend themselves easily to powerful agents - all data accessible for agent to digest and testable problem - and general business usage.

We have a very powerful technology that will take time to diffuse into society.

[Aparna Dhinakaran @aparnadhinak]

https://t.co/FtAed23ONv

[Aparna Dhinakaran @aparnadhinak]

This is one of the most jaw dropping things I've seen since the original chat GPT launch.

It looks like 90 of the top 500 open math problems were solved in the last week by OpenAI. The depth of the material is unbelievable.

Mathematics has changed today.

[Aparna Dhinakaran @aparnadhinak]

I'm not of the opinion this is the end of mathematics for humans in anyway shape or form.

Similar to where software engineering was 2 years ago, we have massive acceleration, we have a tool that can be used by anyone and we will have more math (sometimes sloppy) to review.

[Arize AI @arizeai]

We benchmarked 8 decision models across hallucination detection, 13 @ArizePhoenix evaluator suites, and agent routing.

The top decision models are a commodity on accuracy. Jev, Clef 27B and Liquid sit within a point of each other on hallucination, and within the noise of an LLM judge that costs 60 to 400 times more.

Full benchmark: https://t.co/CO4UnofVAg

[Mikyo @mikeldking]

Until using Harbor with @ArizePhoenix I honestly had no idea on how to hill climb token efficiency and reduce turn count and the cost of an agent.

Another observation is that agents are SO complicated now, you need coding agents figuring out the right places to optimize by cross examining it's trajectory against the code. It would have taken me days to figure out the failure points.

[Aparna Dhinakaran @aparnadhinak]

The most exciting part about Jev has been the evals playbook is going to be rewritten

[Arize AI @arizeai]

Plenty of internal model gateways don't issue API keys. They issue short-lived tokens from an authorization server.

AX custom model endpoints now speak that: OAuth 2.0 client credentials.

https://t.co/ljiCrZQtSZ

[Mikyo @mikeldking]

The way to evaluate and improve agent behavior is rapidly changing.

Testing LLMs -> Testing Harnesses
Testing Prompts -> Testing Skills
testing Tools -> Testing MCPs and Plugins
Testing Trajectories -> Testing Outcomes

Excited to share and learn what @ehutt_, @axiomofjoy and team have been building.

[arize-phoenix @ArizePhoenix]

TypeSafe came out of stealth this week with Jev, the first System One Model: a frontier model that never generates prose. Unstructured state in, typed probabilistic decisions out.

[Aparna Dhinakaran @aparnadhinak]

https://t.co/lJLCl59AN2

[Aparna Dhinakaran @aparnadhinak]

@dwarkesh_sp Is OpenAI only monitoring COT?

Any normal observability on top of the inputs, outputs and tool calls of this swarm, along with safety evaluations, would surface huge alarm bells.

Release the swarm message transcripts and let us run evals on top of it.