MLflow

The open source developer platform to build AI applications and models with confidence.

[MLflow @MLflow]

Part 3 of the Self-Improvement & MLflow series by MLflow Ambassador Jonathan Bown walks through automating the verification layer of a self-improving AI workflow with MLflow and CI/CD.

A coding agent reads traces at scale and can propose changes to itself, its skills, tools, memory, or the verifier. CI then checks each change before it is approved automatically or by a person.

Read Part 3 👇

https://t.co/1jlAUwUpuC

[MLflow @MLflow]

Scattered API keys. Different SDKs. No clear view of usage or cost.

MLflow AI Gateway provides one @OpenAI-compatible endpoint, encrypted provider keys, and traces for requests, latency, tokens, and cost.

How it works 👇
https://t.co/0RKPsyRhZ5

#MLflow #LLMOps

[MLflow @MLflow]

What happens when every team has its own agents in production? You need more than LLMs. You need a control plane.

For the third video in the MLflow A–Z series, MLflow Ambassador @shrinathsuresh covers C for Control Plane. He explains how MLflow AI Gateway brings access control, budgets, guardrails, traffic routing, and usage tracking into one place.

🔗 Learn more: https://t.co/DZ4Kvvmyeg

#MLflow #GenAI #LLMOps

[MLflow @MLflow]

The Claude SDK makes it fast to build an agent. With native @opentelemetry export and MLflow, it is just as fast to know whether the agent works.

🔹 Trace every model call by pointing a standard OTel exporter at MLflow
🔹 See each run as one trace, with model calls and tool calls under one agent span
🔹 Find expensive runs from token usage on the trace
🔹 Score every run with an LLM judge that inspects the trace
🔹 Compare system prompts with the same test prompts and judge

🔗 https://t.co/OMZfUQenTl

#MLflow #Claude #OpenTelemetry

[MLflow @MLflow]

Part 2 of MLflow Ambassador Jonathan Bown’s series on self-improving AI workflows with MLflow is live.

He walks a support agent through the loop: traces, feedback, expectations, and judges in; a versioned prompt and an evaluation dataset out; then a production alias move so the agent updates without code changes. @mlflow.test turns that dataset into a quality gate.

When the manual pass stops scaling, a coding agent with MLflow skills proposes the themes, scorers, and tests. A human still accepts the changes. CI can run them as MLflow Evaluation runs.

🔗 https://t.co/GvmmaUmJkZ

#MLflow #GenAI #ODSC

[MLflow @MLflow]

Claude Platform now supports native @opentelemetry tracing across its agent ecosystem. Bring traces from the Claude family into one backend using OTLP, the OpenTelemetry Protocol:

🔹 Messages API and Claude SDKs: model requests and tool-runner activity
🔹 Claude Code and Agent SDK: agent steps, tool calls, and subagent activity
🔹 Managed Agents: agent turns, model runs, and platform-run tools

We published a guide covers setup OpenTelemetry export across the Claude family, including Claude Platform UI and SDK configuration.

👉 Setup guide: https://t.co/0tQqWdYBfv

#MLflow #opensource #otel

[MLflow @MLflow]

MLflow AI Gateway now supports TypeSafe System One calls.

Create an endpoint for a Jev model, keep the TypeSafe API key on the server, and send structured questions using the endpoint name. You can change the configured Jev model without changing each application that calls it.

Gateway tracks usage, supports budgets, and can fall back to a compatible Jev model from another provider, such as OpenRouter, after upstream errors or rate limits. Each endpoint gives you one place to manage access and monitor use for its API.

🔗 Learn more about the TypeSafe Gateway provider:
https://t.co/3DPmyGUcCn

#MLflow #GenAI

[MLflow @MLflow]

Yes/no evals don’t need a general-purpose LLM on every row. That gets slow and expensive at volume.

In MLflow 3.17, TypeSafe’s Jev models work as judges. Keep your existing mlflow.genai.evaluate() flow and set model="typesafe:/jev-latest" on make_judge(). Jev returns a structured decision; MLflow stores it as an assessment and puts the probability in metadata, so close calls show up when you review. A versioned model ID keeps comparisons consistent across runs.

🔗 Learn more: https://t.co/a6L5oCwsVA

#MLflow #GenAI #LLMOps

[MLflow @MLflow]

MLflow 3.17.0 is now available! 🎉

Major new features include:
✅ Jev Decisions: Use TypeSafe’s Jev models as judges in mlflow.genai.evaluate() for yes/no and finite-category checks. Tracing and AI Gateway support are included, so those calls can be inspected and governed in one place.
🔐 Fine-Grained Permissions: Control access to runs, traces, assessments, logged models, review queues, and model, prompt, scorer, and MCP-server versions. Grant an experiment without exposing everything inside it, including with an explicit DENY.
📈 Faster Trace Analytics: SQL-backed dashboards spend less time on trace counts, latency, token usage, costs, and assessments. Optional daily rollups reuse precomputed summaries as datasets grow.

Release notes 👉 https://t.co/TcQcSQUDh0

#MLflow #GenAI #LLMOps

[MLflow @MLflow]

Your AI agents can be autonomous, but your budget shouldn't be. 💰

In the second installment of the MLflow A-Z series, MLflow Ambassador Shrinath Suresh covers B for Budget Policy.

Open the MLflow UI, go to AI Gateway, and create a new budget policy. Set a threshold, pick a daily, weekly, or monthly window, and choose whether to raise an alert or reject subsequent requests.

🔗 Docs: https://t.co/ILhzQfWYKp

#MLflow #GenAI #LLMOps

[MLflow @MLflow]

At the next Agentic + AI Observability meetup in SF, Yuki Watanabe will show how open-source MLflow observability uses agent traces in production. 🔍

Agent traces already capture where your agent gets stuck, which mistakes repeat, and where users get frustrated. This session walks through automatic trace analysis in MLflow to surface those patterns, then turn the findings into improvements you can verify.

🗓️ Oct 13 · 5–8 PM PT · @databricks SF

🔗 View the full agenda & join the waitlist: https://t.co/mKsESn57wH

#MLflow #OpenSource

[MLflow @MLflow]

Clinical RAG can sound confident while citing the wrong passage.

MLflow Ambassador @pavan_mantha1 built a claim-level verdict gate using MLflow. One server serves the embedding model, reranker, and claim checker from the Model Registry, and the LLM sits behind MLflow AI Gateway. Each statement is checked against the passage it cites. The gate then releases the answer, drops unsupported claims, or abstains.

🔗 https://t.co/GB97Z5D4Mq

#MLflow #RAG #OpenSource

[MLflow @MLflow]

What does it take for an AI system to improve over time?

In this blog, MLflow Ambassador Jonathan Bown breaks down the loop:
→ Capture signals from traces, feedback, judge verdicts, and test results
→ Store them in trace logs, evaluation datasets, or a prompt registry
→ Update a component, then measure the difference

🔗 Read Part 1: https://t.co/5z9XbG09x6

#MLflow #OpenSource #AgentOps

[MLflow @MLflow]

Jev faced a harder LLM-judge dataset.

✅ Jev: 64/72 human labels
✅ GPT-OSS-120B: 72/72
⚡ Median latency: 0.20s vs 1.44s
💰 Estimated Jev cost: $0.020 per 1,000 judgments

In retrospective analysis, routing uncertain judgments to a second model corrected the observed misses while escalating <20%.

Learn more: https://t.co/OuoHjL5brb

#MLflow #LLMOps

[MLflow @MLflow]

We're going from A to Z with MLflow. 🔤

MLflow Ambassador Shrinath Suresh kicks off a 26-part series: one feature per letter.

First up: A is for Autolog.

mlflow.openai.autolog() captures prompts, responses, token usage, latency, and tool calls. Also works with LangChain, DSPy, LlamaIndex, and more.

📘 Docs: https://t.co/RoJGizoZtR
🎥 Walkthrough: https://t.co/ErjhBUEElO

#MLflow #GenAI #LLMOps

[MLflow @MLflow]

MCP has become a popular way for agents to connect with external tools and data. The MLflow MCP Registry provides a central place to register, version, discover, and alias MCP servers. 👇

In part two of this MLflow MCP series, learn how to:
🗂️ Register custom and third-party MCP servers
🔖 Version and alias them as tools move from staging to production
🔌 Use registered tools from an MCP client or Claude

https://t.co/mh04VjO1WN

#MLflow #MCP

[MLflow @MLflow]

We compared Jev, TypeSafe’s model for structured decisions, with GPT, Claude, and DeepSeek using MLflow.

On 30 human-labeled QA examples, Jev matched the best agreement at lower cost and latency:
✅ 30/30 agreement with human labels
⚡ 369 ms median latency
💰 $0.0247 estimated cost per 1,000 judgments

Our new blog walks through building a Jev scorer in MLflow, measuring quality, cost, and latency, and deciding when to use another judge. Try the same comparison on your own dataset.

📖 Full blog: https://t.co/NuZ5Qt59Ue

#MLflow #OpenSource

[MLflow @MLflow]

A multi-agent app can return 200 OK and still be wrong. Request dashboards miss bad tool calls, empty MCP data, and stale-prompt token burn.

In this video, @LegareKerrison (@RedHat) covers MLflow tracing, LLM-as-judge evals, and production setup.

🎥 Watch: https://t.co/wqHeHDk2hd

#MLflow #LLMOps

[MLflow @MLflow]

Use the MLflow MCP server to query tracking data from code or an assistant. 👇

🔹 Async Python FastMCP client, scoped to genai or traditional ml tools
🔹 Configure Claude, VS Code, or Cursor, then ask in plain English
🔹 ~20–30 read/write tools; runs locally or next to tracking

Watch the full tutorial: https://t.co/cQGNw6C3FE

#MLflow #MCP #GenAI