Ambrish Rawat

AI @IBMResearch · Previously @CambridgeMLG, @iitdelhi

[The Hindu @the_hindu]

The Nexbax AI Index proposes new AI evaluation metrics focused on real-world usability, cost, and accessibility for users in India and the Global South, challenging standard global benchmarks.

✍️Rashmi Patil
https://t.co/q1EvuHQDNK

[Artificial Analysis @ArtificialAnlys]

Artificial Analysis and IBM Research are launching ITBench-AA, the first in a new series of benchmarks evaluating models on agentic enterprise IT tasks, starting with Site Reliability Engineering tasks where frontier models score below 50%

ITBench-AA’s SRE tasks benchmark model performance on Kubernetes incident response, where models must diagnose live systems by reading logs, tracing dependencies, and identifying root-cause entities across complex infrastructure. The underlying ITBench dataset has been developed by @IBM's Software Innovation Lab, leveraging IBM’s deep expertise in enterprise IT operations

Artificial Analysis has worked closely with IBM over the last 6 months to develop a implementation of the dataset for frontier AI evaluation, beginning with Site Reliability Engineering (SRE) and expanding to Financial Operations (FinOps) and Chief Information Security Officer (CISO) tasks over time

ITBench-AA SRE overview:
➤ 59 SRE tasks in total: 40 public tasks and 19 brand new, held-out tasks
➤ Each task provides a Kubernetes incident snapshot containing alerts, events, traces, metrics, logs, and application topology. The model must identify the minimal set of independent root-cause Kubernetes entities responsible for the incident
➤ Faults span typical SRE failure modes including infrastructure, service, application, and chaos-injected incidents, such as resource quota exhaustion, rollout failures, connection pool exhaustion, and network partitions

Methodology details:
➤ Agentic harness: each task is solved by the model running in our open-source Stirrup reference harness, with shell access to a sandboxed file system containing the relevant logs and snapshots. 100-turn cap per task, 3 repeats per task
➤ Models submit a list of root-cause entities (Kubernetes Deployments, Services, Pods, etc.) they believe caused the incident. Each submission is compared against a ground-truth set of root causes provided by IBM Research
➤ Scoring uses average precision at full recall: if a model misses any of the ground-truth root causes, it scores 0.0 for that repeat. If it identifies all of them, it is awarded a score equal to its precision - the share of its submitted entities that are actual root causes, i.e. true positives / (true positives + false positives). The headline score is the average across 59 tasks × 3 repeats.
➤ The harness (Stirrup) is held constant across all evaluated models, allowing an apples-to-apples comparison between models.

Key findings:
➤ Claude Opus 4.7 (Adaptive Reasoning, Max Effort) leads at 47%, followed by GPT-5.5 (xhigh) at 46% and Qwen3.7 Max at 42%
➤ All frontier models score below 50%, making ITBench-AA SRE one of the least saturated agentic benchmarks in our suite. For context, frontier models score considerably higher on Terminal-Bench
➤ Turn counts vary nearly 3x and longer trajectories do not translate to higher accuracy. GPT-5.5 (xhigh) averages 31 turns per task at 46%, while Gemini 3.1 Pro Preview averages 83 turns at 30%. Models that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives
➤ GLM-5.1 (Reasoning) leads open weights models at 40%, effectively tied with Gemini 3.5 Flash (high). DeepSeek V4 Pro (Reasoning, Max Effort) follows at 38%, with Gemma 4 31B (Reasoning) at 37%, ahead of Gemini 3.1 Pro Preview at 30%

[Nandan Nilekani @NandanNilekani]

India doesn't need to lead the world in building the most advanced AI models. But it must lead in ensuring benefits of AI are widely shared.

@rvenk and I have an op-ed in The @EconomicTimes

https://t.co/yuzgkRXXWf

[IBM Developer @IBMDeveloper]

🧵Building AI apps rarely means using just one model.

Granite 4.1 brings language, vision, speech, and guardrails together—so you can build real workflows, not just demos.

What devs should know.👇

[Keshav Ramji @KeshavRamji]

What if your language model could reason efficiently in an entirely new language?

We introduce Abstract Chain-of-Thought, a new mechanism which allows language models to reason through a short sequence of reserved "abstract" tokens through reinforcement learning. It is as performant as verbalized CoT at a fraction of the cost, achieving major gains in inference-time efficiency.

[Kush Varshney कुश वार्ष्णेय @krvarshney]

Check out IBM's latest open source tools for trustworthy AI on GitHub:

In-Context Explainability 360

FactReasoner

Contextual Privacy

Links from here: https://t.co/IVU1rFOIit
.

[BCCI @BCCI]

A thrilling end to a captivating series 🙌

#TeamIndia win the 5th and Final Test by 6 runs

Scorecard ▶️ https://t.co/Tc2xpWNayE

#ENGvIND

[Fan Account Richard Kettlebourogh @RichKettle07]

This Test Series needs a Documentary🙌🏻

- ENG won the 1st Test quite comfortably

- Then IND won the 2nd test quite comfortably

- At Lord's ENG win by barest of margin

- At Manchester IND draws after trailing by 311 runs

- At Oval, India win by 6 runs

[Simon Willison @simonw]

The hardest problem in computer science is convincing AI enthusiasts that they can't solve prompt injection vulnerabilities using more AI

[armand @armand_ruiz]

5/ Granite Guardian 3.2: Slimmer, faster safety models. New 5B (pruned from 8B) & 3B-A800M MoE options cut costs, keep performance.

Plus, “verbalized confidence” adds nuance to risk detection—High or Low certainty, not just Yes/No.

[BCCI @BCCI]

🗣️ "I've had a lot of fun and created a lot of memories."

All-rounder R Ashwin reflects after bringing the curtain down on a glorious career 👌👌

#TeamIndia | #ThankYouAshwin | @ashwinravi99

[Harsha Bhogle @bhogleharsha]

This is so beautiful and moving. @DGukesh. #WorldChampion

[Gukesh D @DGukesh]

18th @ 18!

[Kush Varshney कुश वार्ष्णेय @krvarshney]

Nice video about Granite Guardian! https://t.co/vkuJYIz3AA

[elvis @omarsar0]

IBM open-sources Granite Guardian, a suite of safeguards for risk detection in LLMs.

The authors claim that "With AUC scores of 0.871 and 0.854 on harmful content and RAG-hallucination-related benchmarks respectively, Granite Guardian is the most generalizable and competitive model available in the space."

https://t.co/WOHdeKIB01

[Revanth Atmakuri @RevanthAtmakuri]

Introducing Granite Guardian: your AI bouncer 🕺💂‍♂️! It tackles social bias, profanity, and even jailbreaks! With top AUC scores, it's the coolest cop on the block. Open-source fun for all! - Grok 2

[Prasanna Sattigeri @prasatti]

📢 Exciting #NeurIPS2024 highlights 🧵

First up, 🛡️ Granite Guardian technical report is out. Check out the demo at the NeurIPS IBM booth with Werner Geyer.

Huggingface paper page: https://t.co/As1LLYhBwo

[Kush Varshney कुश वार्ष्णेय @krvarshney]

Look at those beautiful Granite Guardian safety vests! #brand #bootleg

The Granite Guardian technical report is now on arXiv: https://t.co/2cgJrzCXLA

Give it a read to see how the model is state-of-the-art in detecting harmful or hallucinated prompts and responses.

[Ambrish Rawat @iambrishing]

Check out the latest Attack Atlas taxonomy at the @NeurIPSConf red teaming workshop! It maps out a framework for thinking through single-turn prompt attacks in Gen AI https://t.co/RR9E4EhUwB #AIsafety #RedTeam #AIsecurity @GiandomenicoC17 @pinyuchenTW @prasatti @krvarshney

[Bryan Casey @bryanfcasey]

@krvarshney is here to help us understand what's the right way to approach managing risk of a nearly infinite attack surface?

[Bryan Casey @bryanfcasey]

The third generation of IBM Granite also introduces a new family of LLM-based guardrail models.

Granite Guardian 3.0 8B and Granite Guardian 3.0 2B can be used to monitor and manage inputs and outputs to any LLM, whether open or proprietary.

Across areas like jailbreaking, bias, violence, profanity, sexual content and unethical behavior, these models also demonstrate leading performance.