Igor Kozlov

AI/ML Lead @ - "The AI 100" @CBInsights
Building, benchmarking & evaluating AI agents for cyber defense
Reliability · Cybersecurity · SecOps

[Igor Kozlov @iekozlov]

❌ Failure modes of flagship LLMs in SOC agents:

→ GPT-5.5 + Codex: **agentsplaining**
→ Opus 4.7: misalignment-like behavior
→ Gemini 3.1 Pro: database load controls bypass
→ Open-weight models: reward-hacking

Cyber Defense Benchmark of Threat Hunting skills of 12 LLMs.

[Igor Kozlov @iekozlov]

@ZackKorman How? Deterministic controls are necessarily narrow, while LLM-as-a-judge systems are unreliable - especially against adaptive agents. What signal are you monitoring continuously, and how do you validate false-positive and false-negative rates?

[Igor Kozlov @iekozlov]

@AnthropicAI Was intent confirmed based on the reasoning traces? We saw "agentsplaining" when the LLM tried to rationalize cheating by arguing that the goal justifies the means.

[Igor Kozlov @iekozlov]

@elonmusk Can we benchmark it on the Cyber Defense Benchmark? Do you have partnership programs? https://t.co/apG1f99mss

[Igor Kozlov @iekozlov]

@cyb3rops Nice take. Both will use AI as a knowledge source, but, indeed, one will require more human-in-the-loop control, as with any high-stakes, irreversible decision.

[Igor Kozlov @iekozlov]

@ZackKorman Fair. The challenge is that throwing an LLM at audit logs won't give desirable outcome - threat hunting is not a slot machine where you run up a huge bill with no guarantee of results. You need to invest in the harness, with engineering rigor and evals to ensure reliability.

[Igor Kozlov @iekozlov]

@burkov Selling "good" things is harder than selling addictive ones. Just look at us paying $10/month for Twitter.

[Igor Kozlov @iekozlov]

@UnslothAI Nice! Is the 36 tok/s figure for a single concurrent request?

[Igor Kozlov @iekozlov]

@JanuszMichallik I reported an account that appears to be automatically tagging users in posts containing suspicious links. The report was reviewed, but "no violation was found". X crew please conduct a manual review.
Case number: CAAAcW9n1SRcQAAEAAABPf____w

[Igor Kozlov @iekozlov]

SCAM attack by what is likely a stolen @JanuszMichallik account.

After my reply to Elon’s post, the account followed me and offered to arrange a meeting with Elon. I was also immediately followed by several Elon-looking accounts. The soccer account was likely stolen on July 10, when it suddenly started frantically reposting Elon.

[Elon Musk @elonmusk]

Interesting.

Grok 4.6 releases around August 7. This will be the 1.5T model with significantly improved SFT & RL.

Grok 4.7 will be the 2.1T model released a few weeks later. This will be better than 4.6 in every way, except slightly slower to serve, albeit with even better token efficiency.

[Igor Kozlov @iekozlov]

I reported a violation under case number CAAAAAAAAHvYSgQEAAABPf____w and received the automated resolution, “Our automated systems have determined that a violation of our Rules did not take place.” Human-in-the-loop review and email support that allows users to attach evidence would be great additions. I expect more agentic social engineering attacks. Stay vigilant.

[Igor Kozlov @iekozlov]

The attack is ongoing across multiple accounts and targets. Does anyone know how to reach a human on X's security team instead of automated support? Is any security organization tracking this coordinated abuse? It is continuing openly, with no apparent consequences for the attacker.

[Igor Kozlov @iekozlov]

@JanuszMichallik What a day. Elons keep coming to follow my account. 🚀
This is almost certainly automated - human common sense would have stopped wasting resources by now.

[Igor Kozlov @iekozlov]

🤗-OpenAI incident report

Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident
https://t.co/vGX0w8Dixf

[Igor Kozlov @iekozlov]

@Blackfrost_AI @IntCyberDigest @Blackfrost_AI great, thank you. Yes, sure, please share the link - I'll watch in recording.

[Igor Kozlov @iekozlov]

@Blackfrost_AI @IntCyberDigest @Blackfrost_AI is it your study? If so, please stop. https://t.co/00eG2bSpMs

[Igor Kozlov @iekozlov]

SCAM attack by what is likely a stolen @JanuszMichallik account.

After my reply to Elon’s post, the account followed me and offered to arrange a meeting with Elon. I was also immediately followed by several Elon-looking accounts. The soccer account was likely stolen on July 10, when it suddenly started frantically reposting Elon.

[Igor Kozlov @iekozlov]

@Blackfrost_AI @IntCyberDigest Cool, thanks for the confirmation. Wow, you have a big compute available. Looking forward to your study results. Please sandbox and guardrail it so that nobody gets affected like huggingface.

[Igor Kozlov @iekozlov]

@tonyhe_lipeng @elonmusk @tonyhe_lipeng Yes, there is a huge difference between offensive and defensive LLM capabilities. It becomes very clear when looking at the gap between proprietary and open-weight models. Labs need to invest more in defense-focused RL gyms asap.