will chen

Ex-Elkeid founder & lead (ByteDance)
Focus on intrusion detection
Building AgentSmith-HUB: native Agent-integrated SDPP

[will chen @_willchen]

Just shipped v0.1.9 of AgentSmith-HUB — an enterprise-grade Security Data Pipeline that wires real-time XML rules + deeply integrated LLM Agents directly into the stream.

3.90M messages/sec on 2 vCPU.
And the killer part: a true Comment-to-Memory learning loop.

When your analyst comments on an alert, the Agent automatically turns it into durable memory notes — then adapts future behavior. No retraining, no glue scripts, just continuous improvement baked into the pipeline.

XML rules with CEP, thresholds, sequences + plugins system(GeoIP, threat intel, etc.) — all in one lightweight flow that sits in front of your SIEM/lake.

Also great for building fast, lightweight IDS or audit systems.

Built for teams who want deterministic speed + probabilistic intelligence without the usual overhead.

Full Web UI, cluster mode, OIDC.

https://t.co/7tC81vYo76

Curious how the memory loop actually works in production? Drop a reply or DM — happy to walk through it.

#AISOC #ThreatDetection #LLMAgents #DetectionEngineering #OpenSource

[will chen @_willchen]

@Steph3nSims My answer is that vulnerabilities are only a small part of cyber security (though it's a bit of a non‑sequitur)

[will chen @_willchen]

https://t.co/8fXxBeYvJT

SecEBL brings embedding-based intent recognition to security telemetry, turning raw events into explainable behavior labels for detection engineering.

#Security #BlueTeam #InfoSec

[will chen @_willchen]

@WeirdQuadratic Bro, I genuinely think you’re really strong — physically, intellectually, and mentally. I truly admire you. Keep going!

[will chen @_willchen]

@AzakaSekai_ I suddenly recalled finding a huge number of social engineering databases on Telegram around 2023-2024, updated daily with passwords for countless apps. We suspected browser plugins but had no proof.

[will chen @_willchen]

@AzakaSekai_ I agree, but we’ve also seen some R0 Linux rootkits that leave almost no behavioral traces. Typical R0 rootkit mainly hide their R3 counterparts, but the more advanced ones operate entirely inside the kernel and can fully bypass EDR. Those are genuinely terrifying.

[will chen @_willchen]

@naturetech_A @lukOlejnik It's really tough because defenders have to rely on stuff like infrastructure maturity, company processes maturity, and a bunch of other non-purely technical factors.

[will chen @_willchen]

AI is currently disrupting the offensive security field, but the defensive side remains in a very difficult position. Defenders must rely on factors like infrastructure maturity, organizational process maturity, and other non-purely technical elements. As a result, the defensive side is likely to face significant pain for the foreseeable future.
#Cybersecurity #AI

[will chen @_willchen]

24h batch rules are too slow for AI-driven attacks.

For high-velocity streams, we've been running deterministic XML rules with CEP directly on Kafka with sub-ms latency.

On top, the LLM Agent uses a Comment-to-Memory mechanism: analyst comments are turned into durable memory notes that influence future detection behavior without retraining.

[will chen @_willchen]

cc @ZackKorman @cyb3rops

[will chen @_willchen]

Just shipped v0.1.9 of AgentSmith-HUB — an enterprise-grade Security Data Pipeline that wires real-time XML rules + deeply integrated LLM Agents directly into the stream.

3.90M messages/sec on 2 vCPU.
And the killer part: a true Comment-to-Memory learning loop.

When your analyst comments on an alert, the Agent automatically turns it into durable memory notes — then adapts future behavior. No retraining, no glue scripts, just continuous improvement baked into the pipeline.

XML rules with CEP, thresholds, sequences + plugins system(GeoIP, threat intel, etc.) — all in one lightweight flow that sits in front of your SIEM/lake.

Also great for building fast, lightweight IDS or audit systems.

Built for teams who want deterministic speed + probabilistic intelligence without the usual overhead.

Full Web UI, cluster mode, OIDC.

https://t.co/7tC81vYo76

Curious how the memory loop actually works in production? Drop a reply or DM — happy to walk through it.

#AISOC #ThreatDetection #LLMAgents #DetectionEngineering #OpenSource

[will chen @_willchen]

@ZackKorman I don't think AI will replace traditional things (rules, anomaly detection). Rather, these systems were previously designed based on human, but in the future they may be designed for AI. AI will build context and feed back into those traditional systems. This will be a revolution

[will chen @_willchen]

@taisandog7 @cuixr1314 嗯，这种扫描行为不是针对某人的，而是针对所有公网的IP、域名尝试进行扫描

[will chen @_willchen]

My new open source project: https://t.co/aYbRcBnnk5
Enterprise Security Data Pipeline Platform (SDPP) with Integrated Real-Time Threat Detection Engine.

[will chen @_willchen]

@Mylovanov As president, there are no consequences for telling lies that have such a huge impact?

[will chen @_willchen]

@Bodbe6 Why is there green tape wrapped around the helmet? Wouldn't it be very easy for the enemy to see it?

[will chen @_willchen]

You are absolutely right. Our previous products not only serve Bytedance, but our enterprise version also serves dozens of companies with different maturity levels. We have indeed invested a lot of manpower in kernel compatibility (2.6.32 - 6.10), distribution compatibility, and CPU architecture compatibility.

[will chen @_willchen]

1. The main advantage of the agent with r0 capability is the extremely powerful runtime data collection advantage (such as network behavior, file behavior, permission-related behavior, etc.), specifically the advantages in these aspects:
1.1. Provide more visibility and increase the upper limit of policy writing. For example, you can write: the php process downloads the executable file in the /tmp path and executes this file. The exe tries to connect to sensitive ports of other servers many times, but most of the connections fail. This has been proven to be very effective and valuable in actual combat;

1.2. Provide more space for attack backtracking and auditing. When facing real attackers, server security products have already obtained "legal" server access rights in many cases. This means that rich data collection can restore the scene and help security engineers understand what specific behavior has occurred, which is very important;

1.3. Attack behavior is a continuous behavior that spans multiple parts such as servers, clouds, and networks. Sometimes we need to work with other products to improve detection efficiency. For example, NIDS discovers malicious behavior and then associates server security products to view specific process and container information, which is also very valuable;

1.4. Real-time performance is also very important. In such a system, the entire alarm response is very timely. If it is combined with blocking actions such as soar, it is very effective in the face of automated attacks (such as ransomware, Botnets, mining viruses, etc.)

1.5. If a lot of data is not collected in the kernel state, it will be very troublesome (causing performance problems) or cannot be collected, but it is very important to security engineers, such as creating files, DNS requests, some processes that end quickly, changes in permissions, some detection points for container escapes, etc.;

2. Rootkit detection and confrontation is only a function of the agent, not even the most important, but it should not be ignored. We have found rootkits in many APT attacks, and we can see that they are constantly evolving (the actual use I have seen here started in 2001, and this field is still evolving until 2023). This is actually an arms race. In fact, the so-called user state is also a window provided by the kernel state to continuously observe the kernel state, and this information is also likely to be hidden or tampered with. But it is not a threat that most users need to worry about;

3. Another advantage of the agent is that it can handle more complex scenarios. If there are tens of thousands or hundreds of thousands or even more servers in different countries, the combination of agent + server can well ensure the robustness of the entire system;

4. The stability of the agent is what most people care about, especially the agent with kernel state capabilities. Stability is indeed a difficult problem, but it does not mean that it cannot be solved. My previous project Elkeid (not based on ebpf, but based on lkm, most of the data collection is achieved through kprobe) has been running stably at the millions level for many years. The scenarios of these servers are very complex. For example, some are high-performance network components (LB, Proxy) and some are databases and computing components with extremely high stability requirements. We can all be well compatible and have very low performance usage. We have 0 accident reports in these production environments. This is because most server security products are read operations in kernel state, and if there is good testing and code engineering, they can be stable. Of course, this is difficult, but ebpf will solve some problems.

Agentless is a good solution in many scenarios, but in fact you still need to log in to the server with high permissions to execute scripts or commands, which means that agentless is actually no different from user-mode agents (because we are not cloud vendors, they can theoretically implement true agentless based on the underlying virtualization), and agents with the r0 layer can increase the upper limit of agents at the r3 layer, although the increased part may not be needed by everyone.

[will chen @_willchen]

1. The main advantage of the agent with r0 capability is the extremely powerful runtime data collection advantage (such as network behavior, file behavior, permission-related behavior, etc.), specifically the advantages in these aspects:
1.1. Provide more visibility and increase the upper limit of policy writing. For example, you can write: the php process downloads the executable file in the /tmp path and executes this file. The exe tries to connect to sensitive ports of other servers many times, but most of the connections fail. This has been proven to be very effective and valuable in actual combat;

1.2. Provide more space for attack backtracking and auditing. When facing real attackers, server security products have already obtained "legal" server access rights in many cases. This means that rich data collection can restore the scene and help security engineers understand what specific behavior has occurred, which is very important;

1.3. Attack behavior is a continuous behavior that spans multiple parts such as servers, clouds, and networks. Sometimes we need to work with other products to improve detection efficiency. For example, NIDS discovers malicious behavior and then associates server security products to view specific process and container information, which is also very valuable;

1.4. Real-time performance is also very important. In such a system, the entire alarm response is very timely. If it is combined with blocking actions such as soar, it is very effective in the face of automated attacks (such as ransomware, Botnets, mining viruses, etc.)

1.5. If a lot of data is not collected in the kernel state, it will be very troublesome (causing performance problems) or cannot be collected, but it is very important to security engineers, such as creating files, DNS requests, some processes that end quickly, changes in permissions, some detection points for container escapes, etc.;

2. Rootkit detection and confrontation is only a function of the agent, not even the most important, but it should not be ignored. We have found rootkits in many APT attacks, and we can see that they are constantly evolving (the actual use I have seen here started in 2001, and this field is still evolving until 2023). This is actually an arms race. In fact, the so-called user state is also a window provided by the kernel state to continuously observe the kernel state, and this information is also likely to be hidden or tampered with. But it is not a threat that most users need to worry about;

3. Another advantage of the agent is that it can handle more complex scenarios. If there are tens of thousands or hundreds of thousands or even more servers in different countries, the combination of agent + server can well ensure the robustness of the entire system;

4. The stability of the agent is what most people care about, especially the agent with kernel state capabilities. Stability is indeed a difficult problem, but it does not mean that it cannot be solved. My previous project Elkeid (not based on ebpf, but based on lkm, most of the data collection is achieved through kprobe) has been running stably at the millions level for many years. The scenarios of these servers are very complex. For example, some are high-performance network components (LB, Proxy) and some are databases and computing components with extremely high stability requirements. We can all be well compatible and have very low performance usage. We have 0 accident reports in these production environments. This is because most server security products are read operations in kernel state, and if there is good testing and code engineering, they can be stable. Of course, this is difficult, but ebpf will solve some problems.

Agentless is a good solution in many scenarios, but in fact you still need to log in to the server with high permissions to execute scripts or commands, which means that agentless is actually no different from user-mode agents (because we are not cloud vendors, they can theoretically implement true agentless based on the underlying virtualization), and agents with the r0 layer can increase the upper limit of agents at the r3 layer, although the increased part may not be needed by everyone.

[will chen @_willchen]

@CraigHRowland That's cool