Armaan Waels

FDE @ Accenture. Prev. co-founded Cogentic (enterprise AI → local-first AutoML). Evals, inference cost, MCP, RAG in production. Georgia Tech MSCS.

[Armaan Waels @ArmaanWaels]

Notion's agent could not finish an edit through the API, so it opened a browser and typed into Notion like a person. @sarahmsachs at Ship SF: models don't give up, and that is not always good.

Her other ask: move work off the GPU. There is a place for state machines.

[Sarah Sachs @sarahmsachs]

We need bring work from GPUs to CPUs for durable and predictable agent actions. Loved talking all things open context and sandboxes with @cramforce at Ship.

[Armaan Waels @ArmaanWaels]

"If the agent inherits my user permissions, we've already lost." Aaron Brown, Head of Security at @mercor.

Under a dozen projects last year, 2,000 now, most outside engineering. No review per project. Instead: identity in front, every key with an owner, expiry, spend cap.

[Armaan Waels @ArmaanWaels]

Vercel's Slack agent v does not act as the person asking. Ask in a public channel and it mints a short-lived token scoped to what everyone in that channel can see, because that is where the answer lands.

Who asks and who will see it: two inputs. @mttlws at @vercel Ship SF.

[Armaan Waels @ArmaanWaels]

70 bugs planted in a 116K-line codebase. One agent with one long context found 14, 15 and 27 across three runs. A dynamic workflow with a separate grader found 66 every time.

@katelyn_lesse (@AnthropicAI) at Ship SF. An agent grading itself thinks it looks great.

[Armaan Waels @ArmaanWaels]

@ismaildascii Were the 2 wrong "met" calls on the parser side or the judge side? That split decides whether you fix the code or the rubric.

[Armaan Waels @ArmaanWaels]

Tell a model to stop using em dashes. A few turns later, they're back.

Sandhya Subramani's fix at AWS Community Day: hooks in the agent loop. Code for rules you can compute, an LLM judge only for judgment calls.

What rule did you move out of your prompt and into code?

[Armaan Waels @ArmaanWaels]

@cgranier Nice. One more for the test set: find, which passes as read-only until -delete or -exec shows up.

[Armaan Waels @ArmaanWaels]

You can't write evals for failures you haven't seen. Your production traces already have.

Yuki Watanabe at @databricks : MLflow issue detection clusters failing traces into issues, and those traces become the eval.

Fix verified on the failing traces. What catches regressions?

[Armaan Waels @ArmaanWaels]

@cgranier That's the whole gap: the prompt said never, the permission mode said yes. Does your hook parse chained commands too, like ls && rm in one call, or only the first command?

[Armaan Waels @ArmaanWaels]

An agent acting with its owner's credentials is indistinguishable from the owner.

At @auth0's Camp AI (SF Tech Week), the harder question on the panel was keeping an agent's actions matched to user intent over a long session.

Hours in, what signal tells you it has drifted?

[Armaan Waels @ArmaanWaels]

An agent is in production when the questions change: can we use a smaller model, who governs it, why is the bill this high.

Heard on the noon panel at the Cortex Cafe, @Snowflake and @googlecloud's SF Tech Week takeover, @AnthropicAI on stage. Which of the three hits you first?

[Armaan Waels @ArmaanWaels]

The faster inference engine flipped when I went from 1 request to 32.

One L4, Qwen2.5-7B: vLLM first token 72 ms vs SGLang 120 at 1 request. At 32 concurrent, SGLang 258 vs vLLM 639.

@NVIDIAAI's Jen Hoskins made the case at @GoogleStartups. Replay agent traffic or bench flat?

[Armaan Waels @ArmaanWaels]

@colepseifert @heavybit Super interesting night, it was amazing to catch up and hear your thoughts on stolen keys!

[Armaan Waels @ArmaanWaels]

At the @GoogleStartups hub with Cogentic, invited for the Google DeepMind panel at SF Tech Week. Best line: build the eval for your domain before you pick the model.

Plan with the strong model, hand it to a cheap one, decide where it runs.

Which eval changed your model choice?

[Armaan Waels @ArmaanWaels]

An agent acting with its owner's credentials is indistinguishable from the owner.

At @auth0's Camp AI (SF Tech Week), the harder question on the panel was keeping an agent's actions matched to user intent over a long session.

Hours in, what signal tells you it has drifted?

[Armaan Waels @ArmaanWaels]

An agent that takes eight steps pays for inference eight times.

@NVIDIAAI's Jen Hoskins made that case at the @GoogleStartups hub. My L4 bench agrees: SGLang 258 ms vs vLLM 639 ms TTFT at 32 concurrent prompts. At 1 request vLLM wins, 72 vs 120.

Do you bench at real fan-out?

[Armaan Waels @ArmaanWaels]

The agent output I'd trust is one a checker can reject.

@MistralAI showed Leanstral 1.5 at Meet the Lab in SF: 44/672 PutnamBench proofs at 50k tokens, 587 at 4M, each checked by Lean.

Catch: hunting Rust bugs, the model also wrote the properties Lean checked. Who writes yours?

[Armaan Waels @ArmaanWaels]

Two lines from this night that describe the FDE job better than any JD.

Reducto: "Congrats, you're in sales now." Exa: "Evals are the FDE's product spec."

They're the same point. You win the customer by being the engineer in the room, and you keep them by turning what they actually mean by "correct" into an eval suite that drives the roadmap.

Demos win the room. Evals are what ship.

[Reducto @reductoai]

Last week we hosted "From the Frontlines: Technical Talks with FDE Leaders" at our new office in SF! This was an evening built for forward deployed engineers, solutions engineers, and the customer-facing builders who live at the intersection of product and people.

Thank you to our amazing partners and speakers for the evening:

🎤 Nikhil Harithas (@FactoryAI) — "Deploying Droids at Blackstone: How one enterprise made us rethink compute at Factory"
🎤 Josh Nkoy (Reducto) — "Congrats, You're in Sales Now: How engineers win customers without becoming salespeople"
🎤 Yi Y. (@ExaAILabs ) — "Evals Are the FDE's Product Spec: Closing the customer <> product loop at Exa"

Thanks to all the amazing FDEs who came out for a great evening. Just the beginning for community events in our office! Follow for future events! 💜

[Armaan Waels @ArmaanWaels]

After codegen, the next thing worth automating is judgment. Only the kind a team can write down, like what counts as a P0 vulnerability. People keep the accountability.

Best idea I heard at @datadoghq Summit SF, from @embirico of @OpenAI.

[Cogentic @CogenticHQ]

Everyone says it in the meeting.

Then somebody asks a real question.

Whether this defect shows up more on nights, or which supplier's lots are actually causing the rework.

And the room goes QUIET.