Oussama Zbir

AI/ML Engineer | Agentic systems, RAG, LLM evaluation | 4x AWS | Open-source contributor (Nvidia NeMo Agent Toolkit) | Building eval-preflight

[Oussama Zbir @oussama_zbirr]

what I’d learn, what I’d skip, and a 6 month plan if I started AI engineering from zero https://t.co/V4eM0bIVhS

[Oussama Zbir @oussama_zbirr]

who should be the next Nvidia CEO?

[Oussama Zbir @oussama_zbirr]

@i_mika_el yeah. my guess is they wont open it to everyone, just sell access to a few partners

[Oussama Zbir @oussama_zbirr]

United Airlines says it will restrict AI agents on its site. eBay appears to be blocking them too. Amazon, Delta and some banks already do.

everyone is building agents that use websites. the websites are starting to say no

[Oussama Zbir @oussama_zbirr]

@OmarBoukherys 🙌

[Oussama Zbir @oussama_zbirr]

what I’d learn, what I’d skip, and a 6 month plan if I started AI engineering from zero https://t.co/V4eM0bIVhS

[Oussama Zbir @oussama_zbirr]

the short version:

- learn the boring software stuff first
- use models, don’t train them
- build one agent by hand
- go deep on evals
- skip most of the rest

the full thing has a 6 month plan

[Oussama Zbir @oussama_zbirr]

what I’d learn, what I’d skip, and a 6 month plan if I started AI engineering from zero https://t.co/V4eM0bIVhS

[Oussama Zbir @oussama_zbirr]

honest question for people building with LLMs:

what’s something you spent weeks learning that you never ended up using?

[Oussama Zbir @oussama_zbirr]

build one AI agent without a framework before you pick one

it’s a model, a few tools, and a loop that knows when to stop. about 100 lines of Python

after that, LangGraph or any agent SDK makes a lot more sense, because you know what it’s doing for you

did you start by hand or with a framework ?

[Oussama Zbir @oussama_zbirr]

The best AI engineers I have worked with were data scientists who learned to ship, not developers who learned to prompt.
The model call was never the hard part.

[Oussama Zbir @oussama_zbirr]

@mikeldking seen this from the contributor side on NVIDIA’s repo. the AI reviewer caught two real bugs in my PRs, and what made it easy was that every finding came with the why, so checking it took a minute

[Oussama Zbir @oussama_zbirr]

@sermakarevich would add one: check how many items actually got scored. when a judge call fails, most frameworks just drop it from the average and the number still looks fine

[Oussama Zbir @oussama_zbirr]

You should be counting how many eval items were actually scored.

I checked 9 eval frameworks. Only one reports it out of the box.

8 checks I run before I trust an eval number:

- Scored items vs total. 0.82 on 60% of the set is not 0.82.
- Judge errors: dropped, counted as 0, or flagged? Know which.
- Two evaluators with the same name? In one framework, a failure shifts scores to the wrong one.
- Mean over items or mean of per-case means? With repeats they differ.
- Same dataset version as the baseline.
- Same judge model and prompt as the baseline.
- How many items hit retries or timeouts.
- One run or several. A 2-point gain inside run-to-run noise is noise.

What would you add?

[Oussama Zbir @oussama_zbirr]

@sermakarevich yeah small makes sense. in depositions what worked for us was keeping each question with its answer in one chunk, with page and speaker on it. otherwise you get a chunk that just says “Yes.”

[Oussama Zbir @oussama_zbirr]

@omarsar0 70.2 vs 70.0 is noise, which is the result: same score on a quarter of the context. I’d want to see if compaction changes which tasks fail, not just how many

[Oussama Zbir @oussama_zbirr]

@svpino and the deterministic steps are the ones you can actually test. the model steps you can only eval

[Oussama Zbir @oussama_zbirr]

@JoshARosen this is how our HR agent ended up. country and department come from Workday before the first turn, and retrieval is pre-filtered on them. an agent that discovers its own context will sooner or later quote another country’s leave policy

[Oussama Zbir @oussama_zbirr]

Where does the first real bug in your LLM eval pipeline usually hide?

[Oussama Zbir @oussama_zbirr]

@svpino 77 of 255 is wild. what is the tell your bot uses, length or the structure ?