kali

Coding && Jiu Jitsu

[kali @glanikali]

@moneyimgoneget @TheOGOutlaw @thegenesisbl0ck Are the demographics for these jobs 98% women? if so than it makes sense, otherwise seems discriminatory

[kali @glanikali]

If you want to check out what I’m building with Team-OS:
https://t.co/ZQfq3frdZW

[kali @glanikali]

AI products don’t fail because the model is bad.

They fail because nobody can clearly answer:

“Is this actually getting better?”

When I built Team-OS, I treated evaluations as part of the product not something you bolt on after.

Here’s the eval framework I ended up using 🧵

[kali @glanikali]

9/ The goal for Team-OS wasn’t to completely remove humans from evaluation.

It was to use human judgment to establish the gold standard, then progressively get the LLM judge closer to that standard.

Binary criteria made that much easier because disagreement became measurable.

[kali @glanikali]

6/ The gold standard was still the human grader.

I’d grade the same criteria myself and compare:

Human Pass/Fail ↔ LLM Pass/Fail

That told me one of two things:
The judge needed better calibration…
or the rubric itself wasn’t clear enough.

[kali @glanikali]

8/ I think this is one of the biggest mindset shifts when building AI products.

Traditional software asks:
“Does this work?”

AI software often needs to ask:
“How well did this work, why, and is it getting better over time?”

You need infrastructure that can answer all three.

[kali @glanikali]

7/ That created another feedback loop:

Build the product → run scenarios → grade each criterion → compare LLM vs human → inspect disagreement → improve the judge → improve the rubric → run it again

The eval itself gets evaluated.

[kali @glanikali]

5/ Then I used LLM-as-a-judge for criteria that required interpretation.

Things like:
Was the handoff actually useful?
Were the requirements followed?
Did the implementation match the intent?

But the LLM was still answering specific pass/fail rubric criteria

[kali @glanikali]

4/ I used 3 different types of graders.

Code as grader for things that are deterministic.

Did the file exist?
Did tests pass?
Did the build succeed?
Was the expected artifact produced?

No reason to use an LLM when code can answer the question reliably.

[kali @glanikali]

3/ One thing I learned quickly: I didn’t want graders inventing arbitrary 1–10 scores.

I made the rubric as binary as possible.

Did it preserve the context? Pass / Fail etc

The score came from the rubric, not from the grader’s opinion.

[kali @glanikali]

2/ For example, an agent might eventually complete a coding task, but along the way it could

- ignore existing context
- call the wrong tool
- repeat failed approaches
- overwrite good work
- hand off incomplete artifact

If I only graded the final output, I’d miss most issues.

[kali @glanikali]

1/ I broke evaluation into 3 layers:

Trajectory — Did the agent take a good path to the result?

Component — Did the individual skill, tool, or agent do its job correctly?

Scenario — Can the entire system complete a realistic end-to-end workflow?

That distinction mattered.

[kali @glanikali]

@edinsoncode @viaintisol Full Stack Developer / AI Dev -> https://t.co/ZQfq3frdZW

[kali @glanikali]

@anupamrjp Short term better to use AI. Long term devs that understands code/systems will be unicorns

[kali @glanikali]

@solopribuilds Scale horizontally to stop the bleeding, then identify the bottleneck. Add rate limiting, caching, and queues to stabilize traffic.

[kali @glanikali]

@omkarships Every line of code is a liability that must be maintained especially in large systems serving many clients.

[kali @glanikali]

@TheSuranaverse Hey let's connect!

[kali @glanikali]

@FlagBlack007 cant hate the hustle

[kali @glanikali]

@thetrainerclubb What are we raiding for? Also there is an unreleased gigantamax for duraludon

[kali @glanikali]

@NianticHelp 48 raids, 0 shiny 💀