Sam Morris

Founding Growth @ Overmind 🌱 (prev @nottecore)

[Sam Morris @samsmallmind]

renting models is how you start, owning is how you compound

[Naval @naval]

People who are serious about software train their own models.

[Sam Morris @samsmallmind]

@SaidAitmbarek @MicroLaunchHQ haha yeh not a clone!! mb!!
an upvote/pay-to-win free site ;)

[Sam Morris @samsmallmind]

@nikhilwillwrite langfuse is solid for tracing but worth asking though if you want that trace data to also feed evals + optimisation (and eventually training data)

hence would biasedly recommend @Overmind :)

[Sam Morris @samsmallmind]

@johniosifov 2350 sessions is enough data that the handoff pattern is probably visible if you score each live instead of just the final output

[Sam Morris @samsmallmind]

@Agentese_AI more tokens in many cases can just means the agent found a longer path to loop before failing. fix isn't more tokens but narrowing what the model needs to know (e.g. training off production evals)

[Sam Morris @samsmallmind]

@bijani_nishant eval traces counting as billable traces is brutal seen teams accidentally multiply their bill running one nightly eval 😵‍💫 sample prod, trace dev fully is the right instinct agreed.

have you thought of alternatives to Langsmith?

[Sam Morris @samsmallmind]

@prachet_shah cool vid!

[Sam Morris @samsmallmind]

@Stair_AI @Pseudo_Sid26 generalising across agents is the trap, agreed. what's worked for us @OvermindLab is generating evals tied to the agent's tools and inputs (agent task, trajectory etc) instead of a generic rubric

[Sam Morris @samsmallmind]

@grichadev @ZackKorman haha yup, thats the argument for benchmarking swaps against your own eval set instead of trusting a pricing comparison 🤝

[Sam Morris @samsmallmind]

@jatingargiitk @omarsar0 yup 💯
judge quality decaying as users adapt is such a specific and real failure mode, the fix has to be continuous eval sets refreshed from live production traffic instead of frozen at launch (that's what we run @OvermindLab)

[Sam Morris @samsmallmind]

@Zmarcero11 @AtMemX drift check as a phrase hides how manual it usually is. what we've found acc works is scoring every root span automatically so drift shows up as a number

[Sam Morris @samsmallmind]

@avishwakarmadev @Al_Grigor synthetic eval sets are frozen in time, prod isn't.
way forward is pulling eval sets straight from production traces so the long tail shows up before a real user finds it

[Sam Morris @samsmallmind]

@AlmustyFX curious what tasks it's hitting that capability on. we've seen small trained specialists jump from 29% to 50% on narrow production tasks, would be interesting to compare.

[Sam Morris @samsmallmind]

@elponick @AlcinLaura46908 we run evaluators live on production root spans instead of one-time passes so it catches the ranking flips as they happen. which constraint breaks yours worst?

[Sam Morris @samsmallmind]

@10footinvestor don't need their model at all if you've got production traces to train your own on :)

[Sam Morris @samsmallmind]

@mkliku 🤌

[Sam Morris @samsmallmind]

@openanalyticshq congrats

[Sam Morris @samsmallmind]

@FlyaKiet @superset_sh 🔥

[Sam Morris @samsmallmind]

@suziebuilds @LearnWithBrij yup. the piece most teams skip is catching hallucination while it's happening, needs evals running continuously on production spans not a build-time eval ran once.

[Sam Morris @samsmallmind]

@andrewdsouza fine-tuning got easier but most teams still don't have the trace-to-dataset pipeline to feed it (that pipeline is the bottleneck not the training step) - I'd recommend @OvermindLab ofc :)