@Ariella1213 The handoff matters as much as the threshold. A bare ‘not sure’ dumps the ticket on a person who starts from scratch. If the model says what’s missing, like the order number or which charge, the human closes it in one message.
@RobGuerra90@yaserabbass Done this ourselves. What worked: only write a doc once the same question shows up three or four times, and start it from the reply that actually closed the ticket. Otherwise you end up with 200 pages nobody searches.
@kobikwelu Agreed, and it changes what the last step should be: a check, not a report. Before closing, read the ticket status and the refund back from the source systems. If they don’t match what the agent claims, reopen it and pull in a human.
@aydinsaas Add one more source: what people type into your in-app chat or search box in week one. Far more users quietly ask than file a ticket, and the quiet ones are the ones who churn without ever telling you.
@adeiji1@am56ay Worth finding the exact step where they stalled. With early accounts it’s usually one thing, like importing their data or connecting the first integration. Do that step with them on a call and they hit value before they’ve had time to give up.
@Emad0x1 The step that decides quality is the one before #1: chunking. Split a refund policy mid-table and retrieval hands back half the answer, and the model fills in the rest with confidence. Reading a few raw chunks before tuning anything else.
@sugumaran___ Good rule. One trap I’d add: teams fine-tune to fix wrong answers that are really retrieval misses. When the answer never made it into context, no amount of training fixes that. Check what got retrieved before touching the model.
@I_mdimeji In HVAC the window is minutes. Someone with a dead AC calls three companies and books whoever picks up first. The win usually isn’t a smarter nurture sequence, it’s answering after hours and getting a time on the calendar before they dial the next number.
@TGWorldPlus Two triggers cover most of it: the action can’t be undone (refund, cancel, payment, deleting a record), or two sources disagree. Reading, searching and drafting can run freely. Most teams get it backwards and gate the harmless steps instead.
@patelnik7 Good test. My guess at what it flagged: no pricing it could read and no plain ‘who this is for’ line. People skim past those gaps, an assistant reports them straight back to the buyer. What was the verdict?
@Levi_2z Fair at 24 a week. The math flips with volume, or when those 15 questions show up at 2am and the person leaves before you wake up. Worth checking how many arrive outside your hours before deciding. Below that, a good FAQ page does most of it.
@ppgold3 Before building anything, tag a week of call logs by reason. Usually 3 or 4 intents (order status, refunds, password resets) are half the volume, and they’re the safe ones to automate first. The angry calls should still reach a person, just faster.
@dg_codes_apps I’d add one item to the don’t-automate list: a customer asking the same thing twice. If they rephrase, the bot already missed once. Hand off on the second ask, not the fifth. That rule alone kills most of the ‘AI support is a wall’ complaints.
@elfcelbunni New support problem: customers walk in with a confident answer about your store that you never gave. The chat bot guessed from old pages or a competitor. The only real fix is having your own site answer ‘do you stock X’ first, and correctly.
@JnBrymn Underrated shift. A customer’s assistant can’t click through a pricing page built for eyes. Sites that publish plain answers (plans, limits, refund rules) win those visits. Most still bury them in images, tabs and accordions.
@elliottrayneai The hard part is getting the agent to notice what it’s missing. Most failures are stale data, not absent data: an order edited 10 min ago, two accounts on one email. Both look complete, so it answers confidently. Freshness checks beat prompt tuning there.
@AutumnCapital The tell is what the bot is allowed to do. If it can only quote the help center and can’t refund, rebook or cancel anything, it’s a wall with a chat bubble on it. The ones that work get the same powers as a tier 1 agent, plus a fast way to a human.
@mohammadali0820 Write 30 real questions from your users before picking any framework. Then grade retrieval alone: did the right chunk land in the top 5? Most bad answers are retrieval misses, and you never see that if you only judge the final output.
@matt_ambrogi One thing to add: dedupe before it hits Slack. Agents will file the same missing field complaint fifty times in fifty phrasings. Clustering and counting them is what tells you which fix matters, otherwise the channel turns into noise within a week.
@MikeCoulier@parkersmith Worth checking that score against what your team actually changes. Early on, model confidence and edit size barely line up. Letting Echo go solo per ticket category, once its edits there stay tiny, has been safer for us than trusting the score.