This is the AI deployment hill I will die on:
Subject matter experts should not own the product.
My company Atherial has done over 60 AI deployments. I learned this the hard way.
Subject matter experts should give the playbooks and know-how via discovery interviews, creating a rubric, and writing evals on sets of outputs.
They should not design the solution or write the requirements.
Leave the product roadmap to the SME and you get the obvious automation (copying and pasting between tools) and intern-level AI. Worth doing but not where the value is.
Here's our approach now.
You own:
• Business strategy
• Industry knowledge
• Functional expertise
Atherial owns:
• AI strategy: finding high value use cases
• Solution design and product roadmap
• Engineering
You want an FDE or AI consultant who understands why strategy, product, and engineering all matter, and who owns what.
Product management is a real skill. AI application companies pay a lot of money for it.
The best product people are mini founders. They find big and valuable problems and design solutions that take full advantage of what AI can do.
Defining what good output looks like is step one because software/harness you build changes as you test and hill climb.
Your experts should help train the agent and judge its work. FDEs who know product and strategy should decide what to build.
I think we’re talking passed each other but are in agreement. My only addition is that instead of offering software open source that a commercial license that is source available is better bc it doesn’t expose code publicly for a group that isn’t technical and is likely to make errors that will result in even easier vulnerability exploitation bc of the public nature of the codebase.
Otherwise yeah let’s build all of the legal software.
@BitGrateful I’m talking things like a real file management system (can be headless) to make Claude’s life easier and is often better than Claude projects especially for large file collections.
I think that’s arguably worse bc small firm and solo have little resource to defend. I’m basically advocating with a private software license like the old days of buying software but instead you get the source code too.
Agreed on Claude and skills getting you a lot of the way, especially early on, if there are some software modules that are worth having separately that make using Claude even better.
@JesterInABarrel@cmiglaw Let me rephrase, what’s the concern or risk compared to other cloud products? That’s usually my first question for lawyers flagging these concerns. I’ve even met some voicing this concern via email with a @yahoo.com address