This kinda exploded.
The consensus: stop testing if someone can write code. AI can do that. Start testing if they can work with AI, know when it's wrong, and make good architectural decisions.
The most interesting ideas from the replies:
1️⃣ Real codebase work
2-3 hour sessions implementing a feature in your actual repo. AI tools allowed.
2️⃣ Discuss trade-offs instead of watching them code
Pick parts of their solution and dig into why they made certain choices. You learn way more about their understanding.
3️⃣ Paid short-term trials
Work with 2-3 candidates part-time on real tasks. Pay them properly. Then decide.
The questions you should be asking:
• Can they prompt effectively?
• Do they know when AI output is garbage?
• Can they handle ambiguity?
• Do they understand architecture and trade-offs?
• Can they debug and iterate on generated code?
System design interviews got more valuable because AI is great at implementation but terrible at making higher-level decisions (not sure for how long).
Behavioral stuff matters more now. References. Past work. How they talk about their decisions.
So our plan: live coding session (implement something real with AI tools) + deep dive on past work + system design discussion.
Still figuring this out, but the replies helped a lot. Thanks everyone!