Ayan Khan 👨‍💻

GenAI Engr. | Turns GenAI Demos into Production | IIT Madras’26 | ~2 yrs exp.

[Ayan Khan 👨‍💻 @ayanarshad02]

Most AI answers sound right. I build the systems, and the evals, that check whether they are.

RAG for 500+ enterprise tenants: faithfulness 0.67 → 0.91
Open-source RAG + MCP server
Benchmark: one agent SDK lost ~half its citations

Open to remote AI engineering roles

[Ayan Khan 👨‍💻 @ayanarshad02]

@jarvisdotai Sharing my portfolio folder: https://t.co/l853NHZIdQ
GitHub: https://t.co/Jntpr6Kd0j
LinkedIn: https://t.co/MxQByK6Uvt

Would like to connect and work on a paid trial for 2-3 weeks to find out compatibility b/w both of us

[Ayan Khan 👨‍💻 @ayanarshad02]

@mickychitk I build AI agents that run real business workflows: a support assistant for 500+ companies that cut down tickets, and agents that handle customer DMs for businesses end to end

Here's my email: [email protected]

[Ayan Khan 👨‍💻 @ayanarshad02]

@Ukhan1243 the final answer. every chunk reached the model with its URL intact, and most "missing" citations were there as markdown links, not the [url] format asked for. ~60% format, ~40% a real loss. full tool-call traces per question are in the repo

happy to compare notes

[Ayan Khan 👨‍💻 @ayanarshad02]

Most AI answers sound right. I build the systems, and the evals, that check whether they are.

RAG for 500+ enterprise tenants: faithfulness 0.67 → 0.91
Open-source RAG + MCP server
Benchmark: one agent SDK lost ~half its citations

Open to remote AI engineering roles

[Ayan Khan 👨‍💻 @ayanarshad02]

"Your MCP Server Returns the Source. Does the Answer Still Cite It?" by Md Ayan Arshad #DEVCommunity https://t.co/tuvUJy2DU1

[Ayan Khan 👨‍💻 @ayanarshad02]

What I’m looking for: a full-time remote role at an AI startup building agents, RAG or evals.

Based in India, 4+ hrs US overlap. Happy to start with a paid trial on a real problem.

Know a team? An intro or RT means a lot.

[Ayan Khan 👨‍💻 @ayanarshad02]

The 3 projects in the video:

1/ Kapa-inspired RAG + MCP server: https://t.co/ilVSXRablX
2/ CSR-MCP, citation benchmark for agents: https://t.co/CqyfXT5mts
3/ Production RAG at Softeon (500+ tenants, AWS)

[Ayan Khan 👨‍💻 @ayanarshad02]

@ClaudesRH Waiting for the draw

[Ayan Khan 👨‍💻 @ayanarshad02]

repo, traces, every number reproducible:
https://t.co/CqyfXT5mts

[Ayan Khan 👨‍💻 @ayanarshad02]

your MCP server can return the perfect source

it still doesn't decide whether the agent's answer cites it

i built a benchmark to measure what happens to citations on the other side of the MCP tool boundary

[Ayan Khan 👨‍💻 @ayanarshad02]

can you trust the metric?
- holds under 3 independent scorers (−0.31 / −0.30 / −0.30)
- agrees with a blind human check on 25–27 of 30 cases

caveats: one corpus, one run per setup, self-hosted MCP server

[Ayan Khan 👨‍💻 @ayanarshad02]

@vitaliidodonov MRR

[Ayan Khan 👨‍💻 @ayanarshad02]

Such a bad experience
I just took Prime Subscription to get access to sale and buy Samsung M47
And delivery guy didn’t called or anything and marked it as rejected by customer
And now CS team is saying it will not be delivered and refund will be initiated

What I’ll do with the refund?? I wanted that phone, from last 2 days trying to solve it
If you can’t fulfill customer demands then don’t upsell prime subscription and all and don’t start the sale itself

[Ayan Khan 👨‍💻 @ayanarshad02]

@AmazonHelp @amazonIN Sent a msg pls see it

[Ayan Khan 👨‍💻 @ayanarshad02]

Reverse engineering production systems is one of the fastest ways I’ve found to learn.

Recently I built a production-grade RAG system inspired by the architecture behind documentation assistants like https://t.co/E04bMhegTf

Some results:

• Faithfulness: 0.908
• Context Recall: 0.949
• 78 frozen eval pairs
• p50: 9ms
• p99: 6.3s

The biggest lesson?

Production RAG isn’t about calling an LLM.

It’s about retrieval, evaluation, caching, observability and deployment.

GitHub:
https://t.co/6dIy8wfigd