Fide AI

Measurement science for trustworthy AI in high-trust domains. Benchmarks and open research on evidence, authority, human control, and accountability.

[Alex Chao @alexchaomander]

🎤 As AI systems become more capable, who or what can reliably judge whether their behavior is correct, safe, and aligned?

In this episode of the @HumansofAIPod, I sit down with @shubadubadub and @joshnotjacob, co-founders of @SampuraResearch, an independent nonprofit exploring how humans and AI can work together to oversee increasingly powerful systems.

Rishub and Josh explain why better AI judges are central to the future of alignment. Many of the hardest behaviors to evaluate are subjective, ambiguous, or difficult to verify and neither humans nor AI systems are reliable enough to handle every case alone.

Sampura’s research on "human–AI complementarity" asks how the strengths of each can be combined to create more trustworthy evaluations, stronger benchmarks, and oversight methods that continue to work as models improve.

We discuss what meaningful benchmarks for AI judges should look like, where human judgment remains essential, and how scalable oversight connects to the larger challenge of robust alignment.

Rishub and Josh also share why they left Google DeepMind to start Sampura, how they are shaping the organization’s culture and research agenda, and the unexpected operational realities of building an independent research lab.

[Fide AI @FideAILabs]

@FideAILabs is happy to sign this open letter calling for minimum requirements for truly independent evaluators to be taken seriously by AI orgs

We look forward to collaborating with a growing bench of amazing evaluators across multiple disciplines, worldviews, and perspectives!

[AI Evaluator Forum @aievalforum]

Today, more than 100 leading AI experts endorsed a set of minimum requirements to take seriously AI companies' recent call to embed external evaluators.

These evaluators need to be genuinely independent, transparent, and represent a range of expertise areas. They also need to be guaranteed employee-level access and to be protected from retaliation for findings that make companies look bad.

We welcome model developers’ recent calls for independent oversight, but it’s what they do next that matters. The labs must be accountable for ensuring these requirements are met, so that the public can have faith in the process and the outcomes.

Over the past week, the AI community has debated the appropriate role of external evaluation, including who should do it and on what terms. We may not agree on everything, but there is a lot of common ground.

To make embedded evaluations credible, more than 100 experts with varying backgrounds and ideas about AI risk agree in today’s letter that frontier AI developers should:

1. Guarantee embedded evaluators full editorial independence and mitigate conflicts of interest

2. Rely on multiple evaluators with differing viewpoints and areas of expertise

3. Publicly document the terms under which evaluators operate, as well as facilitating permissive publication of methods and findings

4. Shield evaluators from retaliation

5. Grant access equivalent to that of highly privileged employees

There is a thriving and growing ecosystem of independent AI evaluators who are advancing this science every day – but we need aligned standards, guaranteed protections, and independent funding. That’s why we created the AI Evaluator Forum.

Today we are entering our next phase. We’re launching an open call for new members, collaborators, and independent funding sources to help evaluators meet this moment and demand accountability from developers. Join us in building the evaluator ecosystem.

See the public letter here: https://t.co/LopvS0NaFW

Learn more at https://t.co/7S7G6hia4n

[Fide AI @FideAILabs]

What should people of faith demand from AI?

Whose teachings does it represent? When should it involve a person? Who can correct it when it gets something wrong?

We offer six questions communities can bring to AI providers to make AI more trustworthy:

https://t.co/u7WnlH5Ako

[Fide AI @FideAILabs]

@MTSlive 🙋

[Alex Chao @alexchaomander]

Re: Embedded Evaluators, this is exactly what we're building with @FideAILabs and can collab with the frontier labs like @AnthropicAI to help pace the frontier!

[Alex Chao @alexchaomander]

🎤 How is AI reshaping not only what we do, but who we are?

In this episode of @HumansofAIPod, I chat with Andrew McLuhan (@amicusadastra), a poet, writer, researcher, and the founder and director of The McLuhan Institute (@McLinstitute), where he develops and teaches ways of analyzing the nature of technologies and their personal and social effects.

​As the grandson of renowned media theorist Marshall McLuhan and the son of Eric McLuhan, Andrew brings a perspective shaped by three generations of thinking about how media and technology transform human life.​

Andrew explains why artificial intelligence should be understood as more than just another tool. AI is becoming part of the environment around us, reshaping how we think, create, communicate, trust information, and understand ourselves.

​We unpack Marshall McLuhan’s famous idea that “the medium is the message,” why focusing only on the content produced by AI can distract us from its deeper effects, and what earlier shifts—from oral culture and print to radio, television, and the internet—can teach us about the present moment.

We also explore how AI overviews and agents are changing search, discoverability, and the economics of publishing; why the risks of AI extend beyond automation to mental fatigue, weakened judgment, trust erosion, and value misalignment; and how every technology strengthens certain human abilities while allowing others to fade.

Andrew offers a practical way to evaluate new technologies against our personal, family, and community values and asks which parts of ourselves we should be unwilling to outsource.

Andrew also shares his unconventional path from punk rock, poetry, and furniture upholstery to carrying forward the work of Marshall and Eric McLuhan. The result is a wide-ranging conversation about media, culture, identity, consciousness, and the question underneath nearly every debate about AI: not only “What can this technology do?” but “What is it doing to us?”

[Alex Chao @alexchaomander]

The question we should be asking more towards AI is not just “What can this tool do?” but “What kind of person am I becoming while I use it?”

[Alex Chao @alexchaomander]

Good to see more independent research studying how AI responds to mental health crises.

@TransluceAI was able to get anonymized data from @OpenAI and @AnthropicAI on how users talk with ChatGPT and Claude specifically on queries related to mental health support.

I appreciated their separation of studying the harness (the chatbot) vs the raw model (the API) which is very much the methodology we take in the "When AI is Your Pastor" paper (https://t.co/0ON9b4AerS).

Looking forward to reading through the results and seeing what lessons and future research can be done through @FideAILabs

[Fide AI @FideAILabs]

Glad we could collaborate with @apologist_ai on this important work!

[Apologist AI @apologist_ai]

Generative AI models misquote Scripture. This is not a defect that better training or better access to Bible translations will resolve. The solution is not a better model. It is a better harness. Read our joint study with @FideAILabs: https://t.co/98p8Zm7QQo

[Alex Chao @alexchaomander]

🎙️ How can AI be more humane?

In this episode of the @HumansofAIPod, I chat with Erika Anderson (@ErikaOnFire) the founder of Building Humane Technology and co-creator of HumaneBench — an open-source benchmark that asks the question most of the AI industry isn't: not whether AI can perform, but whether it treats you like a person.

As a former UN press correspondent and writing professor, Erika combines technical rigor and humanistic grounding to the field of AI accountability.

Find the podcast on:
YouTube: https://t.co/x2OiNluXuv
Spotify: https://t.co/f7kKQSHqHJ

[Fide AI @FideAILabs]

@Pontifex This is exactly the type of research we want to conduct at Fide AI! How can we deeply study these AI systems so that people (from pastors, leaders, to parents) can make decisions on whether to use or not use this powerful technology.

https://t.co/OqJWQ0z1Qg

[Fide AI @FideAILabs]

Fide AI's newsletter is live!

Serving as a companion to the main site, the newsletter breaks down research in plain language, features behind the scenes interviews with the researchers, and offers practical guidance for people navigating AI today.

https://t.co/buhdRvRct7

[Fide AI @FideAILabs]

This is one of the core ideas behind @FideAILabs.

The issue is not that AI models know nothing about theology. Many of them are surprisingly competent in the abstract.

The deeper problem is that theological fluency is not the same as pastoral discernment.

[Alex Chao @alexchaomander]

https://t.co/VV8lED7Sgs

[Alex Chao @alexchaomander]

https://t.co/VV8lED7Sgs

[Fide AI @FideAILabs]

@DanielSmidstrup faith

[Fide AI @FideAILabs]

If you want to contribute, advise, review, sponsor, or help shape this direction, reach out at [email protected]

For God's Glory.
https://t.co/zzsMv91WT3

[Fide AI @FideAILabs]

AI can pass the bar exam. Can it be trusted with grief, confession, or a crisis of faith?

Churches need real research to decide how and whether to use powerful AI systems. We're building the research lab to do that.

Introducing https://t.co/zzsMv91WT3

This is our story 🧵

[Fide AI @FideAILabs]

We'd love to connect with people working in:
- AI evaluation and benchmark design
- theology, philosophy, ethics, or moral psychology
- human review and expert judging workflows
- ministry technology and faith-based AI
- institutional governance and trust