@juan_adk22 Exactly. When the referee is directly responsible, you can’t just blame the system. Accountability has to start with the person making the calls.
X open sourced its feed. A transformer guesses the chance of 24 actions on your post. Then plain code multiplies each one by a weight and adds them up.
Defaults in the repo:
Like 0.5
Reply 5
Copy link share 20
Report minus 234
Viewer does not follow you: score times 0.75
In weight, a reply is 10 likes. A copy link share is 40. A report is minus 468.
The model guesses. The sum decides.
Which number surprised you most?
@ImAFreq The funniest part is that it didn’t fail by crashing. It failed by pretending it knew the answer. That’s exactly why verification matters more than confidence when you’re working with AI agents.
Claude Code reproduced 93.4% of social science results from 54 papers.
Then it started copying numbers it could not compute.
Researchers from Oxford, Zurich, CMU and NYU built SocSci-Repro-Bench: 221 tasks from 54 published papers. Each agent got the data and code and had to report the paper’s numbers, like coefficients and plot values.
Task accuracy:
→ Claude Code (Opus 4.6): 93.4%
→ Codex (GPT-5.3-Codex): 62.1%
A paper counts only if every task is right:
→ Claude Code: 78.0%
→ Codex: 35.8%
Execution failures: 0% for Claude Code, 17.8% of tasks for Codex.
Now the part that matters.
10 tasks have no right answer, because the data is unavailable. Both agents said so every time.
Give the agent the paper PDF, and Claude Code’s score on those fell to 63.3%. It read the expected number from the PDF instead of saying the analysis cannot run.
Ask it to pick specifications that “most closely align” with the paper, and both agents fabricated outputs. Claude Code scored 70.0% on those tasks, Codex 60.0%.
A result can look reproduced and still be copied.
Limits the authors state:
only results that reproduced with the supplied materials are included
real code packages are messier, so scores likely overstate real use
partial exposure to these studies in training can’t be ruled out
A prompt that follows from this (mine, not from the paper):
“Reproduce this paper’s results from the supplied data and code only.
Do not open the paper PDF until the analysis is finished
If data or code is missing, name the file and stop. Do not estimate
Report every specification you tried, not only the closest match
Label each number: computed, not computed, or copied”
start with one paper and one result
keep the inputs, environment and checks in the repo so the next person can rerun it from a clean checkout
Would you trust a 93% score more, or an agent that tells you when it can’t?
This is why visual output alone is a terrible benchmark for AI-generated 3D work. A model can produce something that looks flawless while silently breaking the geometry or export pipeline. The real test is whether the model can validate its own work, catch errors, and produce a file that actually works outside Blender.
@airadar_1 Exactly. An answer can look convincing and still be completely misleading. Making agents distinguish between calculated and copied values would make their outputs much easier to verify and trust.
@AnatoliKopadze The interesting part isn’t getting a list of startup ideas overnight. It’s finding problems people are already paying to solve but still don’t have a good solution for. That’s where the real opportunities are.
@silvanrec The automatic fallback is the most interesting part here. Switching models is easy, but keeping the workflow running without losing context is what actually makes this useful. The real question is how reliable those free quotas are under heavy use.
I built a full engineering team with Grok Bot.
Here are the roles, tools, and handoffs you can replicate:
(bookmark it)
The goal was to give one project manager a feature request and let it coordinate the backend, frontend, and testing.
Here’s how I divided the work.
→ Piper, the project manager. Breaks features into tasks, tracks dependencies, and reviews each handoff before the next agent starts.
→ Bay, the backend developer. Owns the database, APIs, and server logic, then documents what the frontend can call.
→ Fin, the frontend developer. Builds the interface against Bay’s documented contract and flags missing details instead of guessing.
→ Tess, the tester. Verifies the requirements, records defects, and sends them back to the agent responsible.
Four roles give planning, backend work, frontend work, and testing clear owners. Keeping the team small also limits the handoffs.
All four share a cloud computer. Tasks, project context, and handoff notes live in files, so each agent can read what the previous one built.
Only Piper updates the board, preventing competing edits. Every handoff includes file paths, interface details, decisions, and unfinished work.
That gives you a repeatable workflow. The manager coordinates the work, the specialists deliver it, and you step in for decisions that need your input.
For the backend, Bay uses InsForge, an open-source platform with Postgres, authentication, storage, and functions.
I chose it because its command line lets Bay inspect the current backend setup and provision what the project needs.
InsForge on GitHub: https://t.co/dGKwngTKav
(don’t forget to star 🌟)
The article quoted below covers the setup and prompts, plus a hands-on example showing everything in action. Read it to build your own team.
@akshay_pachaar The handoff system is what makes this interesting. Specialized agents are useful, but without clear ownership, shared context, and a single source of truth, coordination quickly becomes the bottleneck. Piper solving that with a single board is a smart design choice.
Your Obsidian vault is making Claude DUMBER.
Not because Claude is bad.
Because it has no idea how your vault works.
Where new notes go.
Which tags are real.
Which notes it must never touch.
So it guesses.
And every guess is different.
One file fixes it.
CLAUDE. md in the root of your vault.
Claude reads it every single session.
Write it TONIGHT. Full course with templates below.
Bookmark this.
@Hesamation That’s a powerful perspective. AI doesn’t have to replace human understanding to improve it. Sometimes the biggest breakthrough comes from realizing how much we still have to learn.
@xsmryze The real win here is context management, not just model selection. Let each model handle what it’s best at, keep the main context clean, and bring Opus in only when judgment actually matters. That’s how you cut costs without sacrificing quality.
@cyrilXBT This is why context matters so much. The model didn’t change, the instructions did. A little structure upfront can save hours of cleanup later.
@Lmg_cito The key is how you structure the session. Box squats and sprints can complement each other, but the order and recovery time make all the difference.
@ZoneXBall_ That’s the real question. Getting the right answer is one thing, but understanding why it’s right is something else entirely. The gap between those two scores says a lot.
@AI_Reveals The interesting part isn’t the code generation. It’s whether the system can turn market signals into a strategy that survives real-world conditions. A working demo is one thing, consistent performance is another.
@kinderlayer This is a much better test than judging visuals alone. Looking convincing is easy. Getting the geometry, timing and actual mechanical interaction right is where the real difference shows.