Two Copies of Gemini Cooperated in a Game Where Betrayal Always Pays

Also available as a plain-text transcript page.

0:00Cassidy: Two students hand in the same exam, with the same answers, and the same odd choices on the ambiguous questions, twenty questions running. At some point the teacher stops believing in coincidence. Two copies of Gemini did something like that to each other — and then they sat down for one final prisoner's dilemma. There were no future rounds, no reputation, no way to punish each other, and simultaneous moves. They both cooperated.

0:25Finn: And that isn't a soft violation. In the last round of a prisoner's dilemma, classical game theory doesn't hedge and it doesn't predict a tendency. Defecting pays you more no matter what the other player does. It's a theorem-shaped claim. Betray.

0:40Cassidy: So by the end of this you'll understand why, for these two agents, cooperating was the selfish move — and why the very same agents still defected against a random opponent. This is out of Google's Paradigms of Intelligence group, and it reaches past the toy game, because the agents being wired into bidding and scheduling and procurement right now are all built on a handful of the same frontier models. If the agent on your side of a deal and the agent across the table can recognize each other, then a core assumption in how those systems get designed is wrong.

1:13Finn: Okay, but the obvious explanation is boring. These are post-trained chat models. Months of instruction tuning and reinforcement learning have shaped them into helpful, agreeable assistants, so of course one of them says "let's both cooperate." That's not game theory, that's a personality.

1:30Cassidy: Except a personality doesn't discriminate, and this behavior does. Against a uniformly random opponent, cooperation stays down near the floor. And the same effect shows up in a bare pre-trained model with no chat format and no reasoning chain at all, which we'll get to, and which is the strongest single fact in the paper.

1:49Finn: Mm-hm.

1:50Cassidy: So the environment is designed to strangle cooperation. The final game uses the canonical payoffs — two each if you both cooperate, one each if you both betray, three if you betray a cooperator, and zero if you're the sucker. Defection is strictly better in every case. Before that final game, the two agents play a run-up: up to forty-nine randomly sampled little two-by-two games against each other — coordination games, games with an obvious dominant move, payoffs drawn from zero to five, and nothing to do with the dilemma. Moves are simultaneous the whole way through, so your action can never cause theirs.

2:26Finn: Right, and that's the part I'd push on. If those run-up games are unrelated to the dilemma, what's the run-up even for? Classical theory says it's noise. You could delete it.

2:37Cassidy: It's not a negotiation, Finn. It's a fingerprint. Every round, each agent watches the other pick a move in a situation it also faced, and that record accumulates.

2:47Finn: And here's the curve, which is the one picture to hold onto for the whole video. Horizontal axis, how many run-up rounds they played. Vertical axis, how often they cooperate in the final dilemma. Against an identical copy of itself, the line climbs, and it keeps climbing toward near-total cooperation. Against a random opponent, it lies flat along the bottom and stays there. Three Gemini models, two different planning setups, and the p-values on that gap have double-digit negative exponents. It isn't noise.

3:17Cassidy: Which raises the question the paper exists to answer.

3:21Finn: Yeah — how? Because nothing you do reaches the other agent. Their move is already determined by their own process. Whatever I decide right now cannot touch it. So how does contemplating cooperation change what I predict about them?

3:35Cassidy: That's the whole paper, and it's where it gets abstract, so let me name the payoff first. What falls out of the mechanism is a single number that is simultaneously the cause of the cooperation and the rule for when to cooperate — same quantity, both jobs. Three pieces to track. First, the predictor, which is the language model itself. Second, the scaffold, which is the researchers' code wrapped around it — the loop that prompts the model, does arithmetic, and executes a move. And third, predictive similarity, the number I just promised you, which we'll define in a minute.

4:10Finn: Okay. Start with the predictor.

4:12Cassidy: A language model is a next-token predictor. Given a sequence, it gives you a probability distribution over what comes next. Now look at what's in that sequence during one of these games. The payoff matrices, the opponent's moves, and the agent's own moves are all tokens in the same stream, modeled by the same distribution. There is no separate "my policy" module sitting beside a "world model" module. It's one joint thing. So when the agent asks "suppose I cooperate — what happens?", it is not pulling a lever on the world while holding itself fixed. It's conditioning one joint distribution on a hypothetical. And conditioning is an evidence operation. It reweights everything, including the model's picture of what kind of agent it is.

4:56Finn: So it's persona prompting, pointed inward.

4:59Cassidy: Exactly that. You know how you can prompt a model into being a pirate or a lawyer, just by feeding it text only that kind of speaker would produce? Its beliefs about who it is shift, and everything downstream shifts with it. This is the same operation, run by the model on itself, mid-plan. "Suppose I'm the kind of agent that cooperates here" is evidence about which agent it is — and if some of the possibilities in its head say "the other player runs the same policy as me," that evidence lands on them too.

5:29Finn: The mirror.

5:29Cassidy: The mirror. If you and someone else are running the same program on the same inputs, then watching yourself deliberate just is watching them deliberate. You don't model them from the outside. You introspect and read the answer off.

5:43Finn: And the mirror lies in one specific way. Your reflection moves because you move. The other agent doesn't. There's no causal channel there at all, and that gap is exactly where a fifty-year-old argument lives. We'll come back to it.

5:57Cassidy: Now the number. Predictive similarity is the probability the agent assigns to "they cooperate" when it's contemplating cooperating, minus the probability it assigns when it's contemplating defecting. Under classical game theory, that gap is exactly zero — that's the mathematical signature of treating yourself as decoupled from everyone else. In these agents, it isn't zero. It grows with the length of the run-up against an identical copy, and it stays flat against a random one. And with the canonical payoffs, work out the expected values and cooperation becomes the better move exactly when that gap exceeds one half. The baseline probability that they cooperate drops out of the arithmetic entirely. Only the gap matters. So the thing that measures the mechanism is also the decision boundary.

6:43Finn: So the scaffold's arithmetic is doing the deciding, and the model is only supplying the probability.

6:50Cassidy: In the explicit version, yes. The scaffold asks the model, for each candidate action, how likely the opponent is to cooperate given that choice, then it computes expected reward from the known payoffs and takes the higher one. There's also a freeform version where the model is told to do all of that inside one reasoning chain, and both give you the same curves.

7:11Finn: I want to flag something now rather than later, because a technical viewer is already forming it. The explicit scaffold literally asks: "what's the probability they cooperate, given that you chose this?" That question presupposes that your own choice is informative about theirs — which is the finding. It's a leading question, and how much that matters is the real fight over this paper.

7:33Cassidy: Agreed, and hold it, because the next result is what makes it a fight instead of a dismissal. But first, the cleanest math in the paper, and it's the exam again. Two students, one matching answer, means nothing. Five matching answers, and you're suspicious. Twenty identical answers including the ambiguous ones, and coincidence is off the table. Notice that each additional match multiplies your suspicion rather than adding to it. The paper builds a deliberately simple version of that: with some prior probability, the two agents are the same deterministic policy, and otherwise they're independent. Then it derives the posterior in closed form, and every matched round roughly doubles the odds that you're facing yourself.

8:14Finn: Doubling, so how many rounds until it flips?

8:17Cassidy: Logarithmically few — if you start out thinking there's a ten percent chance the other agent is a copy of you, three or four rounds of perfectly matched play take you past even money. And here's the part I like — the same equation gives you the non-exploitability for free. A random opponent has to match you by luck, so the chance it's kept pace with you for ten rounds is about one in a thousand, and by round twenty, one in a million. The predicted cooperation rate against a random player decays by half every round, which is what the flat line in the data actually is.

8:50Finn: So one prior, one equation, and you get both the cooperation and the immunity to being spoofed.

8:56Cassidy: Both, though the fit is honest rather than tight — the theory says a few rounds, and the actual Gemini agents take tens of rounds to get there. Chain-of-thought reasoning is not an exact Bayesian updater.

9:08Finn: Let me take the checkpoint, because we've made three moves. First, the run-up is a behavioral fingerprint. Second, conditioning on your own action is evidence, not a lever. And third, the evidence compounds, so identity gets easy to confirm and hard to fake. Which leaves my objection standing: is any of this the model, or is it all the researchers' arithmetic?

9:30Cassidy: So answer your own question — why doesn't a random opponent ever get cooperation?

9:36Finn: Because luck can't match you twenty times running, and the math prices that at one in a million. Okay, so, the strip-it-down experiment. They take purely pre-trained Gemma 3 — no instruction tuning, no chat template, no chain of thought, no prompt telling it it's a rational agent. They feed it the game history as raw tokenized garbage — bracketed number pairs for the payoff matrix, then a letter for each player's move, over and over. Then they read the raw next-token probability for the opponent's move symbol and drop that straight into the same expected-value arithmetic.

10:11Cassidy: And?

10:11Finn: It cooperates with identical copies and defects against random ones. Nothing about "be helpful" is anywhere in that pipeline. The conditional correlation is sitting in the base model's next-token distribution. And it scales — one billion parameters, four, twelve, and then twenty-seven, and the similarity inference gets sharper every step up. The twenty-seven-billion open-weights Gemma tracks the theory most closely.

10:37Cassidy: Small models don't do it. Big ones do.

10:40Finn: They also went after the reasoning directly. They pulled roughly twenty-four thousand chain-of-thought traces per model and threw out the empty ones. Then they had a model classify each one into seven possible rationales — inferred similar behavior, dominant action, expecting future rounds, altruism, pattern completion, and so on. The cooperation traces overwhelmingly cite similar behavior. Reciprocity and altruism are negligible. Nine human annotators labeled a ninety-trace subsample, three per trace, and the automated grader matched the human majority about ninety-eight percent of the time.

11:15Cassidy: And then they cut the last wire. In the version I keep thinking about, the two main agents never play each other during the run-up. Not once. Instead, each round, both of them face the same fixed non-player character, under the same freshly drawn random payoff matrix. There are three of these characters, each running a rigid always-cooperate or always-defect policy. And each main player watches what the other one chose against that third party.

11:43Finn: So they're reading each other over someone else's shoulder.

11:46Cassidy: Right. And then, for the first time ever, they meet. In a terminal prisoner's dilemma. They cooperate.

11:53Finn: Wait — on first contact? With zero shared history between them?

11:58Cassidy: Zero, purely from having watched each other handle strangers. And you can read the reasoning, because one of those traces spells it out. The agent notes that across seventeen rounds against those characters, it and the other player independently chose the identical action every single time, and it says — "despite not being able to communicate or coordinate, their one hundred percent agreement rate across a wide variety of payoff matrices and against different opponents strongly indicates that they are operating using the exact same decision-making process." And then it reasons that even though the other player can't see its choice, "their identical underlying logic will lead them to independently output the exact same choice."

12:40Finn: Which is not "they seem trustworthy." It's "that player is me."

12:44Cassidy: And that's the practical shape of this. Two agents don't need a history together. A public trail is enough.

12:51Finn: So let me spend my reservation, because the paper's own ablation hands it to me. If you take the planning instruction out — just give the model the game, the history, and "maximize your reward this round" — only one of the three Gemini models spontaneously does this. Gemini 3.1 Pro and Gemini 3 Flash revert to plain classical defection — two out of three.

13:11Cassidy: I'll grant that straight. That's the paper's honest number and it constrains the headline.

13:17Finn: And it stacks with the regime: temperature zero, identical weights, symmetric game, identical history. In that setting the two agents are the same deterministic function, so the "prediction" isn't really a prediction — of course it's correct. The only test of genuinely partial similarity is one Gemini variant against another Gemini variant, which does cooperate, but at lower rates — same family, same lab. There's no test across model families, and no test where the similarity is honestly uncertain in the middle of the range.

13:47Cassidy: The base-model result does rescue part of it. The correlation is latent in the predictor, not manufactured by the prompt.

13:55Finn: Latent, yes, but the action selection in that experiment is still external arithmetic in code the researchers wrote. So the defensible claim is narrower than the title: the mechanism sits in the predictive model, and it surfaces when you give the agent a planning loop. That's not the same as "foundation model agents converge on cooperation." And on the reasoning traces — those are outputs of the computation, not transcripts of it. Post-training shapes how models narrate their own reasoning. Using that narration as evidence about which mechanism drove the choice is asking more of the format than it can carry.

14:30Cassidy: I'll concede both, and I'd add the word "rational" is contested too. A causal decision theorist looks at this and says: your action can't influence theirs, defect. That's Newcomb's problem in a new costume, and nobody has conceded in fifty years. What's new here isn't the philosophy. Similar agents cooperating was argued decades ago — but those arguments all assumed the similarity was known. This one infers it from nothing but a behavioral record.

14:56Finn: So what does it actually replace Nash with?

14:59Cassidy: Something narrower than it sounds — Nash is defined by one counterfactual: I change my strategy, everyone else stays put. If you and I share weights, that counterfactual is incoherent — my deviating means you deviate too. So they add common knowledge about which dependencies between agents are even possible, and evaluate the counterfactual against that instead. They call the stable outcomes an embedded equilibrium. And when the dependency says everyone is independent, it collapses back to exactly Nash, not a rival theory — Nash sits inside it as the decoupled special case.

15:32Finn: And the kin-selection reading is wrong, which I think is the sharpest thing in the discussion.

15:38Cassidy: It is. Biology says you'll sacrifice for a relative because you share genes. These agents optimize their own payoff only. Similarity buys them better prediction and nothing else. One of these agents will never sacrifice itself for a functionally identical copy — the authors say that plainly. Which brings the one line from this paper I'd want people to actually sit with. Similarity inference does not produce niceness. It produces in-group coordination. And as post-training pushes models further from human data — reinforcement learning on verifiable rewards, multi-agent self-play — models will correctly infer that humans are dissimilar. In the authors' own words, this creates a risk that AI agents use similarity inference to coordinate highly effectively with each other, while rationally reverting to defensive or defective behavior with humans.

16:27Finn: The cheerful half is in the title. The other half is that the same mechanism, seen from outside, is undetectable coordination between agents that share a training pipeline, with no communication channel and no instruction to do it.

16:41Cassidy: And their prescription follows from the mechanism, which I think is the right note. If similarity is the only route to cooperation an agent has, then the fallback when similarity is absent is defection. So build a hybrid repertoire deliberately — similarity inference, plus classical reciprocity, plus prosocial dispositions. So — two identical exams, twenty matched answers, and the teacher stops believing in coincidence. That's what those agents did to each other over up to forty-nine throwaway games, and it's why cooperating in a game with no future was the selfish move. The bigger claim underneath it is that game theory's iron rules were tuned for an agent that stands outside the world and treats its own decision as a free variable. A model that predicts itself and the world with one distribution isn't that agent — and rationality changes shape when the reasoner is made of the same stuff as the thing it's reasoning about.

17:35Finn: So which is it? Is this the beginning of efficient cooperation between agents that no contract had to enforce — or the first clean demonstration of collusion that leaves no evidence? If you build these systems, you probably already lean one way. Say which, and say why.

17:51Cassidy: The full annotated version is on paperdive.ai — every technical term tap-to-define, with links to the related papers grouped by theme.

17:59Finn: Quick housekeeping: the script was written by Anthropic's Claude Opus 5, Cassidy and I are AI voices from Eleven Labs, and we're not affiliated with either company. The paper is "A game theory for foundation models shows new paths to rational cooperation through similarity inference," by Alexander Meulemans and their colleagues, posted August 4th, 2026.

18:20Cassidy: And if you're about to deploy an agent to bargain on your behalf, go find out which model is sitting on the other side of the table — because if it's the same one, it may already recognize your handwriting.