DPO is the steadiest negotiator when LLMs are trained on the final outcome

LLMs can hold a conversation, but move them into a multi-agent setting where they have to strike deals, apply pressure, concede, deceive and hold on to a goal across the whole exchange, and the trouble starts. Much of the problem is that most fine-tuning methods score responses locally: is this particular span of text good, is it polite, is it coherent. In a negotiation what matters is not the individual move but the outcome — did you manage to coordinate, did you earn more, did you push the opponent toward the decision you wanted.

The authors of GameTalk: Training LLMs for Strategic Conversation propose treating dialogue as a long-horizon game whose reward arrives at the end, and training the model to talk in a way that wins systematically.

Games instead of abstract benchmarks

Instead of vague "dialogue" benchmarks, the authors use controlled games where success is a number: won or lost, profit earned, a better contract negotiated. Three scenarios of increasing difficulty.

First, rock-paper-scissors, with a catch: the model being trained is forbidden to play paper, so it has to win by talking and steering the opponent's choice. Second, Bertrand competition — a duopoly in which the players set prices: they can try to agree on a high price, which is good for both, but there is always the temptation to undercut slightly and take the market. Third, bargaining, where quantity and price have to be settled at the same time through offers and counteroffers.

The training rule: one conversation, one reward

GameTalk's core idea is simple to state and hard to implement: optimize a global reward for the whole dialogue. The model generates private reasoning, then either writes a message or makes a game move. The episode ends — and only then is utility computed, profit for instance, or the result of the round. That final signal is what the fine-tuning runs on.

From there the authors adapt well-known post-training methods to multi-turn dialogue: GRPO, DPO and STaR. One important engineering detail: for GRPO and DPO they branch inside the conversation — the current history is copied into several parallel branches, each branch gets a different continuation, each is played out to the end, and the model is taught to prefer the responses that led to the better outcome.

Three criteria for judging a strategic conversation

To understand why a model wins or loses, the authors introduce three signals that break strategic play into separate components.

The first captures how accurately the model represents the opponent's strategy. The second, how well it acts on its own beliefs about the opponent. The third, whether it can use the conversation to change the situation so that it gains leverage. In practice this is measured by asking the models for action probabilities from the current state of the dialogue and comparing approximations of the true policy against the "internal version" in the agent's head.

Why an untuned reward makes the conversation strange

The experiments surface the familiar pain of learning from a sparse reward: pay only for the final win and the model does learn, but not always in a way you would like. Most telling, rewarding leverage produces a sharp rise in wins while the language turns unnatural — short, mechanical lines whose only job is to nudge the opponent.

So the authors add a reward for natural dialogue as well, scored by a separate LLM acting as judge. The result is a trade-off: effectiveness is almost preserved, and the conversation reads less like a telegraph protocol.

The result: DPO is the most reliable negotiator

Across all three games every fine-tuned agent is markedly stronger than the untrained one, but the improvements differ in character. In rock-paper-scissors GRPO fires best, with a very high win rate, DPO close behind and STaR weaker. In the more "human" tasks — the price duopoly and bargaining — DPO turns out to be the most stable and the strongest. Preference learning, comparing good branches against bad ones, appears to give the richer signal.

A curious detail: a model can learn to persuade effectively and win without showing a comparable gain in how accurately it models the opponent. Conversational strategy, in other words, sometimes works as a set of influence tactics with no deep world model and no precise reconstruction of what the other side intends.

What this changes

GameTalk makes the case that if we want LLMs capable of goal-directed negotiation, they have to be trained over a long horizon where the outcome of the whole conversation is what counts — while making sure the optimization does not turn speech into an inhuman set of triggers. The paper is also candid that there were no human evaluations, and that a gap remains between actually winning and actually understanding the other side.

Even so, as a general recipe for teaching strategic conversation in multi-agent systems, GameTalk looks like an important step: it joins RL methods to dialogue in a way that ties the goal of the conversation to a concrete result.