ML//evaluation//performative prediction

2026-10-03

Performative prediction is the setting in which deploying a model changes the very data it predicts, and it is what has to be accounted for whenever a prediction is acted on: a credit score that changes who gets credit, a traffic forecast that changes the routes drivers take, a risk score that changes how a person is treated. Perdomo and colleagues formalised it in 2020 as a distribution of data that depends on the model deployed.

In the usual setting a model is trained on one distribution and tested on the same one, so accuracy measures how well it describes the world. Once predictions trigger actions, the world it is tested on is partly the world it produced. A model can then look accurate because it made itself right (a worker scored as inattentive, given worse tasks, who then performs worse) or look wrong because it worked (a warning that prevented the failure it predicted).

A good score after deployment does not validate the model. The test has to separate what the prediction described from what the action it triggered caused, which takes comparisons where the action is varied, never more accuracy on data from the same loop.

Retraining on deployed data can settle into a fixed point. Repeated retraining converges to a model that is stable under its own effects, which is not necessarily the model that would serve best; the formal results say when the loop converges and how far that point is from the optimum.

It is a feedback loop between a model and the people it describes, and the usual remedy for distributional shift (collect fresh data and retrain) is part of the loop instead of a way out of it.