The Disparate Effects of Strategic Manipulation

When consequential decisions are informed by algorithmic input, individuals may feel com-pelled to alter their behavior in order to gain a system’s approval. Models of agent responsive-ness, termed ”strategic manipulation,” analyze the interaction between a learner and agentsin a world where all agents are equally able to manipulate their features in an attempt to“trick” a published classifier. In cases of real world classification, however, an agent’s abilityto adapt to an algorithm is not simply a function of her personal interest in receiving a pos-itive classification, but is bound up in a complex web of social factors that affect her abilityto pursue certain action responses. In this paper, we adapt models of strategic manipulationto capture dynamics that may arise in a setting of social inequality wherein candidate groupsface different costs to manipulation. We find that whenever one group’s costs are higher thanthe other’s, the learner’s equilibrium strategy exhibits an inequality-reinforcing phenomenonwherein the learner erroneously admits some members of the advantaged group, while erro-neously excluding some members of the disadvantaged group. We also consider the effects ofinterventions in which a learner subsidizes members of the disadvantaged group, lowering theircosts in order to improve her own classification performance. Here we encounter a paradoxicalresult: there exist cases in which providing a subsidy improves only the learner’s utility whileactually making both candidate groups worse-off—even the group receiving the subsidy. Ourresults reveal the potentially adverse social ramifications of deploying tools that attempt toevaluate an individual’s “quality” when agents’ capacities to adaptively respond diff