Lost in translation: the troubling logics underpinning the embrace of governmental machine-learning based prediction tools for ‘citizen scoring’

Machine learning (ML) applications referred to as ‘predictive analytics’ or ‘big data analytics, now ubiquitous in retail, entertainment and logistics, are increasingly common in public sector contexts too. Applications include algorithms that claim to estimate an individual’s ‘risk’ of specific behaviours, such as an offender’s likelihood to reoffend or the probability that a child will be subject to abuse or neglect. This chapter asserts that the embrace of these data-driven ‘citizen scoring’ systems is underpinned by a set of promises, assumptions, beliefs and rationalities (collectively referred to as ‘logics’) that seek to replicate the success of ML in commercial contexts into the public sector with no regard for the fundamental differences between the two contexts. I critique three specific claims that have encouraged the adoption of commercial ML techniques by the state: (a) that ML produces more accurate predictions (b) that these predictions offer valuable ‘actionable insight’ for public authorities, and (c) that ‘early intervention’ based on such actionable insight is desirable. I argue that although it may be legitimate for profit-seeking firms to use probabilistic estimates derived from algorithms to inform low-stakes decisions (such as identifying which web-ads to display to users to encourage more clicks), far more significant state interventions such as denying the early release of a prisoner due to their perceived risk of reoffending or taking a child into care identified as ‘at risk’, cannot be justified on the same terms. Yet, thanks to the uncritical adoption of commercial ML methods in the public sector, power and authority are being illegitimately, and sometimes unlawfully, redistributed in ways that produce injustice without public awareness or democratic debate, to the detriment of some of the most vulnerable members of society.