Automating government: algorithmic dehumanisation
High-profile scandals involving the use of digital tools by public authorities reveal the scale and seriousness of the hardship and injustice that such capabilities can produce in the hands of government. These range from sophisticated algorithmic systems to automate the administration of welfare benefits, such as the the ‘robo-debt’ scandal in Australia and the Dutch childcare benefits scandal, through to errant financial reporting in the Horizon software system used by the UK Post Office. The resulting scandal has been described as “the most widespread miscarriage of justice in UK history” with more than 700 innocent post office operators criminally prosecuted for crimes they did not commit thanks to flawed accounting software installed in the late 1990s.
These failures share a common pattern: a public organisation that routinely makes decisions that affect the lives of citizens, and which entail a high volume of transactions, turns to digital automation in the quest for efficiency. But the way in which individuals are ‘seen’ through the traces of their digital interactions fails to account for the messy, lived reality of human experience.
Meanwhile, these digital systems are intentionally configured to produce favourable outcomes that best serve the needs and interests of the public sector organisation. Default responsibility for mistakes is routinely placed on the individual who must demonstrate that an error has been made and persuade a public official to correct it. Designed to ‘optimise’ processes for public sector organisation, these systems off-load the financial, emotional and health consequences onto affected individuals, typically without explanation or meaningful opportunities for recourse.
They show how the turn to digital machines to undertake governmental tasks can transform how public authority is exercised, distributed and experienced in ways that entail the unlawful and arbitrary exercise of public power, with devastating impacts on people’s lives, Thanks to the power of networked automation, they can produce injustice at scale without meaningful transparency, entailing the collective violation of the rights of affected individuals, including the presumption of innocence and the right to a fair hearing.
For those in positions of vulnerability who lack the skills, competencies and ready internet access, and whose encounters with the state are now mediated primarily via digital systems rather than frontline human officers, their experience of the state has become increasingly and systematically and Kafkaesque, dehumanising and unjust.
