Human Oversight Not Good Enough For AI War Machines

In the military arena, the UK government recognized the importance of human oversight in its February 2024 response to a parliamentary report on AI in weapon systems . The report emphasizes“meaningful human control” through the provision of appropriate training for humans. It also stresses the notion of human accountability and says that decision-making in actions by, for instance, armed aerial drones cannot be shifted to machines.

This principle has largely been kept in place so far. Military drones are currently controlled by human operators and their chain of command, who are responsible for actions taken by an armed aircraft. However, AI has the potential to make drones and the computer systems they use more capable and autonomous.

This includes their target acquisition systems. In these systems, software guided by AI would select and lock onto enemy combatants, allowing the human to approve a strike on them with weapons.

While not thought to be widespread just yet, the war in Gaza appears to have demonstrated how such technology is already being used. The Israeli-Palestinian publication +972 magazine reported described a system called Lavender being used by Israel.

This is reportedly an AI-based target recommendation system coupled with other automated systems, that tracks the geographical location of the identified targets.

In 2017, the US military conceived of a project, named Maven , with the goal of integrating AI into weapons systems. Over the years, it has evolved into a target acquisition system. Reportedly, it has greatly increased the efficiency of the target recommendation process for weapons platforms.

In line with recommendations from academic work on AI ethics, there is a human in place to oversee the outputs of the target acquisition mechanisms as a critical part of the decision-making loop.

Nonetheless, work on the psychology of how humans work with computers raises important issues to consider. In a 2006 peer-reviewed paper , the US academic Mary Cummings summarized how humans can end up putting excessive trust in machine systems and their conclusions – a phenomenon known as automation bias .

This has the potential to interfere with the human role as a check on automated decision-making if operators are less likely to question a machine's conclusions.

Sign up for one of our free newsletters

• The Daily Report Start your day right with Asia Times' top stories
• AT Weekly Report A weekly roundup of Asia Times' most-read stories

In another study published in 1992, the researchers Batya Friedman and Peter Kahn argued that humans' sense of moral agency can be diminished when working with computer systems, to the extent that they consider themselves unaccountable for consequences that arise. Indeed, the paper explains that people can even start to attribute a sense of agency to the computer systems themselves.

Given these tendencies, it would be prudent to consider whether placing excessive trust in computer systems, along with the potential for the erosion of humans' sense of moral agency, could also affect target acquisition systems. After all, margins of error, while statistically small on paper, take on a horrifying dimension when we consider the potential impact on human lives.

The various resolutions, agreements and legislation on AI help provide assurances that humans will act as an important check on AI. However, it's important to ask whether, after long periods in the role, a disconnect could occur whereby human operators start to see real people as items on a screen.

Mark Tsagas is senior lecturer in law, cybercrime & AI ethics, University of East London

This article is republished from The Conversation under a Creative Commons license. Read the original article .

Already have an account? Sign i Sign up here to comment on Asia Times stories OR

Thank you for registering!

An account was already registered with this email. Please check your inbox for an authentication link.