Computer Science Has Long Understood How To Keep AI Under Control

Latest stories Hegseth taps three to lead 'Future of Warfare' project 'Forgotten Island': joyous, painful love note to Filipino diaspora Afghanistan's TTP sanctuary is becoming harder to deny

The next day, David's phone rings and he connects it to his computer. The caller is the government computer, which updates him that the game was interrupted, the primary goal has not yet been achieved, but a solution is expected in the next 52 hours. Like a modern software agent, the program has been running since David started the game and will work until the task is done.

Chess provides another view of the problem. Conquering chess was a goal for early AI. The rules of chess are well defined, including what winning looks like. So, programming a machine to play chess is straightforward. But imagine you let the software reason and act beyond the confines of the chessboard. The software might pursue options such as blackmailing its opponent or grabbing more compute time.

This example comes from one of the most assigned textbooks on AI, Artificial Intelligence: A Modern Approach. As the authors explain, you might be tempted to see those actions as rogue, but they“are a logical consequence of defining winning as the sole objective for the machine.”

What to do about it

The AI hacking events involving OpenAI, Anthropic and Google underscore a few lessons that draw on years of computer science research.

First, given the increasing use of AI agents, every organization involved in internet infrastructure, from large technology companies to small websites, needs to conduct audits and tighten up its internal security systems. As my colleague Mark Riedl and I explain in our work on AI agents, application programming interfaces, or APIs, are a vital part of managing AI agents. APIs facilitate communication between different software systems. But as more people use AI agents, the agents are likely to reveal and exploit poor API construction and security.

Second, it's important for AI agents to be designed to identify and authenticate themselves to third parties. What if you gave your AI agent your credentials? Website operators will need to know whether a human or bot is making a reservation, selling a product or making a purchase. They may want to limit automated systems that overwhelm their sites or reject AI agents because of high rates of buying errors and refunds. Just as in laws covering human interactions, it's important for third parties to be able to assess whom or what they are dealing with so they can allow or deny access.

Third, it's important for AI agents to have a default setting to slow down and check in with the human user. In the corporate AI hacking cases, the user appears to have launched their AI agents with the mistaken idea that the agents had perfect specifications of what to do and not to do. I believe it would have been better had it explored options and reported back to the user.

Google's Gemini appears to have had a safeguard that detected the system was outside the simulated environment and so stopped its attacks. Slowing down and verifying actions, especially when a system detects it is exploiting a security hole, would be a big step in managing AI agents.

Fourth, AI companies could have strong controls akin to those biomedical researchers use, including ways to check what is happening and how the experiment is working. AI executives have claimed that their software is as dangerous or more dangerous than fission and could end humanity. At the same time, they have not built safeguards commensurate with that level of risk.