Everyone's had that coworker, the one who never asks for help even when fully out of their depth, unaware of their own incompetence. But what happens when your colleague isn't a human suffering from Dunning-Kruger but artificial intelligence?
That's a question Vishal Chatrath has had to consider as the CEO and co-founder of Prowler.io, an AI platform for generalised decision making for businesses that aims to augment human work with machine learning. Prowler.io's customers include fund managers — the system decides what to buy and sell — as well as managing logistics and supporting education, with one client using the platform to develop an AI tutor. "The decision-making process can be quite similar [across different businesses], if abstracted at a low-enough level," he says. "In some cases, the decisions are fully automated, in some cases, there's a human in the loop.
Keeping a human as part of the process is partially because of a lack of trust in machine-based decision making, but it's also an admission by Chatrath that we remain in the early years of AI. Such systems aren't perfect, and likely never will be, and one failing of AI is it doesn't inherently understand its own competency. If a human worker needs help, they can ask for it — but how do you build an understanding of personal limitations into code? Chatrath points to the two deadly 737 Max crashes. "In both crashes, the commonality was that the autopilot did not understand its own incompetence," Chatrath says.
Prowler.io built an awareness of incompetence into its system, teaching its AI to not only understand its limitations but to forecast when it's going to reach a situation where it has no experience or background. "Then it gently taps the human on the shoulder, so to speak, for the human to take control," he says. The system can learn from those interactions, and after enough training may eventually be able to stop asking for help.
Such limits to AI could be placed by regulators, as is the case in the financial industry where levels of risk are carefully weighed, or by the business itself. When working with a new customer, Prowler.io considers four questions as it sets up the platform: when does the AI know for certain that it's right, when does it know it's wrong, and when does it know that it's about to go wrong — timing is key, so humans in the loop have time to react. The fourth consideration is how are we even sure the AI is asking the right questions. "There is no cookie cutter answer to these," he says.
Another consideration is risk. If there's a 10 per cent chance a logistics scheduler is wrong, and a lorry is therefore a bit late, that's okay. If there's a 10 per cent chance that shape in front of a driverless car is a human, the car should stop — the risk are too high for any uncertainty. "Rather than doing stupid things like running someone over, it brings the human into the [process]," Chatrath explains, as it's been told when the risks are too high for it to screw up.
That's important, says Taha Yasseri, a researcher at the Oxford Internet Institute and the Alan Turing Institute for Data Science, because while we can delegate decision making to machines, we can't delegate responsibility. "The ultimate responsibility in implementing the decisions made by machines are on us," he says. "This distinction is important. In practice, whenever the expected accuracy of a human is higher than a machine, it is practically justified to use human judgment to overlook machine decisions."
Plus, says Chatrath, it gives confidence to users and helps address issues of liability. "Unless we have a strong notion of competency awareness in AI, it won't become mainstream.... it gives confidence to humans," he says. "Because ultimately there is a liability element to the whole thing."
But adding a human to the loop doesn't solve all problems. What happens if incompetent AI is paired with an equally incompetent human? That's a challenge, as humans are prone to lean on the decisions spat out by computers rather than their own instincts. "We tend to rely on outsourcing (to machines or other humans) when we feel the responsibility is too heavy for us," says Yasseri. "There have been reports on mis-policing or mis-judgment in legal systems or decisions on loans and financial support in banking systems that involved outrageous mistakes."
Plus, humans working with or accountable for AI need to fully understand the systems they're working with. "We need to understand what's going on in the black box," says Sandra Watcher, also a researcher at Oxford Internet Institute and the Alan Turing Institute for Data Science. "Even if there's a human in the loop that has oversight, how valuable is the human if they don't understand what's going on?" That doesn't require being able to unpick the code behind an algorithm, but understanding what data point impacted a result in a certain way.
There's a growing movement to make AI explainable, so we can pick apart its decisions in case of bias or error. That's helpful for ensuring accountability – if we know how facial recognition spotted someone, we can better understand if and why it might be incorrect in its identification – but Chatrath notes that businesses demand correct decisions now, not after the fact. Understanding why your AI-driven fund manager lost money is less useful than preventing bad buys in the first place. "Explainable AI is not enough, you have to have trusted AI – and for that to happen, you need to have human decision making in the loop," he says.
Watcher argues that humans won't simply be overseeing AI, helping out a system when it admits it's incompetent. Instead, such systems and humans will work in tandem, helping each other with their blind spots. "It might be that a human doesn't catch something, but the algorithm does, or the other way around," she says. "I think having the idea that a human always has to overrule or an algorithm always has to overrule is not the right strategy. It really has to be focusing on what the human is good at and what the algorithm is good at, and combining those two things together. And that will actually make decision making better, more fair and more transparent." And perhaps then we can replace that annoying coworker who doesn't know when he's an idiot with a machine that does.
Vishal Chatrath is speaking at WIRED Pulse: AI at the Barbican on June 15, 2019. Find out more at wired.uk/ai-event
This article was originally published by WIRED UK
