Brent Harrison, assistant professor of computer science at the University of Kentucky, received bachelor’s degrees in English and computer science before earning master’s and doctoral degrees in computer science from North Carolina State University. He spent two years as a post-doctoral researcher in Georgia Tech University’s Entertainment Intelligence Lab, where he and a graduate student named Upol Ehsan led a team of researchers interested in creating explainable AI systems. Their system provides “rationales”—reasons that describe why an action was taken in language ordinary human beings can understand.
Using an adapted version of the 80s video game Frogger, the team had users play the game and verbalize rationales for why they made certain movements. The AI agent received this data and, while also playing the game, learned to supply rationales for its movements based on how humans justified their own actions. When human participants were asked to choose between human rationales and these artificial rationales, participants preferred human rationales. The AI responses were not far behind, though. Harrison calls that a huge win.
Harrison arrived at UK in 2017, and Ehsan began a Ph.D. program at Cornell University, but the work has continued. Their most recent work resulted in a paper that was accepted to ACM IUI—the premier international forum for intelligent user interfaces research—in Los Angeles and presented in March. Since the conference, the team’s work has appeared in articles by ScienceMag, Motherboard and other outlets. We sat down with Harrison dig further into explainable AI systems and where they may take us in the future.
Why are explainable AI systems important? Do we really need to know a machine’s rationales?
I think it’s essential that we bring technology up to human standards, rather than forcing humans to adapt their ways of thinking to fit the AI system. The problem with machine learning systems is that it can be hard to determine why the system did what it did. Why did it choose this label or this caption—especially if it’s wrong? We’re inquisitive people, and we want to know why, even if it’s not wrong. Maybe it made a correct choice, but not the one we expected. Explainable AI seeks to make these systems more transparent.
What makes explainable AI systems difficult to create?
Mainly, if you don’t have a higher degree in computer science, it can be hard to reason about these things. Early attempts to make things explainable would often generate explanations that computer scientists could understand, but they still had this problem where it required the average person to come up to the level of the machine, whereas we wanted to bring the machine up to the level of people. So we asked if we could teach a machine to reason about its own behavior in human terms. We collected examples of people talking about their own actions, and then we trained a machine learning system to recreate that. So whenever it takes an action in the world, it takes its action according to its own processes, but then it thinks about that action in terms of human decision-making and how people would justify that action.
How did Frogger come into the picture?
We picked Frogger for a couple of reasons. First, it’s easy to understand. We could put it in front of people, and even if they had never played it, we could easily explain what to do. Apparently, there is a natural human instinct to “get to the other side.” But also, there are a surprising number of complex decisions that need to be made in the game.
The machine can give rationales for past actions; can it supply rationales for future actions?
Yes, but one of the limitations is that it can only explicitly look one step into the future. So it can say, “Okay, I know where I am. I know the things I could possibly do, and this is the action I want to take.” And then it would reason, based on its understanding of how a human would think about it, why it wants to take that action. It can’t say, “In four moves, I’m going to go left, and this is the reason why.” It can’t explicitly reason about that. But one of my favorite things that came out of this is that we would see agents mimicking long-term decision-making. The agent would jump back from a log and the rationale might be, “I jumped back because the three logs in front of me were not aligned, and I’m going to wait for them to align so I can jump all the way to the end.” That was unexpected, because we’re not giving it trajectories of people’s games, but people’s actions. But enough people reasoned in a long-term way that the machine learned to mimic it. I use the term mimic because it’s not itself making these kinds of inferences, it just learns what people do in situations and does it too. “This is the explanation human players would generate, so this is the explanation I’m going to generate.”
One of your research areas is computational storytelling. How does that relate to this work?
I’m currently working on a “reverse” version of this project, which gets at the understanding side of it. If you can tell a story, we believe you understand a little bit of it. So the process in reverse would be: What if the agent receives human language telling it what it should be doing. How can it ground that into the world it’s in? So if a person tells the agent, “Move up when a car is to the left of you,” can it use that information? If we can do this, then we have a limited instance of story understanding. Now people can provide advice in a natural way, in the way they’re used to giving advice to people, and an agent can learn more effectively from it. Again, it’s about bringing agents up to people, being able to understand the directions they’re being given so you aren’t yelling at Siri because it doesn’t understand how to help you make an appointment.
Where do you see this going?
In this initial study, we really wanted to understand what makes for a good explanation. Very few people have sat back and thought, “As a human, what do I find compelling in an explanation?” So we were trying to unpack that with the rationales that our system is generating. Can we inspire higher levels of trust and confidence in a system because we’re explaining things in a human-like way? Can this be used in an actual system with higher stakes? Say you and an AI agent have to build a chair together. Can explainable AI be used to foster trust, fluency and task effectiveness? Fluency is a term you hear a lot with regard to teams. Highly fluent teams are those that work well together, and are more efficient at finishing tasks. If you can trust your teammate, then you can work well together. If you have confidence in your teammate’s ability to do his or her job, then you won’t be tempted to break in and start doing their job for them, which eliminates all of the potential benefits you would have gotten from them. Overall, that’s really where we want to go with this.
You once told us that you want to create Iron Man’s J.A.R.V.I.S AI system, but for real. Still think it’s possible?
I have stopped questioning what’s achievable. I’ve seen things happen that even five years ago I would have said, “No, we can’t do that.” Look at all of the stuff that’s happening with smart homes. Why can’t we get there in our lifetime?
