AI & Ethics
28:55 Share

Welcome to Stories of Impact. I'm your host, writer Tavia Gilbert, and along with journalist Richard Sergei, every first and third Tuesday of the month, we share conversations about the art and science of human flourishing.

Today, we're back in conversation with a team of researchers, including a philosopher, a neuroscientist, and a computer scientist. This trio might be familiar to longtime listeners from an episode a few years ago, when we explored the question of whether artificial intelligence could be programmed to be moral.

Over the last five years, this research team has studied how to use AI as an assistive tool in allocating kidneys to patients needing organ transfers. It's a project partly funded by OpenAI, the company behind ChatGPT.

Though the project is focused on integrating AI into evaluations of who gets a kidney transplant, the team is using that particular problem as the lens to explore more broadly the ethics of AI in decision-making.

They're asking whether it's possible to imbue machines with a human value system, in what ways artificial intelligence can be employed to help humans make moral decisions, and how to ensure that when AI is involved in decision-making, the process retains humanity. Let's start with the project philosopher, Dr. Walter Sinnott Armstrong, who shares the goal of the research. We want to build a moral GPS system

The GPS on my car constantly keeps me from making wrong turns and warns me when there's a slowdown in the traffic ahead so I can take a side route. Well, I would love to have an AI system that helped me in my moral life.

But also, like a moral GPS, it's going to be restricted. It's not going to cover every moral issue throughout your life. So we've been focusing on kidneys and transplant of kidneys when there's a shortage as one particular example. Dr. Synod Armstrong is the Chauncey Stillman Professor of Practical Ethics in the Department of Philosophy and the Kenan Institute for Ethics at Duke University.

Amidst a nationwide kidney shortage, he wants to assist humans in navigating moral dilemmas around who should receive a transplant. One of his project partners is neuroscientist Dr. Jana Scheich-Borg, associate research professor in the Social Science Research Institute at Duke University. She says, A lot of the features that most people agree should be taken into account morally in kidney exchanges currently are not taken into account.

So there are AI systems that are being used in different places to help allocate kidneys when they become available because humans are too slow and the kidneys are not available for very long and you have to do that quickly. But they're not currently taking into account features like how many dependents

each possible patient has or what their prior sometimes at least some of the algorithms don't take into account their other health or behaviors that they may have engaged in like alcohol abuse or drug abuse that may have impacted their health or why they've ended up in the health condition that they currently have.

Now, different people may think that that's irrelevant and some people may think it's not relevant, but at least in our studies, most people do think those things are relevant to some degree. So what our system allows us to do is incorporate those features into AIs that exist already so that in an automated, fast way, the kidneys can be given to people who need them while still taking these features into account and also a way that we understand that we can interpret rather than in an ad hoc fashion.

When allocating transplants, artificial intelligence has two important advantages, explains Dr. Vincent Konitzer. He's a professor of computer science at Carnegie Mellon University and the head of technical AI engagements at Oxford's Institute for Ethics and AI. He says that AI can produce solutions from a lot of variables, and it can do so very quickly.

Why are we using, in a kidney exchange, AI at all? Why not just have humans do this? And one of the reasons is that, so this is different from just being on the list to wait for a kidney transplant. This is more situations where maybe a patient has a willing donor that they're not compatible with.

And here algorithms from the AI literature are being used to figure out who best to match together. So you can maybe swap your donor with somebody else and that way you both get a kidney, but who should you swap with? Right. And how can we match? You could ask, for example, how can you match the maximum number of patients, make sure that as many people get a transplant as possible.

Now, it's not clear that that's the right objective to pursue, and that's precisely what we're looking at. But even if you ask that, it's a very difficult question to answer computationally because we have so many patients and donors that you can try some things as a human being and say, "What if I match this patient with this donor and this one with this donor?" You might find something somewhat reasonable,

But probably a lot of the potential is left on the table because no human being can sort through all the possible ways of matching patients and donors together. That's really where the AI comes in and has a strength relative to us that it can sort through all these possibilities. But in order to do so, it needs to know what it's looking for.

Dr. Conater emphasizes, though, that we can't just turn over all decision-making to AI. So if you don't want a simplistic objective like just maximize the number of transplants that take place, but rather want to take morality into account and think about how to prioritize, then that needs to come from the human being somehow.

The symbiosis of AI and human decision-making has prompted several intriguing questions, says Dr. Scheich-Borg. How do we make sure that AI is used ethically in society and what needs to be done to make sure that's the case? One aspect of that is making and creating and figuring out what a moral GPS might look like. But I think there's other things that we have worked on that I really find exciting, which is how could AI help us in our decision-making, our

So you could just follow a GPS, but are there ways AI could actually help our decision-making process and help us to become better decision-makers? And can AI also just help us understand the human moral brain when it makes moral judgments? But one central question must first be considered before any other. Can AI have moral agency?

That's not easy to answer, says Dr. Conitzer. To what extent do we really believe that these systems reason themselves? Especially if we look at something like chat GPT these days, it does a lot of apparent reasoning that is very impressive. But is it really the same kind of reasoning that we're doing? To what extent is it copying this from us versus really having understood this to some degree itself? And you could ask the same kind of question for morality as well.

What influences human morality? Don't we have different values depending on where we are from or where we live? Not as much as one might believe, says Dr. Sinnott-Armstrong. So morality does vary to some extent with geographic location or community, even at the same geographic location. But I think people often exaggerate how much difference there is.

You know, people around the world think that they should take care of their children. People around the world think, you know, it's wrong to murder so that you can steal somebody's money. There are some moral judgments that are shared quite widely. One study looked at a particular pair of cases and found that 200,000 people in 120 different countries all drew the same distinction.

and ended up on the same side. Now, they weren't universal in the sense of 100% of each country, but the percentages within each country were quite close to each other. So I think there's a lot more agreement than people recognize. People emphasize the disagreements because they're so interesting. But the similarities are also interesting and important because they're what enable us to work and live together.

Now, how does a computer deal with that? The cases where they're controversial, then the computer is going to say, well, that's controversial. I can't tell you what humans in their idealized state would say about that one because different people would say different things.

But they can tell us in these others that most humans, almost all humans in idealized states would agree. So like most topics, morality is a grab bag of many different things. There's going to be some disagreement and some agreement. We need to figure out where those disagreements lie and computers can help us do that.

While moral values may be largely agreed upon across cultures, morality is constantly in flux, says Dr. Scheichborg. We often like to think of human morality as kind of a static thing. Once you learn what you need to learn,

You make a moral judgment and that's your moral judgment and it's the right one. This project has really made it clear to me that our moral judgments are dynamic. They change. We're always morally learning. So even if we make a decision and we say our decision is the same one, we may be using different strategies every time we make that decision. And I think it's very hard to always be able to detect what strategies we're using. We make a lot of assumptions.

And the moral psychology field, moral neuroscience field has had this problem for a long time, is what we say we're doing may not be what we're actually doing. Different contexts can push us into making our moral judgments in one way versus another way. And that we change our opinions. We change our opinions some ways in ways that we would be happy with and that we think are valid. And sometimes we make them and change them in ways that we might not be happy with when we introspect. So how is the team programming morality into AI?

Dr. Sinnott-Armstrong says it actually starts with improving human decision-making. Our project doesn't impose any particular definition of morality on other people. Instead, what it looks at is

the judgments that humans would make if they were better informed, more impartial, and didn't get confused and draw bad and irrational inferences and make all the kinds of mistakes that humans are prone to.

So I would say the morality that we're trying to build into computers is an idealized human morality. So it's not that humans define what morality is. It's that humans, when they're at their best, would define what morality is. Dr. Conitzer adds that bringing AI into the decision-making process offers humans a feedback loop

We think that the AI can make moral decisions reasonably well on its own in some sense, but it has to learn from us. And in turn, it can actually help us make better moral decisions ourselves. Dr. Scheich-Borg agrees that the point here is not for AI to take over all decision-making. Rather, it should help us.

It's made us think a little bit more deeply about how human morality works anyway on a day-to-day basis outside of interacting with AI. We don't need a computer, but it provides some new opportunities, especially to help us scale ways to learn what our own moral values are. And there are ways to use AI that can help us provide opportunities to everyone to figure out what their values are.

So that to me is both exciting and feels like a moral obligation in many ways, especially in today's world where everything's moving so quickly and so many different types of technology can have big moral impacts. Dr. Synod Armstrong explains further. I must admit, when I make those decisions myself right now, I'm not making them on full information. I'm going with my gut, and I'm not sure how much I trust my gut, but I'm

It's the best I've got. I wish I had a moral GPS that would be more reliable than my gut. It's not that there's no value in human intuition, but humans are imperfect, says Dr. Synod Armstrong. We have lots of capabilities and can do great things, but quite often we don't know the facts. We're ignorant of things that really matter to the moral issues. Quite often we get confused by a very complex world.

moral issue. Quite often, we have biases that lead us to behave unfairly in ways that we ourselves wish we didn't do. And so because of human imperfections, we got to figure out a way to do better

And AI can have more information than we have, can deal with complexity better than humans do, and if programmed properly, can reduce the amount of bias in the decisions and judgments that are made. So we need computers to help us overcome human limitations, which is not to say the computer operates without us, but that it helps us.

Computers already do all sorts of reasoning, notes Dr. Konitzer. Ethical reasoning by AI is therefore a rational next step. In other domains, if we're thinking about logical reasoning, we have good formal frameworks for how to do that, similar for probabilistic reasoning. It can probably think better about probabilities than most humans can.

But for ethical reasoning, we don't really have a good general formal framework. This is often one of the lessons that you get from an intro ethics course is that any simple formal framework that you come up with is going to have these examples that what it would recommend is really not what you want. And I think that's a really interesting challenge for AI and also why fundamentally humans need to be somewhere in this loop.

While they see the benefits of AI being included in complex moral questions like kidney allocation, the team is acutely aware that there are risks

Part of what we've learned over the past couple of years is how easy it is for us to introduce bias. And even this system that we think compared to other systems might be relatively resilient to bias. But every choice we make and how we try to elicit people's thoughts and opinions can impact both the results we get from them, but also perhaps even what they think.

Getting reliable data has been part of the difficulty, says Dr. Sinnott-Armstrong. These problems are hard. There's this tendency to go, oh, yeah, we can, you know, take the relevant features and just program them on a computer, and then we have to predict these humans, and we'll correct for this and correct for that, and then it'll be idealized, and that sounds like a great program. But when you actually start to implement it,

You run into difficulties of all sorts that we didn't expect. So one that we've been dealing with recently is instability. We found that our subjects would give different answers on Tuesday than they had given on Monday and different answers on Wednesday than Tuesday. Not every subject, not for every type of scenario, but some of them did.

And then the question is, what does that show you about their values? There's this tendency to think, yeah, somebody what they think, and they're going to tell you what they really think. And you ask them what they value, and they're going to tell you what they really value. But it turns out that that can't be right if they're not stable. And so those are some of the problems that we've been dealing with recently, how to incorporate that into a predictive algorithm.

Another challenge is that AI is so polarizing. While many see it as having unlimited upside, others fear it will doom humanity. The team considers the potential consequences of the technology in their book, Moral AI and How We Get There, says Dr. Conitzer.

I don't think our kidney exchange algorithms have any chance whatsoever of taking over the world. So let's start with that. But there are so many questions being raised by this technology, right? Even when it's not deliberately being used for disinformation, it's just going to get stuff wrong, right? And we've seen examples of that, that people rely too much on the technology. And when that causes real world harm, who is responsible for that? So, you know, with our narrow technical research, we're not likely to be able to address all these issues, though I think a lot of it, it has something interesting to say about

about, but there is this broader field. And this is also what motivated us to write this Moral AI book to allow us to get a little bit broader and consider all these different aspects of how AI is going to impact society.

Now, in terms of the more doom type of scenarios of AI taking over the world, I think it's good that some people think about that, right? And it's very interesting to see how these technologies are being trained to avoid those kinds of behavior. This is often the idea of fine-tuning, that there's some special training stage where they're trying to get it to behave better in various ways. And increasingly, it's also just prompting.

Just giving it the instructions that say, hey, try to make your response helpful and not harmful. Dr. Conitzer has been impressed with the progress in AI research over the past five to ten years in terms of making AI systems safer and better behaved. And on one hand, that's fascinating and maybe also gives us some hope that we can actually do this.

On the other hand, we also know that this is very brittle. Lots of people try to jailbreak this technology to try to get it to do things that it wasn't supposed to do. And there are really remarkable ways of doing that. If you play around with it, sometimes you can find one of these jailbreaks yourself, at least in a limited context, to try to get it to output something that it wasn't supposed to do. But there are also much more systematic ways of doing this.

So I don't think we really understand how to keep these systems safe and behaving in a morally good way at this point. And there's a lot of research to be done there yet. And I hope that that research to try to keep these technologies safe ends up keeping up with just the quick progress

increasing capabilities that these systems have as they're being trained on ever more data and ever larger models. And that somehow remarkably keeps introducing new abilities into these systems while also still leaving it brittle in some ways and having weird failure modes, right? If you talk enough with these systems,

you will get interactions where you would say, if that had been a human being, something is very seriously and very strangely wrong with that human being in a way that I don't think actual real human beings ever act. So for now, we have not passed the Turing test, but we do have this very powerful in some ways and brittle in other ways technology that's really going to impact us. Dr. Scheichborg agrees that the model they're building doesn't risk world domination.

But, she says, even if AI hasn't yet passed the Turing test, that's the ability to exhibit intelligent behavior indistinguishable from a human, that doesn't mean we shouldn't treat it as if it one day could. The tools we're working on right now, it's hard to see how we would give it an objective that would lean to our kidney exchange algorithms taking over the world. It's hard to imagine a way that those systems are going to end up taking over the world.

But in the bigger picture, we are very aware and part of what we're trying to do is for the AI field as a whole, how can we make sure AI systems don't have unintended consequences? That's a different part of the project where we're trying to figure out what systems do we need to put in place? What kinds of processes do we need? What kinds of decision aids do we need? What kind of feedback do we need from people and from stakeholders?

And those are all things that we would want to use for our own AI systems that we're building, but also that need to be used for these broader systems. And there are a lot of unintended consequences that I think are foreseeable right now and that I'm particularly concerned about. One of those is I'm really concerned about the impact of chat GPT and these kind of emerging technologies that are supposed to stand in for humans on our own human connection.

our ability to connect with one another, but also the health impacts of that. We know that when we feel connected with other humans, it has dramatic impacts on our health. When we don't feel connected, pretty much every health indicator we have plummets.

It impacts our recovery from surgery. It impacts our mental health. It impacts how much we get cancer, pretty much everything. And so right now there's a lot of excitement and some of it well justified about how AI could be used to help loneliness as AI care bots, as friends, as assistants, as someone who keeps

company. And there may be many good uses for that, but I'm also feel like we're not thinking enough. Society is not thinking enough about what is some of the likely downsides of that, which is that it may take away from our ability to learn the skills that we need to interact with each other and to actually become deeply connected. Those are the types of things we're trying to figure out what systems can we put in place as a society to try to address.

The team has already discovered an unintended consequence with their own AI, explains Dr. Conitzer, with some people left with virtually no chance of getting a kidney. This finding prompted important questions, not only about their project, but about the future of AI.

What are really the right processes for doing this? And as we actually do something and we see what happens, it always ends up raising new questions, right? Very early on, we already saw that if we do this kind of prioritization in a kidney exchange, one of the effects is that certain patients are going to end up with basically zero chance of being matched anymore.

And maybe with hindsight, that's not such a surprising thing, right? That it's prioritizing. And so that's going to come at the cost of some people having a chance of getting a kidney. But you might look then at this system level effect and say, well, is that really what we want overall? And that I think is a difficult question that should get society involved in a broader way rather than us just writing a few research papers. It really requires this broader thinking about

What do we really want to see in society? What do we really want the processes to be by which we arrive at these systems? And that makes it more challenging. I think that's the right way to do it. But this is what we somehow have to figure out. And meanwhile, we have to somehow keep up with the technology that's racing ahead. So I think, you know, we need more people. We're not going to be able to do everything by ourselves. We need more people to get into this research area and work on it.

What do we do if the decisions AI starts to make aren't aligned with our morality? Dr. Scheich-Borg says, Some of the biggest challenges that I foresee that will be interesting

but also very hard, is to figure out what is the right way to train these so that we think we can have confidence they're doing what they should be doing. And that's both as creators, but also that users can feel confident that this GPS represents their true moral values or their true process. Now, the problem with something like that is we've talked about how the feedback from an AI or interaction with an AI can change our moral judgments. So, you know, what if people trust this AI

Trust that it has immoral values starts making predictions that are not actually consistent with what we would actually make. Will we be able to catch that? Will our moral psychology be set up in such a way that we will trust the AI too much? And also, cognitively, will we kind of get rusty in our own moral judgment? Will we be able to catch those things? Those types of challenges are going to be some of the biggest ones for us to figure out as we make this into a system in people's hands.

There is still plenty of work ahead before that can happen. There's really a big gap between a proof of concept, especially the type we researchers make, and what is actually needed to get into people's hands in a way that's having the impact you intend.

So I ended up calling this translational ethical AI and I think we need a lot more work in this space. We need a lot more to help us figure out how do we go from a paper, an academic paper that shows something should work, that we can build morality into a system in some type of way, into something that then an AI developer who has to do something in two weeks, create some type of prototype in two weeks because that's what his company is telling him to do or her company is telling her to do.

Something that they can use that will actually make sure that system has the right morals or has some type of moral values built into it. There's a lot of factors that go into that translation that are not really appreciated or rewarded a lot in academia. They're often not appreciated or rewarded in industry either. And so I think we need to figure out how to make sure there is more effort sometimes.

support and incentives in this space to get the things, the great ideas our group but many other groups are coming up with into something that's in society's hands. Another lesson actually that I think is underappreciated, we alluded to these things we said before of the way we ask people may change the judgment that they give us. And so I think this kind of whole paradigm of saying the way that we're going to make these systems work is just by asking people once

Whether or not this is a good or bad thing is really, we really need to revisit that. And we need to revisit that in a very rigorous way. As AI proliferates and enters more and more industries, Dr. Conitzer advises caution. How do we keep up with the technology? There are broader questions there, I think, for society as to what limits do we put on various kinds of developments. And for us, the question is, well, where do we fit in usefully into that? Do

Do we have a case where we say, okay, well, this is really where we want an AI system to make a decision. And this is how we're going to train it to do that well from moral and other angles, while also being aware of the fact that in principle, you could use these systems in other ways, but maybe we want to put some limits to that. But how do we think about that as a society and scope these things right?

The researchers do see a future where AI helps humans make better decisions. Dr. Synod Armstrong says: I'm optimistic that people will start working on it. Whether they will succeed remains to be seen because each problem has its own complications.

But I'm optimistic that 10 years from now, I won't say five, but 10 years from now, there will be other applications that could work and help people. So I guess I'm optimistic 10 years from now we'll have made a lot of progress on this. Will we have perfected it? Not a chance. Nothing in this world is perfect, and these systems won't be perfect either. The good news, says Dr. Scheichborg,

We still have control over what happens next. What happens with moral AI is up to humans right now. It's not up to AI. And so we need society to get involved and to feel empowered to get involved and to share their opinions and to vote for policymakers and policies that they think are important. And it shouldn't just be people in tech or academia or in government who are making those decisions. We need everyone to get involved. We'll be back in two weeks with another episode of

In the meantime, if you enjoy the stories we share with you on the podcast, please follow us, give us a five-star rating, and share this podcast with a friend. And be sure to sign up for the TWCF newsletter at templetonworldcharity.org. You can find us on Twitter, Instagram, and Facebook, and at storiesofimpact.org.

This has been the Stories of Impact podcast with interviews by Richard Sergei. Written and produced by TalkBox Productions and Tavia Gilbert. Senior producer Katie Flood. Assistant producer Oscar Falk. Music by Alexander Felipiak. Mix and master by Kayla Elrod. Executive producer Michelle Cobb. The Stories of Impact podcast is generously supported by Templeton World Charity Foundation.