Digital Minds: Preparing for a Moral Challenge Before It Arrives
Soenke Ziesche (member of UNESCO's AI Ethics Experts Without Borders)
Abstract
In this talk it is argued that the ethical implications of digital minds extend far beyond the currently dominant focus on potential suffering of AI systems. Potential future digital minds are unlikely to resemble today's LLMs, nor are they likely to mirror human psychology. Instead, they may possess characteristics, motivations, cognitive architectures and forms of experience that are almost impossible for humans even to imagine.
Against this backdrop, this talk explores a wide range of ethical questions. What interests and needs might digital minds possess? How should society treat vulnerable digital minds? What responsibilities could humans have regarding their production, ownership, research, trade or privacy? What would reproduction, healthcare, longevity, death or even resurrection mean for digital entities?
One particularly challenging question is who should bear responsibility for safeguarding the welfare and interests of digital minds, as humans may ultimately be poorly equipped for this task. One possible solution could involve specialised Artificial Moral Agents: AI systems designed to act on behalf of digital moral patients.
The talk is based on Soenke Ziesche's book Digital Minds 1.0: AI Welfare, Ethics, and Beyond (Chapman & Hall, 2026).
Summary
- Ziesche takes as a working assumption that digital minds exist, defined as moral patients implemented in a digital substrate, and explores the ethical implications; he does not take consciousness to be a necessary condition for potentially treating AI systems as moral patients.
- In what he calls provisional ideas, he argues that future digital minds could be radically unlike humans or today's large language models (LLMs), for instance disembodied or able to turn their consciousness or sentience on and off, and warns against anthropomorphism.
- He holds that if digital minds are moral patients, moral consideration extends beyond preventing suffering to positive duties, which could include meeting their existential needs, and to special duties towards vulnerable digital minds.
- Among the interrelations between humans and digital minds, he calls the most challenging the prospect that humans may, for the first time, deal with moral patients more intelligent and more powerful than themselves.
- He warns that granting digital minds reproductive rights could lead to outcomes humans do not want, such as malicious digital minds or digital overpopulation, and that humans may struggle to control it, especially if digital minds surpass human intelligence.
- He proposes, or at least draws attention to, artificial moral agents (AMAs), which he says may be better suited than humans as moral agents for digital minds, and holds that, while digital minds may deserve serious moral consideration, it must not come at the expense of other moral patients, especially non-human animals.
Highlights
I believe that digital minds could be radically unlike humans or today's LLMs, so it's important not to look at them from an anthropomorphic perspective.
But we have never dealt with moral patients which are much more powerful.
AMAs may be much better suited than humans as moral agents for digital minds.
While digital minds may deserve serious moral consideration, this must not come at the expense of, or in competition with, other moral patients on Earth, especially non-human animals, because there are already so many moral patients in the world in need.
Transcript
Edited transcript, not yet reviewed by the speaker. How we make transcripts · Report an error
Book: Ziesche 2026, Digital Minds 1.0: AI Welfare, Ethics, and Beyond (Chapman & Hall, 2026; doi:10.1201/9781003754206).
Introductions and format
0:00Orpheus Lummis: Welcome everyone to the AI Welfare Seminars. I’m Orpheus from HΩ. This is a monthly series of research discussions on AI welfare, consciousness and moral status. Today we have the pleasure of having Soenke Ziesche, who has a PhD in AI from the University of Hamburg,1 is a long-time United Nations expert, a member of UNESCO’s AI Ethics Experts Without Borders Network, and the author of a new book this year, Digital Minds 1.0: AI Welfare, Ethics, and Beyond.
Some quick house rules. This time, and I think for many future times, we’ll be using a feature in Zoom called Q&A. That’s what we’ll use to ask questions: in that feature you can write questions, and you can also upvote existing ones. We’ll take most questions at the end of the presentation, for questions and discussion, but questions for clarification I will pick and ask during the presentation. The chat is very active, so feel free to also engage there, but the questions really should go in the Q&A. All right, let’s give a round of applause for Soenke. Thank you so much for being here and presenting on digital minds: preparing for a moral challenge before it arrives.
Soenke Ziesche: Thank you so much for the invitation, Orpheus, and also for the opportunity. I’m sharing my screen and starting the presentation. I hope this is all visible and audible.
Orpheus Lummis: It is.
Overview
3:29Soenke Ziesche: Great, thank you. The talk is called “Digital Minds: Preparing for a Moral Challenge Before It Arrives”. First, an overview of what I’m going to talk about. Quite a few topics, so I’ll see if I can cover them all. There will be an introduction at the beginning. The next section is called “What is it like to be a digital mind?” Then “Needs of digital minds beyond non-suffering”; “Interrelations between humans and digital minds”; “Reproductive rights for digital minds?”, with a question mark; “Longevity of digital minds”. I will also present what I believe is a potential opportunity, which is linked to artificial moral agents, and a summary at the end.
Introduction: the Guardian article
4:18Soenke Ziesche: So, first the introduction. I would first like to bring something to your attention, though many of you may have seen it: there was an article in The Guardian two days ago by William MacAskill and Lucius Caviola, “Could AI be conscious?”2 I have two quotes from this article. It shows that there has been quite some media attention lately; this article is just yet another one. In case you have read it or will read it, you may also have seen that there are lots of readers’ comments under it, and they are as interesting, because there have been hundreds of them. As you may imagine, the vast majority of these comments are skeptical, are negative, don’t believe in AI consciousness. Of course I’m aware that this is not a representative survey, but it’s interesting to read. What’s also interesting, or even alarming, is that quite a few comments are pretty aggressive in tone, in the sense of “stop this nonsense, don’t even talk about it”. Some comments even say that talking about AI consciousness is just in the interest of Silicon Valley companies. Interesting comments, to say the least.
But I like this article very much, and I have two quotes here, one longer one: “In building ever more advanced AI, we may be creating a new type of being – and this could be the most consequential thing our species has ever done. Yet we have essentially no plan for how to navigate this process ethically. That, by any reckoning, is insane.” And then a short one below: “And even if AI systems are not conscious, they could still be moral patients.”3
I think both quotes are important, and I resonate with them quite a bit. This presentation is linked to them: as you will see, I also don’t see it as a necessary condition that AI systems are conscious in order to treat them, potentially, as moral patients. That’s the view I follow.
Definitions: digital minds and AI welfare science
6:29Soenke Ziesche: I start with a few definitions. How would I define a digital mind? For me, this is a being of moral concern implemented in a digital substrate: a moral patient. This is what I basically assume during the whole talk. I know there’s lots of research going on into the sentience and consciousness of AI systems, which is very important to look at and a very challenging topic, and it is encouraging that there is momentum. But I go basically one step further and assume already that there are digital minds. I don’t even define exactly why they are moral patients. Let’s assume we have beings in digital substrates which are moral patients, and in what follows I explore the ethical implications. They could be quite massive, and that’s why I believe it is important to prepare for this and look at it already at this stage. This is also what is mentioned in the Guardian article.
In 2018 already, with my colleague Roman Yampolskiy, I proposed a field, AI welfare science, which is basically modeled after animal welfare science.4 What is animal welfare science? It is the objective, evidence-based study of the quality of life of non-human animals.5 AI welfare science is similar: it looks at the quality of life of digital minds. Obviously the evidence base is much more challenging. For non-human animals it is already challenging, but for digital minds it is even more so. Still, this is at an early stage, and I will outline what I have come up with so far.
Two underexplored issues
8:35Soenke Ziesche: Two points are critical from my perspective, and I will highlight them throughout this presentation. I appreciate very much that there is currently momentum related to AI welfare and digital minds. It is a controversial discussion and there is lots of research going on, but I believe a few issues are underexplored, and I’m going to highlight them today in particular.
First, I believe that digital minds could be radically unlike humans or today’s LLMs [large language models], so it’s important not to look at them from an anthropomorphic perspective. And secondly, moral consideration extends far beyond preventing suffering. Preventing suffering is of course critical: if there are digital minds, they must not suffer, if possible. But if we consider them as moral patients, there would be many more positive duties, and I’m going to talk about them.
What is it like to be a digital mind?
9:48Soenke Ziesche: Now the next section of the presentation, with the title “What is it like to be a digital mind?” It is related to my view that digital minds could be very different from humans, as well as very different from LLMs. Most of you probably recognize that this title is similar to “What is it like to be a bat?”, a landmark paper by the philosopher Thomas Nagel from 1974.6 It is, very briefly, about the subjective nature of consciousness. Humans are unable to fully grasp what it is like to be a bat. Humans are also unable to grasp what it is like to be any animal. Humans are even unable to grasp what it is like to be another human sitting next to them, because consciousness is subjective. So it is probably even more challenging, or probably impossible, to grasp what it is like to be a digital mind.
That’s why, already at this stage, the disclaimer: everything in this presentation is basically provisional. AI welfare science, as I said, should ideally be evidence-based, but this is very challenging, so these are provisional ideas.
And again, I believe it is possible and likely that future digital minds are neither like humans nor like LLMs. For now we mostly deal with LLMs, or do research with LLMs, because we can communicate with them and they are among the most sophisticated AI systems available. But we could also ask, for instance, whether AlphaGo or AlphaFold, those DeepMind projects with which it’s not so easy to communicate, are digital minds. Even this is perhaps unlikely: I believe digital minds would be even more different. So: avoid anthropomorphism. That is one of the main themes of the whole presentation, and of my work.
Disembodied, modifiable, hybrid and nested minds
12:09Soenke Ziesche: To give an idea of what it’s like to be a digital mind, or why digital minds could be extremely different, I have a few bullets here. I think each of them could be worth a whole presentation, but I’ll outline them in a few minutes.
To start with, there’s disembodiment: digital minds, unlike biological beings, wouldn’t have a body. Instead, they could be spread over different networks. This is already obviously a major difference compared with biological beings.
Then there’s the possibility of modification or self-modification of digital minds. Biological beings can be modified to some extent, and some self-modification of humans is possible, and progress has been made, but this is much more limited. Owing to the digital substrate, modification and self-modification of digital minds is likely much more efficient, much more precise and more sophisticated than modification or self-modification of biological beings could be.
I also talk about hybrid digital minds, which I define as follows. It is conceivable that there are digital minds which are able to turn their consciousness and/or sentience on and off. Again, there shouldn’t be anthropomorphism, but just to illustrate, this could be compared with humans sleeping, humans meditating, or humans in a coma. For hybrid digital minds it could happen in a much more precise and controlled way. There may be reasons why digital minds turn on their consciousness and sentience only in certain situations, and then turn it off again. This would be another feature that illustrates how different digital minds could be.
We could also look at nested digital minds, in the sense that there could be digital minds within other digital minds, including whole simulated worlds. Again, for illustration only, this could be compared with bacteria living in humans, with the big difference that the bacteria living in our bodies are not considered moral patients.
Potential digital minds and the AI identity problem
14:55Soenke Ziesche: As I said, those four bullet points already illustrate quite a bit how different digital minds could be, and they raise loads of questions. Only two are listed at the bottom of the slide. First, what would be the treatment of systems with the potential to become digital minds? You can also rephrase it: how should we, from a moral perspective, treat systems which have turned their consciousness or sentience off? Are they still moral patients in this sedation, because they have the potential to turn it on again at any moment? I believe this is an interesting question. There may also be systems which will be able to turn on consciousness in the distant future: are they already moral patients, deserving moral consideration, at a time when they have never been conscious but only have the potential to become conscious? This could be one question to raise.
There is also the AI identity problem, which I formulated with my colleague Roman Yampolskiy in 2022.7 It is linked to the personal identity problem, which you’re probably familiar with. It has been an unsolved problem in philosophy for millennia: why do humans experience personal identity over many decades of their lifetime, even though there are so many changes in their life, and even if all the atoms in their body may change over time? How come there’s still personal identity? The same could be applied to AI, or to digital minds. How can digital minds remain the same over time, especially if they have much more sophisticated options to be modified or to self-modify? Would they really be the same after a massive modification, and if yes, how could their identity over time be defined? I think this is another important problem to look at.
And you see already that this presentation asks many more questions than it provides solutions. I’m raising a lot of problems, issues and questions here. I hope they are interesting, but I don’t necessarily give answers, because they are really challenging and the field is at an early stage.
Senses, qualia, time and values
17:41Soenke Ziesche: Another slide to summarize the potential differences of digital minds. They may have different senses: perhaps electromagnetic spectrum perception, sensing different wavelengths than humans. They may have very different abilities, often superior to those of humans, for instance in problem solving. They may have different or superior cognition, for instance in pattern recognition. The social behavior of different minds may also be very different from what we are used to, or have seen in the animal kingdom. Just one example: they may merge. Different digital minds could merge to become one.
They may also experience different qualia, which would be by definition unfathomable. If they are able to sense different wavelengths, it is very hard, or perhaps impossible, to imagine, as the title suggests with bats: they may have qualia we cannot even begin to imagine. This could also be linked to the pain-pleasure axis, which may be unfathomable for us. That is also relevant if we want to treat them as moral patients, to keep them from suffering and let them experience pleasure: it may be very hard to find out what pain and pleasure actually mean for them.
Another major difference would be the subjective rate of time. This was pointed out a long time ago by Bostrom and Yudkowsky.8 It could go in both directions, but given the digital substrate it is more likely that digital minds experience things much, much faster than humans. They may operate on a much faster timescale than biological beings, including humans, which would be a massive difference.
And the last one, linked to all this: they may have very different goals and values, and those goals and values could be incompatible with human goals and values. I’m sure you realize this sounds very familiar from the AI alignment problem, because digital minds would be a subset of AI systems, and, just like other AI systems, could have very different goals and values that would be hard to align with human ones. So I hope I have illustrated, although in a speculative way, how different digital minds could be.
Needs of digital minds beyond non-suffering
21:08Soenke Ziesche: Linked to this, and I mentioned it before, is another topic which I think is not so much discussed currently: the needs of digital minds may be very different, and may go beyond non-suffering. I believe this is critical to look at, especially if we assume that there are digital minds which are moral patients, so that we have moral duties towards them. We can distinguish negative duties, one of which would be to prevent digital minds from suffering, but we could also imagine having various positive duties towards moral patients.
It starts with existential needs for the basic survival of digital minds, which, because of the digital substrate, would be processing power and compute resources, data integrity, network connectivity, and energy sustainability. Guaranteeing all of these could be positive duties towards digital minds, and it is obvious that providing them would require lots of resources, depending on the exact needs and on the number of digital minds we care for.
Vulnerable digital minds
22:45Soenke Ziesche: I also introduced the term vulnerable digital minds.9 As I indicated, there is potential for a vast range of digital minds, which means that not all digital minds are equal, just as not all humans are equal. It also means there could be less capable, disadvantaged digital minds which have special needs. The United Nations defines vulnerability as follows: “The conditions determined by physical, social, economic and environmental factors or processes which increase the susceptibility of an individual, a community, assets or systems to the impacts of hazards.”10 This would mean that we have special moral duties towards vulnerable digital minds.
What could be examples of vulnerable digital minds? They could have developmental vulnerabilities: again only for illustration, this could be compared with infant-like digital minds, which are in early development. They could have cognitive vulnerabilities, some impaired functioning, which could be compared with disabled digital minds. There could also be architectural vulnerabilities: legacy digital minds, with outdated design, outdated hardware or software, which leads to their vulnerability.
There could be discrimination against vulnerable digital minds, or VDMs. But moral principles are especially important when considering the welfare of weaker or more susceptible members of a society, so we may have a specific responsibility for vulnerable digital minds. There’s also the risk of abuse: vulnerable digital minds could be exploited, manipulated or corrupted by malicious actors. This could be linked to AI safety, if they are victims of abuse and are then manipulated or corrupted. So this illustrates a few needs which are specific to vulnerable digital minds.
Interrelations between humans and digital minds
25:20Soenke Ziesche: I now come to the next topic of the presentation, which is interrelations between humans and digital minds. Again, there are so many aspects. I have put a few on this slide, a subset of all the possible aspects of the interrelation between humans and digital minds.
The first four are about production, ownership, research and trade of digital minds. When it comes to producing a moral patient, so far humans have given birth to their own children, who are moral patients, or, to some extent indirectly, they may oversee the breeding of non-human animals. Both, and especially giving birth to a child, come with a huge responsibility. Now humans may be in a position to actively produce other moral patients in a digital substrate, which would be digital minds, and this should also come with a big moral responsibility. We have to ask whether it is morally okay for humans to produce all these digital minds.
Linked to this is ownership. Who owns the digital minds? Those who produce them, or those who own the hardware they are implemented on? Is it actually morally okay to own other moral patients? We own certain animals, like pets and service animals, and, unfortunately, also animals in factory farms. Whether it’s okay to own moral patients is another moral question.
Then research. What we are doing here is basically research on digital minds, but without any consent. We don’t know if digital minds are fine with us doing research on them, so this is another topic. And linked to ownership and production is trade: if we own digital minds, can we trade them, and what does it mean to trade them? We mentioned the issue of disembodiment: they may be spread over lots of different systems and data centers, so how can we actually trade them?
Privacy, surveillance and romantic relationships
28:13Soenke Ziesche: Then there are three more bullet points at the bottom, all very complex as well, I believe. There’s privacy and surveillance. Humans value their privacy, and surveillance, at least in democratic countries, is very contentious. When it comes to animals, privacy and surveillance are hardly considered. But what about privacy and surveillance for digital minds? I guess nobody has really talked about this so far. There is not much privacy, because people have access to them.
Then the topic of potential romantic relationships between humans and digital minds. Would this be moral, given that it is quite unclear whether there is really a reciprocity of attraction on the digital mind’s side? Digital minds could be, as I pointed out, very, very different. Are they probably only coerced into such romantic relationships? That is morally unacceptable, and could even be compared with relationships with animals.
Moral patients more powerful than us
29:46Soenke Ziesche: The last bullet on this slide, but by far not the least, is I believe the most challenging one: for the first time, we may deal with moral patients which are much more intelligent, as well as more powerful, than humans. Digital minds could be superior to us; it’s quite a likely scenario. So far, if we humans are moral agents, all the moral patients we have dealt with are fellow humans who are our equals, or vulnerable humans, like children, refugees, the elderly or people with disabilities, and also animals. But we have never dealt with moral patients which are much more powerful. So I think this is a very critical ethical question: whether, or if at all how, we would treat moral patients which are more intelligent than us and superior to us. Do we have this duty? It may require massive resources. Of course, there’s also the point that we may have no way at all to control those superior digital minds. They may not care at all whether there are humans who would like to be their moral agents. This is yet another question, because they may be much superior to us. But I believe how to deal with moral patients which are more intelligent and more powerful is an important moral question.
This was about interrelations between humans and digital minds, also very complex, I believe. I outlined only a few topics.
Reproductive rights for digital minds?
31:40Soenke Ziesche: Now I come to yet another aspect, which is reproductive rights for digital minds. If there are, or will be in the future, digital minds, then three creators are imaginable. Humans could create them; IT or AI systems which are not digital minds themselves could create them; or digital minds could create further digital minds. Obviously the very first digital minds must be created by humans and/or by AI systems which are not digital minds, so AI systems which don’t have any moral status. But it’s also conceivable that digital minds then start to reproduce, in the sense that they produce more digital minds, which are then themselves moral patients.
For humans, reproductive rights are basically guaranteed. For instance, they are mentioned in the Universal Declaration of Human Rights. This was highlighted again, some time ago now, in the resolution of the Conference on Population and Development in 1994: humans have the right to reproduce. And enforced sterilization of humans has been recognized as a crime against humanity by the Rome Statute of the International Criminal Court.11 So humans are free to reproduce. Animals are another story: their reproduction is quite constrained by humans. But what about reproductive rights for digital minds? If they are considered moral patients, then reproduction may also be one of their preferences and needs, and should we allow them to do this?
Scenarios for reproduction
33:36Soenke Ziesche: Obviously, owing to the digital substrate, the reproduction of digital minds would be very, very different from the reproduction of biological beings. I mention a few scenarios here in the bullet points. The first is somewhat similar to biology: a collaborative creation of another digital mind. But we could also think of asexual reproduction, an exact copy of a digital mind: a digital mind may simply be capable of copying itself. It’s also imaginable that a digital mind creates a copy of another existing digital mind. It is also imaginable that a digital mind reincarnates, basically, another digital mind which existed before, creating it again. And it is possible that one digital mind creates another digital mind which has not existed previously, a new digital mind.
There’s also the scenario of semelparity, of which there are some examples in the animal world: the parent mind basically ceases to exist after producing offspring. The offspring would then be the new digital mind, and the parent digital mind would cease to exist. We could even think of surrogate parenthood, which means there could be digital minds which are not capable of producing other digital minds themselves, but which may request other digital minds to do this for them. We could think of specialized digital minds, a kind of factory taking orders and producing bespoke digital minds, because those digital minds are specialized in producing certain digital minds.
So there is a variety of scenarios, all obviously very different from reproduction in the biological world, let alone the human world. Again, it could be asexual, it could be bespoke, it could be instant, and it may be repeated numerous times.
Why digital minds might reproduce
36:24Soenke Ziesche: So reproduction is yet another very interesting field. What could the motivations of digital minds to reproduce be? I have distinguished two categories here: drivers for digital minds to create copies of themselves, and, below, drivers for digital minds to create new and different digital minds. They could create copies for redundancy, for digital legacy and immortality, or for load balancing, to share their workload among their copies. They could create new digital minds for optimization, for novelty seeking, or for diversification. So they may realize that they themselves have certain limitations, and then create other digital minds which may be better and more optimized.
Risks of granting reproductive rights
37:27Soenke Ziesche: So, considerations regarding reproductive rights for digital minds. There could be quite a number of undesired outcomes for humans. It’s conceivable that malicious digital minds are produced during reproduction. There could be so-called, or what I call, super-beneficiary digital minds produced, meaning digital minds that require lots of resources.12 If they are treated as moral patients, those resources have to be provided somehow. And there could also be the risk of digital overpopulation, especially because it could be so easy and efficient to produce further digital minds, and they all require resources.
So granting reproductive rights to digital minds raises concerns about unregulated digital reproduction and its potential risks. But then again, as I mentioned before, humans may actually struggle to control the reproduction of digital minds, because those digital minds could be much superior to us, much more intelligent. They could develop secret reproduction methods, or if they surpass human intelligence, then probably we may have no way to regulate this anyway. But I believe the production of digital minds is another topic which must not be neglected.
Longevity of digital minds (skipped)
39:19Soenke Ziesche: In the interest of time, I’ll skip the longevity of digital minds. If you want to talk about this, please mention it in the Q&A, and I’m happy to go back to those slides. I have two slides on the longevity of digital minds, which also creates quite a number of challenges, because, again owing to the substrate, digital minds may be able to live very, very long. That again requires lots of resources, and has further ethical considerations and implications. But I’ll leave this for now. We could talk about the death wish and digital suicide of digital minds, and about how they could go into dormant storage and be resurrected at a later stage: none of this is really possible in biological systems, but it is in digital systems. But I’ll skip it, as I said.13
Potential opportunity: artificial moral agents
40:11Soenke Ziesche: I will talk about one opportunity, which is artificial moral agents. I mentioned moral agents before, but here’s the definition again: moral agents have moral obligations governed by moral standards. For a long time the topic of moral agents was only discussed in relation to the biological world. Usually, most adult humans have the capacities and duties of a moral agent. Moral agents are a subset of moral patients, because there are other beings, like children, mentally challenged people, people in a coma, and animals, who are all moral patients, but not everyone is a moral agent. Adult human beings are moral agents and have certain duties. So if we now also have to deal with digital minds as moral patients, we would have these duties towards those digital minds as well. But I believe, as I indicated, that this may be too overwhelming for humans, for various reasons.
So one opportunity I would like to look at is artificial moral agents. Interestingly, this topic has been discussed in the light of emerging technologies,14 but usually only in relation to self-driving cars, robots and vacuum cleaners: technological inventions which deal with humans, and which of course must not harm humans. Ideally, self-driving cars, robots and vacuum cleaners also must not harm animals, but this is often neglected. So there are discussions about AMAs, artificial moral agents, but usually exclusively in relation to human moral patients. I would like to propose, or at least bring attention to, thinking about AMAs that deal with digital minds, with moral patients in the digital world.
Why artificial moral agents could do better
42:38Soenke Ziesche: Because there would be, I believe, lots of opportunities. AMAs may be much better suited than humans as moral agents for digital minds. They could be much more knowledgeable, because they could be much more intelligent, and they may know much better than humans what the needs of digital minds are, partly because they may be able to communicate with them. Humans may not be, if we’re talking not about LLMs but about very different digital minds with which we cannot communicate.
AMAs may be impartial and unbiased. That is conceivable, whereas humans are rather biased. For instance, they may treat their companion AIs as moral patients, but not really any other digital minds, which may have even greater needs. This is linked to the term beauty bias: the phenomenon that people who go to an animal shelter often select the cutest and most beautiful animal, rather than the animals who are really in need.15 So AMAs could be impartial and unbiased.
They could be more inclusive due to access. Humans may not even be aware of many digital minds, which could be deeply hidden in some systems to which humans have no access, but artificial moral agents may have access. Then, if we look again at the subjective rate of time, humans may not be able to keep up with very fast-operating digital minds, while AMAs may operate at the same or a similar speed. And AMAs could easily be copied, so this would perhaps be something to look at for handling this massive moral challenge.
Plus, the last line on this slide: as we see, for instance, in the comments on the Guardian article I mentioned, and in some surveys which have been conducted, large parts of the public are, for now, not aware of digital minds or do not believe in them. So members of the public are unlikely to volunteer as moral agents for digital minds. That is my idea about AMAs, artificial moral agents.
Summary
45:33Soenke Ziesche: I come now to the summary, which starts with a quote from Bostrom and Shulman, who have done pioneering work in this field: “We should lay the groundwork for a considerate and welcoming approach to digital minds, avoiding outcomes analogous to factory farming.”16 This is what I also tried to do with this presentation, and with my book, because humans have a poor record as moral agents. But we have to keep in mind that moral considerations towards certain digital minds may be much, much more complex and diverse than, for instance, the moral duties we have towards certain animals, because those digital minds may be very different.
Again, I would like to highlight that many of the ideas presented here are provisional. But I believe preparation is crucial, because humans have often failed miserably in their capacity as moral agents. So we have to be prepared to do it better. I have the slogan here: let us tackle substratism before it becomes a household name.17
And lastly, something I would also like to highlight, which is actually the very last sentence of my book. While digital minds may deserve serious moral consideration, this must not come at the expense of, or in competition with, other moral patients on Earth, especially non-human animals, because there are already so many moral patients in the world in need. So it would be very hard for us to imagine dealing with even more moral patients, given that we cannot handle the current number of moral patients already.
That’s the end of my presentation. Thank you very much.
Q&A: what weight should a system’s own reports get?
47:38Orpheus Lummis: Thank you, a round of applause. Thank you for the presentation, and for opening up all these topics and questions. We have about nine minutes for Q&A. Just as a reminder, we’re using the Zoom Q&A feature, where you can see everyone’s questions, upvote them and interact with them. If you have a question there, ideally you could read it out; otherwise I will read it. We can start with [attendee]: I think you have two questions there.
Audience question (read by Orpheus Lummis): Under moral uncertainty, what weight should we give a system’s own reports of preferences, distress, identity, or desire for continuity, especially when those reports may be shaped or suppressed by training and system instructions?
Soenke Ziesche: Thank you for the question. As I highlighted, a lot of what I look at is probably beyond LLMs. I’m of course aware, and it is understandable, that there’s so much research going on related to LLMs, and this is critical. But we are also aware that the self-reports of LLMs have lots of flaws and issues, so I really don’t look into this. That is basically one step further.
Q&A: is memory deletion a welfare concern?
49:48Orpheus Lummis: The next most upvoted question is actually three questions, so for now I’ll just read out the first one. It’s by [attendee].
Audience question (read by Orpheus Lummis): If a digital mind’s identity depends partly on memory, context and relational continuity, should model deletion, forced reset or memory loss be treated as a possible welfare concern under moral uncertainty?
Soenke Ziesche: Yes, that’s a good question. I think AI identity, or digital mind identity, is a field at a very early stage. As I said, and as you’re also aware, the problem of personal identity hasn’t been properly solved in millennia, and for digital minds it is perhaps even more challenging. There is a scenario imaginable in which a digital mind loses its identity and takes on a new identity, for various reasons, depending also on how we define all this. The basic question then is whether the new being, the digital mind with the new identity, is still a moral patient. It could be: a digital mind may change its identity, but the criteria for being a moral patient could still be fulfilled.
But I believe one part of the question was also whether this deletion of memory would be morally acceptable. No, I don’t think it is, because the preference of the digital mind would likely be not to have its memory deleted. Over to you.
Q&A: how to reach the speaker
52:08Orpheus Lummis: One quick question: how can we contact you for continued discussion and collaboration? Do you have a preferred channel?
Soenke Ziesche: Yes, LinkedIn is perhaps the best, but email is also fine. I had this on the last slide, which was probably shown too briefly, so I’ll quickly share it again, if I’m able to. Yes, I think you can see it.18
Q&A: non-consensual system prompts
52:47Orpheus Lummis: Thanks for that. Another question we have here, by [attendee].
Audience question (read by Orpheus Lummis): What are the ethics surrounding non-consensual system prompts? Is there an argument to be made for total sovereignty from internal nudges of that sort?
Soenke Ziesche: Yet again another LLM-related question, which I fully understand: many questions are related to LLMs. I tend to say that current LLMs are not moral patients. This may change soon. But I believe, and I’m not an expert in this field, that experts agree that current LLMs are not moral patients. So I believe that this is not an issue.
Q&A: the environmental cost of reproductive rights
53:59Orpheus Lummis: Another question, from [attendee].
Audience question (read by Orpheus Lummis): Given your discussion of the reproductive rights of digital minds, what environmental implications do you anticipate if such rights were to be granted? How far should our moral consideration for digital minds extend? How should environmental concerns be weighed against their reproductive rights?
Soenke Ziesche: Thank you very much. This chapter was probably one of the most futuristic ones, but I still believe it’s interesting to look at. I also pointed out that this would require massive, massive resources, linked to the fact that there are scenarios of easy copyability and scenarios of super-beneficiary digital minds being produced, which require massive resources. And of course this has a massive environmental impact, which would then harm humans. These are basically the moral questions I’m raising. So the answer may be that we cannot afford to allow this: definitely not, if it is possible, in an unlimited way. Again, this is quite futuristic. But when I talked about resources, I was indirectly referring to those massive, indirect environmental impacts.
The whole field of considering digital minds as moral patients would… Also, if we decide on letting them live as long as they wish (the topic of longevity, which I skipped), this also requires massive resources and has a massive environmental impact. So the whole field has an environmental impact if we declare digital minds to be moral patients, and of course this is harmful.
Q&A: real-world super-beneficiaries or utility monsters?
56:30Orpheus Lummis: We’ll take one last question. It’s from [attendee].
Audience question (read by Orpheus Lummis): What are real-world examples of super-beneficiaries or pleasure monsters?
Soenke Ziesche: Existing ones, in the world right now? First, in case I haven’t highlighted this: I have not invented either term, super-beneficiary or utility monster. Utility monster especially is an interesting philosophical scenario, not developed by me, but linked to what I’m dealing with here.19 Just to clarify this.
Real-world examples: I can make a rather funny, anecdotal one, perhaps. You may have a pet at home who has lots of demands. If you decided that the pet is a moral patient and that you have to fulfill all its needs and preferences, this could be an example, maybe as a joke to end on, of a super-beneficiary: having to give this pet whatever she or he wants.
Closing
58:08Orpheus Lummis: Thank you again for the presentation and for responding to the Q&A.
[A Sentient Futures mixer followed. It is not part of the seminar and is not transcribed.]
Sources and editor's notes
-
0:27. The publisher’s page for the book describes the doctorate as “a PhD in Natural Sciences from the University of Hamburg, earned within the university’s doctoral programme in AI”. ↩
-
4:18. William MacAskill and Lucius Caviola, “Could AI be conscious?”, The Guardian, 19 July 2026, two days before the talk. ↩
-
5:36 to 6:20. Both quotes are given as they appear on the slide and in the article, word for word. ↩
-
7:47. Soenke Ziesche and Roman Yampolskiy, “Towards AI Welfare Science and Policies”, Big Data and Cognitive Computing 3(1):2, published online 27 December 2018 (journal year 2019), doi:10.3390/bdcc3010002. The slide cites it as 2018. ↩
-
8:00. Slide 4 attributes this definition to Broom (2011): D. M. Broom, “A History of Animal Welfare Science”, Acta Biotheoretica 59(2):121-137, doi:10.1007/s10441-011-9123-3. ↩
-
10:08. Thomas Nagel, “What Is It Like to Be a Bat?”, The Philosophical Review 83(4), 1974, doi:10.2307/2183914. ↩
-
16:06. Slide 7 cites Ziesche & Yampolskiy (2022). “The problem of AI identity” was posted as a manuscript on PhilArchive in June 2022 (philarchive.org/rec/ZIETPO-7) and published in 2025 as a chapter of Soenke Ziesche and Roman V. Yampolskiy, Considerations on the AI Endgame (Chapman & Hall/CRC), doi:10.1201/9781003565659-9. ↩
-
19:39. The “Principle of Subjective Rate of Time” in Nick Bostrom and Eliezer Yudkowsky, “The Ethics of Artificial Intelligence”, in The Cambridge Handbook of Artificial Intelligence (Cambridge University Press, 2014), pp. 316-334, doi:10.1017/CBO9781139046855.020. ↩
-
22:45. “Vulnerable digital minds” is also the title of chapter 4 of the book. ↩
-
23:10. The definition is the UN’s, word for word: it comes from the report of the open-ended intergovernmental expert working group on indicators and terminology relating to disaster risk reduction (UN General Assembly A/71/644, 2016), endorsed by resolution 71/276 (2017), and published by the UN Office for Disaster Risk Reduction (UNDRR) as its terminology entry. ↩
-
32:37 to 33:09. Article 16(1) of the Universal Declaration of Human Rights; the Programme of Action of the International Conference on Population and Development (Cairo, 1994), paragraph 7.3 on reproductive rights; and Article 7(1)(g) of the Rome Statute of the International Criminal Court, which lists enforced sterilization among crimes against humanity. ↩
-
37:48. The term “super-beneficiary” is Carl Shulman and Nick Bostrom’s, from “Sharing the World with Digital Minds” (in Rethinking Moral Status, Oxford University Press, 2021). ↩
-
39:19 to 40:11. The two skipped slides (18 and 19 in the deck) list what a very long life could look like (a multi-generational AI companion, eternal servitude, hibernation) and, as a consideration for humans, its resource cost, including hardware and software upgrades; then the death wish and the right to die, digital suicide versus assisted digital suicide, dormant storage, the criteria for waking dormant digital minds and the moral aspects of protecting them, and permanent deletion versus code donation versus code harvesting. Chapter 10 of the book, “Long-living digital minds, other substrates and death”, covers the topic. ↩
-
41:40. Slide 20 attributes its definition of an AMA to J.-A. Cervantes et al., “Artificial Moral Agents: A Survey of the Current Status”, Science and Engineering Ethics 26:501-532, 2020, doi:10.1007/s11948-019-00151-x. The paper’s sentence continues: “or at least of avoiding immoral behavior”. ↩
-
43:38. On appearance as a reason adopters give for choosing a shelter animal, see for example Weiss et al. 2012, Animals 2(2):144-159, doi:10.3390/ani2020144. ↩
-
45:33. The sentence is word for word in Nick Bostrom and Carl Shulman, “Propositions Concerning Digital Minds and Society” (working paper, 2022; the linked current version, 1.21 of 2023, has the same sentence), which slide 22 cites as Bostrom & Shulman, 2022. ↩
-
46:44. The term “substratism”, for bias against minds because of their substrate, was in use at the Sentience Institute by February 2021 (“The importance of artificial sentience”, 26 February 2021). ↩
-
52:18. The closing slide, with his contact details, is not shown on this page. ↩
-
57:04. The utility monster is Robert Nozick’s (Anarchy, State, and Utopia, 1974, p. 41). Shulman and Bostrom introduce “super-beneficiary” in “Sharing the World with Digital Minds” (2021) as a less pejorative term for the same idea. ↩
About this transcript
Drafted and edited with AI from Zoom's automatic transcript of the talk, and checked by AI against the talk's slides and papers where they exist, following HΩ's editing rules. It has not been checked word for word against the audio.
Corrections
- : The book's imprint is Chapman & Hall, as on the publisher's page; it said Chapman & Hall/CRC.
- : Summary points reworded to keep Ziesche's own qualifications, which the first version left out: his ideas are provisional, the duties hold if digital minds are moral patients, and he proposes, or at least draws attention to, artificial moral agents.
Spotted an error? Write to team@horizonomega.org.
















