← All questionsETHICAL QUESTION / 05

A DILEMMA TO THINK WITH

An automated decision.

Who bears the risk of an automated decision, and who gets to challenge it?

Railway tracks running through autumn trees
01 / THE SCENARIO

Imagine this.

A public agency or platform uses an automated system to make decisions that affect people. Every available design has tradeoffs, and its predictions can be wrong.

THE CENTRAL QUESTIONWho bears the risk of an automated decision, and who gets to challenge it?
GO DEEPER / CONTEXT & DEBATE

What makes this difficult?

Understanding the topic

An automated decision combines a prediction with a choice about what to do to a person. Even a technically accurate model cannot decide which errors are acceptable or what explanation someone is owed. The ethical task includes the objective being optimized, the data used, and the institution responsible for the result.

A prediction is only one step in an institutional decision. A model might estimate a risk, while people choose the threshold, the consequences of crossing it, and the process for reconsideration. Each choice distributes burdens differently. False rejection may mean lost support for an applicant, while false acceptance may consume shared resources. Calling the system accurate does not answer whose errors are tolerated, who can contest them, or who remains accountable for the policy.

An illustrative example

Imagine a benefits system that processes applications faster but rejects some eligible applicants. Compare a design that offers a prompt human appeal with one that hides its reasons. The overall error rate could be identical while the practical burden on an affected person differs sharply.

Where the debate stands

Efficiency, equal treatment, and individual rights can support different designs. Human review is not automatically fair either: reviewers may lack time, authority, or relevant information. Ask who can identify a mistake, who can reverse it, and whether the people carrying the risk had any influence over the rule.

Extend the benefits example by giving applicants different access to documents and internet connections. An appeal exists on paper, but it may be unusable for someone with little time or help. Evaluate the entire route from application to correction, including how the institution learns about repeated mistakes. A system can improve average processing speed while worsening the position of a smaller group; both effects deserve an explicit place in the assessment.

Background referencesDeontological Ethics
02 / WAYS OF THINKING

Look through more
than one lens.

These are starting points for reasoning, not verdicts. Different thinkers within the same tradition can disagree.

Waves rolling onto a sandy coastline

Utilitarianism

Assess likely benefits and harms across all affected people.

ASK YOURSELF

How reliable are the predictions, and whose harms are hardest to measure?

Understand this lens

Understanding the topic

Utilitarianism evaluates choices through their effects on well-being, giving comparable interests equal weight regardless of whose they are. Classical versions connect well-being with pleasure and the absence of pain; other versions use preferences or broader accounts of welfare. It belongs to the larger family of consequentialist theories.

An impartial welfare perspective asks that similar interests receive similar weight regardless of whose interests they are. This does not mean equal quantities of every resource: helping someone in severe distress may produce a different benefit from adding another comfort to an already comfortable life. The theory therefore needs an account of well-being and a way to compare improvements. Its appeal to overall benefit is a starting principle, while practical judgments still depend on evidence about the available choices.

Explore the example and debate

An illustrative example

Imagine choosing between a small benefit for many residents and a major benefit for a few. Count the size of each benefit, not just the number of recipients. A majority gaining slightly can still be outweighed by severe losses imposed on a minority.

Where the debate stands

Aggregation, demandingness, and the treatment of rights are persistent challenges. Supporters can consider indirect effects, uncertainty, and the value of dependable rules. Critics ask whether even a fully informed welfare calculation could justify treating one person unfairly. The debate concerns the standard of rightness as well as usable ways of deciding.

Imagine a town deciding between accessible transport and a decorative public project. An outcome-based comparison should include who can use each project, the seriousness of the benefits, and what the town would forgo. Counting participants alone may miss a large improvement for a smaller group. The example exposes a recurring question: can a welfare calculation adequately protect people with unusual needs, or must additional principles constrain what institutions may trade away?

Read the full topic ↗
The Lady Justice statue above the Old Bailey

Duties & rights

Set limits on what may be done to people for the sake of efficiency.

ASK YOURSELF

What rights should constrain an automated rule?

Understand this lens

Understanding the topic

Deontological theories evaluate actions through duties, rights, and constraints that cannot simply be traded for a larger overall benefit. They differ about whether the central concern is an agent’s obligations, a person’s claims against others, or principles of justifiable interaction. Kantian ethics is one influential approach within this family.

A constraint differs from treating rights as one more benefit to maximize. Someone might think that they must not deceive an innocent person even if that deception would prevent several other lies. The prohibition concerns their own action and the claim of the person deceived. Duty-based approaches nevertheless differ about positive assistance, conflicting obligations, and exceptional circumstances. Saying that rights matter therefore begins a discussion about their grounds, holders, and limits rather than ending it.

Explore the example and debate

An illustrative example

Imagine a researcher tempted to use private records without permission because the analysis could help many people. A duty-based assessment asks what participants are owed and whether the use respects their authority, alongside any assessment of the project’s expected benefits.

Where the debate stands

Constraints create hard cases when every option infringes a duty. Some theories permit exceptions beyond a threshold; others distinguish doing harm from allowing it. A key objection asks why one should respect a constraint when violating it would prevent more violations. Different deontologists offer different answers rather than a single universal formula.

Consider a fictional official who can improve service statistics by concealing unfavorable information from applicants. A duty-based objection asks what the applicants are owed as people entitled to make informed choices. It cannot be answered solely by showing that the average outcome improved. But the objection still needs precision: which information is relevant, who has a claim to it, and whether protecting someone else’s privacy creates a competing responsibility?

Background referencesDeontological Ethics
Read the full topic ↗
Sunlit path through an evergreen forest

Virtue ethics

Examine the judgment and accountability of the people and institutions building the system.

ASK YOURSELF

How do good intentions become reliable institutional practice?

Understand this lens

Understanding the topic

Virtue ethics asks what a good person notices, values, feels, and does across a life. Virtues such as courage and honesty involve cultivated dispositions rather than isolated compliant acts. Practical wisdom helps determine how those dispositions apply when circumstances are complicated and virtues appear to pull in different directions.

Virtue concerns patterns of perception, feeling, judgment, and action rather than merely repeated outward behavior. Someone who tells the truth only from fear of being caught differs from someone who recognizes honesty’s place in a relationship. Acting well also requires seeing which features matter in a particular situation. Courage without good judgment can become recklessness, and generosity without attention can become intrusive. The account therefore links character to practical understanding and the reasons for acting.

Explore the example and debate

An illustrative example

Imagine giving honest feedback to someone who is struggling. Bluntness, concealment, and considerate candor differ in timing and attention as well as truthfulness. Ask what a perceptive and caring person would recognize about the situation before choosing how to speak.

Where the debate stands

Critics ask whether virtues provide enough guidance for unfamiliar dilemmas and who sets the model of good character. Different traditions answer with different accounts of flourishing and exemplars. Character also develops within institutions, so judging individuals alone can miss conditions that encourage courage, reward deception, or make good action unusually costly.

Imagine a friend asking for difficult feedback. Honesty supports telling the truth, while kindness shapes timing and expression; practical wisdom helps avoid both cruelty and evasiveness. This does not reduce the answer to a convenient midpoint between extremes. The challenge is to explain what the relationship and situation require. A virtue account becomes more informative when it identifies the relevant facts and reasons, instead of merely describing the chosen action as virtuous.

Background referencesVirtue Ethics
Read the full topic ↗
03 / INTO THE WORLD

Where this question
shows up today.

Thought experiments simplify reality. These links show where a similar tension appears in a richer social context.

04 / READ FURTHER

Follow the sources.

Philosophical connections in this atlas are curated interpretations. Read the original discussions and reference entries to go deeper.

KEEP EXPLORING

The next question
changes everything.

The trolley problemThe transplant case