An automated decision

An automated decision combines a prediction with a choice about what to do to a person. Even a technically accurate model cannot decide which errors are acceptable or what explanation someone is owed. The ethical task includes the objective being optimized, the data used, and the institution responsible for the result.

A prediction is only one step in an institutional decision. A model might estimate a risk, while people choose the threshold, the consequences of crossing it, and the process for reconsideration. Each choice distributes burdens differently. False rejection may mean lost support for an applicant, while false acceptance may consume shared resources. Calling the system accurate does not answer whose errors are tolerated, who can contest them, or who remains accountable for the policy.

Imagine a benefits system that processes applications faster but rejects some eligible applicants. Compare a design that offers a prompt human appeal with one that hides its reasons. The overall error rate could be identical while the practical burden on an affected person differs sharply.

Efficiency, equal treatment, and individual rights can support different designs. Human review is not automatically fair either: reviewers may lack time, authority, or relevant information. Ask who can identify a mistake, who can reverse it, and whether the people carrying the risk had any influence over the rule.

Extend the benefits example by giving applicants different access to documents and internet connections. An appeal exists on paper, but it may be unusable for someone with little time or help. Evaluate the entire route from application to correction, including how the institution learns about repeated mistakes. A system can improve average processing speed while worsening the position of a smaller group; both effects deserve an explicit place in the assessment.