Utilitarianism is the moral philosophy proposed in the 1800s that the right action is the one that produces the most good, as measured in a universal currency of “utility”. An individual experiencing pleasure results in positive utility; pain results in negative utility. At any given point, the morally correct action is the one that maximizes this across all living entities.

Its modern, activist descendant is effective altruism (EA), which uses evidence and explicit calculation to decide where to allocate time and money. Famously, many participants cite studies that one’s happiness plateaus at $75,000 / year and donate the rest of their income. Yet others give away kidneys. It’s one of the most influential philosophies in tech circles, with major influence over companies like Anthropic and its CEO, Dario Amodei. Another infamous proponent was Sam Bankman-Fried of FTX fame. While the philosophy is useful, I think it’s missing something important as you scale to complex problems.

For background, I’ve heard a few classes of objection to the philosophy.

  • People are concerned with the practical calculation of utils. Jeremy Bentham, when introducing the philosophy, suggested exactly that we assign a utility value to every consequence based on the pleasure or pain it causes. This could prove impractical or potentially not totally ordered (in the mathematical sense).
  • Others have posited the existence of a “utility monster” that experiences such pleasure that it would consume all resources in society.
  • A more recent worry is the cluelessness objection, suggesting we can’t actually predict the long-run consequences the theory asks us to weigh.

My issue is different, influenced by Popper and Deutsch’s idea that observations are theory-laden.

I’ve asked EA / utilitarian friends questions like:

“Would it be appropriate to kill a human with no relatives, assuming it were painless and provided the bearer with pleasure?”

“Is a person with no family or friends committing suicide the morally right thing (erasing pain = positive utility gain)?”

“How much present suffering should we cause for future humans’ benefit?”

This isn’t just theoretical. Longtermism, to take the last example, is a popular philosophy in academic and tech circles. The answer I often hear is that “it would all be accounted for in the system”. In other words, the value of life, the imperative to not kill, and the “discounted” value of future human pleasure are inputs into the system.

The argument never sat well with me. Every question gets neutralized by swallowing it into the utilitarian framework. All the toughest moral problems are just inputs to the computation, which spits out the next appropriate action.

I realized my issue with it just recently: the framework dodges foundational moral questions. A functioning moral framework is one that can answer difficult questions of comparison. How much present suffering is worth an investment in green energy? How should we treat the mentally ill? At what point is physical action appropriate in a dispute?

Utilitarianism presupposes that these questions are already answered by assuming we’ve reduced them to the currency of utility. But if we already had a way to compare moral actions, then we would have 90% of a moral framework! And yes, we’d likely want to maximize “utility”.

A utilitarian will object that the theory isn’t empty: it carries a substantive commitment to “wellbeing” as the good, counting everyone’s equally. But not only is “wellbeing” often poorly defined, it says nothing about some of the most pressing comparisons: present versus future people, humans versus animals, safety versus progress. Many modern EAs suggest those are merely “parameters”, as if they were settled, when they are the moral question. The valuation, if it can be done, is more important than the calculation.

None of this is original to me. Philosophers have long noted that consequentialism is only half a theory; it tells you to maximize the good (a theory of the right) but has to import a separate account of what good even is (a theory of the good, or axiology). John Rawls, for example, suggested utilitarianism “does not take seriously the distinction between persons”. My main addition here is a framing that will be familiar to EA / tech circles: utility comparison is theory-laden in Deutsch’s sense. How we ascribe utility in the first place is the “theory”. The “observations”, our utility calculations and actions, are secondary. Using utilitarianism as your primary moral tool is putting the cart before the horse.

Peter Singer’s seminal thought experiment is one where you pass by a drowning child in a pond and are willing to ruin your new shoes for it. But why do we value the life of the child? What’s so great about a nice pair of shoes? What’s the guiding principle such that everyone reading knows a life is worth more than the newest LV Oxfords? That is the real moral question.

To be clear, there are many cases where utilitarianism, given our value-set, gives us obvious guidance. My pleasure at eating pork is minuscule compared to the torture of pigs in factory farms. A $100 pair of shoes that I may not need is worth less than 100 malaria nets. This is a real contribution of the EA system: there are plenty of low-hanging fruit that would obviously improve global welfare. However, I worry that as we move up the hierarchy of needs to more complex moral issues, the utilitarian / EA framework can be more of a distraction than a help. Already, we’ve seen multiple cases of Anthropic at odds with others arguably because they have different definitions of good, not because they don’t believe good should be maximized. If we take AI job displacement and safety concerns seriously, we have to question where the underlying “calculations” come from.