Let’s say we want to identify effective strategies for multi-agent games like the Iterated Prisoner’s Dilemma or more complex environments (like the kind of environments in Melting Pot). Then tournaments are a natural approach: let people submit strategies, and then play all these strategies against each other in a round-robin tournament.

There are lots of examples of such tournaments: Most famously, Axelrod ran two competitions for the iterated Prisoner’s Dilemma, identifying tit for tat as a good strategy. Less famously, there’s Alex Mennen’s open-source Prisoner’s Dilemma tournament, and my own little “open-prompt” Prisoner’s Dilemma tournament.

But there’s a question of what to do with these results and in particular how to reward participants for their strategies’ success. The most principled approach would be to reward participants in direct proportion to the utility they obtain. If your strategy earns 100 units of utility across interactions with all opponents, you get $100.

Unfortunately, this principled scheme is not very practical.¹ Instead, I think tournaments will usually give ranking-based rewards. Alex Mennen and I gave a monetary reward to the top-ranked strategy in our respective tournaments. Axelrod didn’t hand out monetary rewards, but dedicated quite a lot of attention in his book to the top-ranking strategy (or strategies) (crediting Anatol Rapoport for submitting it).

Unfortunately, ranking-based rewards come with a number of problems. One (known) problem is that if you only reward the top (say, top three) programs, high-risk strategies might become more attractive than their expected utility would suggest. (It might be better to get, say, a utility of 100 for 1st place with 10% probability and a utility of 0 for bottom place with 90% probability, than to reliably get a reward of 90 for a ranking between 10 and 20.)²

I here want to focus on a different issue with ranking-based rewards: they create perverse incentives for spiteful or over-competitive behavior. Another’s failure might be your gain, if it makes the other participant fall below you in the rankings. For example, imagine we’re running an Iterated Prisoner’s Dilemma tournament with the following twist. Let’s say that occasionally players get an opportunity to inflict at no cost to themselves a large negative payoff (say, -100) on their opponent in a way that’s undetectable to the opponent (and thus unpunishable). In a normal utility-based framework, there’s no reason to take this action. However, under ranking-based rewards, you would always want to take this spiteful action. Decreasing your opponent’s total utility across the tournament might lower their ranking below yours and thus improve your position in the final standings. Perhaps even more troublingly, for small enough epsilon, you’d be willing to sacrifice epsilon units of your own reward just to inflict this large negative payoff on your opponent.³

There is a relatively simple solution to this spite problem, however: Split the participant pool in two, e.g., at random. Call them Group A and Group B. Instead of having every strategy play against every other strategy (round-robin), have strategies from Group A play only against all the strategies from Group B. Then, rank participants only within Group A. Do the analogous with Group B. So in the end you obtain two rankings: a ranking of Group A in terms of performance (in terms of utility obtained) against Group B; and a ranking of Group B against Group A.

This approach eliminates the spite incentive entirely. Since you’re not ranked against anyone that you play against, you have no intrinsic interest in your opponent’s utilities.

Further considerations: Of course, there’s a cost to this: if we hold the set of submitted strategies fixed (and thereby ignore the problematic incentives of traditional round-robin), then in some sense the two-groups-tournament is less informative than the single-group round-robin tournament. This is because we evaluate each strategy against a smaller sample of opponent strategies. For instance, if we have 40 participants, then normally we’d get an estimate of each participant’s expected utility based on 39 samples of opponent strategies. With the two-group split, we only get an estimate based on 19 samples (from the same distribution) of opponent strategies. So we get worse at deciding whether the submitted strategies are good/bad. (As the number of submission increases, the downside of halving the number of samples decreases, but the motivating risk of round-robin tournaments is also more pronounced if the number of submissions is small.)

A different possible concern is that outside observers will not abide by the “ranking only within group” rule. If all of the submitted strategies are published, then an observer could simply run a round-robin tournament on the submitted strategies and publish an alternate ranking. This alternate ranking might then confer prestige, and to optimize their position in the alternate ranking, participants are once again incentivized to submit spiteful strategies. To prevent this, one would have to make it difficult enough to reproduce the tournament. (In many cases, one might justifiably hope that nobody will bother publicizing an alternate ranking, though.)

Even if just the overall utility obtained is published, then a third-party observer might simply take the highest-scoring strategy across the two groups to be the “real winner”. For small numbers of participants, the two winners are incomparable, because they faced different sets of opponents. But the distribution over opponents is the same. So as the participant pool grows, the comparison becomes valid. If outside parties naively aggregate across groups and the resulting comparisons confer prestige, participants are once more incentivized to submit spiteful programs to fare better in the aggregated rankings.

To combat this, the tournament organizers could try to publish more limited information about the different strategies’ scores.⁴

Related ideas: The spite incentive problem and the population split idea follow a common pattern in mechanism design: In many cases, your first idea for a mechanism will set some bad incentives on participants, because of some specific downstream effect of the participants’ decisions. To fix the mechanism, you can remove that downstream effect, usually at some cost.

For instance, this is how you might get from a first- to a second-price auction:

• The most natural idea for an auction is to just ask participants (bidders / people interested in the item you’re looking to sell) what they’d be willing to pay for the item and sell to the highest bidder. How much do you then charge the highest bidder? Well, however much they’re willing to pay, of course! Or perhaps, however much they’re willing to pay minus $1 (since otherwise they wouldn’t gain anything from participating).
• But now you realize that participants aren’t necessarily incentivized to honestly report how much they’re willing to pay: by reporting a lower number they might get away with paying less for your item.
• Let’s say that you want bidders to report their willingness to pay honestly (e.g., because you want to make sure to sell to whoever wants it most). Then to fix the incentive issue, you need to determine the price paid by the winner in a way that doesn’t depend on the winner’s bid, while still making sure that the price is at most equal to the winner’s bid. One way of doing this is to charge them the second-highest bid. (It could also be the third-highest bid or the average of all the other bids, but, of course, these numbers are all lower than the second highest bid and so result in less profit. If in addition to the participants’ bids, you get a signal from a third party, telling you minimum distance between bids is $20, then you could also charge the second bid plus $20.) Of course, the cost is that the price we charge is not informed by the winning participant’s willingness to pay.

The population splitting idea can be discovered by an analogous line of thinking: to fix the spite incentive, you need to remove the influence from your opponent’s utility on your ranking. The simplest way of doing this is to only play against opponents against whom you’re not subsequently ranked.

Of course, the details of the auction case are quite different from the present setting.

The most closely related idea that I’m aware of is about eliminating similar bad incentives in peer review. In peer review, the same people often serve as reviewers and as authors. Conferences often select papers in part based on rankings (rather than just absolute scores) of the papers. In particular, they might choose based on rankings within specific areas. Therefore, if you serve as a reviewer on a paper in your area, you might be incentivized to give the paper a lower score in order to improve the ranking of your paper. The obvious idea to avoid this problem is to make sure that a reviewer isn’t assigned to a paper that they’re competing against. For a brief introduction to this problem and solution, along with some references, see Shah (2022, Section “Dishonest Behavior”, Subsection “Lone Wolf”). I think the fundamental idea here is the same as the idea behind the above partition-based approach to cooperative AI competitions. Some of the details importantly differ, of course. For instance, each reviewer can only review a small number of papers anyway (whereas in the cooperative AI competition one can evaluate each program against all other programs).

Footnotes:

1. In many cases, the rewards aren’t primarily monetary. Rather, a high ranking is rewarded with attention and prestige, and associated prestige and attention are hard to control at all. But even distributing monetary rewards in proportion to utility is impractical. For example, you now effectively reward people merely for entering the competition at all. Requiring a submission fee to counter this has its own issues (such as presumably becoming subject to gambling regulations). ↩︎
2. There is some literature on this phenomenon in the context of forecasting competitions. That is, in a forecasting competitions with ranking-based reward, forecasters typically aren’t incentivized to submit predictions honestly, because they want to increase (and sometimes decrease!) the variance of their score. See, e.g., Lichtendahl and Winkler (2007) and Monroe et al. (2025). ↩︎
3. How big is the spite issue? Well, for one this depends, of course, on the game that is being played. My guess is that in the simple Prisoner’s Dilemma games that most of the existing tournaments have focused on, these spiteful incentives are relatively insignificant. There just aren’t any opportunities to hurt opponents cheaply. But I’d imagine that in more complex games, all kinds of opportunities might come up, and so in particular there might be opportunities for cheap spite.
   
   Second, the importance of spite depends on the number of participants in the tournament. In the extreme case, if there are only two participants, then the interaction between the two is fully zero-sum. As the number of participants grows, the benefits of harming a randomly selected opponent decrease, as a randomly selected opponent is unlikely to be a relevant rival.
   
   The exact dynamics are complicated, however. For instance, if there are only two participants with a realistic chance at first place (perhaps due to technical implementation challenges), then encounters between these top contenders become effectively zero-sum from each player’s perspective – if they can “recognize” each other. ↩︎
4. A natural first idea is to only publish relative scores for each group. For instance, in each group separately, subtract from each strategy’s utility the utility of the top strategy. This way the published top score in each group will be 0. This will work against a naive third party. But a more sophisticated third party might be able to infer the original scores if there are pairs of very similar strategies between the groups. For instance, in the Iterated Prisoner’s Dilemma, we might imagine that both groups will contain a DefectBot, i.e., a strategy of defecting all the time, regardless of the opponent’s actions. So in some sense this approach obscures too little information. For better protection, one might have to obscure information even more. The question of how to do this is similar to the questions studied in the Differential Privacy framework.
   
   One radical approach would be to only publish the ranking and to never publish numeric scores. Presumably this makes it impossible to compare across groups. This has other downsides, though. If, say, our top-three strategies achieve similar scores, they deserve similar amounts of attention – they are similarly promising approaches to cooperation. If we publish scores, outside observers will be able to tell that this is the case. If we publish just the ranking, observers won’t know whether, say, the second-best strategy is competitive with the best or not. So we’re making it more difficult for outside observers to draw conclusions about the relative promise of different research directions. We might also worry that publishing only the ranking will worsen the incentives toward high-variance strategies, because the reward for coming in at a close second place is lower. ↩︎

Short summary and overview

The SPI framework tells us that if we choose only between, for instance, aligned delegation and aligned delegation plus surrogate goal, then implementing the surrogate goal is better. This argument in the paper is persuasive pretty much regardless of what kind of beliefs we hold and what notion of rationality we adopt. In particular, it should convince us if we’re expected utility maximizers. However, in general we have more than just these two options (i.e., more than just aligned delegation and aligned delegation plus surrogate goals); we can instruct our delegates in all sorts of ways. The SPI formalism does not directly provide an argument that among all these possible instructions we should implement some instructions that involve surrogate goals. I will call this the surrogate goal justification gap. Can this gap be bridged? If so, what are the necessary and sufficient conditions for bridging the gap?

The problem is related to but distinct from other issues with SPIs (such the SPI selection problem, or the question of why we need safe Pareto improvement as opposed to “safe my-utility improvements”).

Besides describing the problem, I’ll outline four different approaches to bridging the surrogate goal justification gap, some which are at least implicit in prior discussions of surrogate goals and SPIs:

• The use of SPIs on the default can be justified by pessimistic beliefs about non-SPIs (i.e., about anything that is not an SPI on the default).
• As noted above, we can make a persuasive case for surrogate goals when we face a binary decision between implementing surrogate goals and aligned delegation. Therefore, the use of surrogate goals can be justified if we decompose our overall decision of how to instruct the agents into this binary decision and some set of other decisions, and we then consider these different decision problems separately.
• SPIs may be particularly attractive because it is common knowledge that they are (Pareto) improvements. The players may disagree or may have different information about whether any given non-SPI is a (Pareto) improvement or not.
• SPIs may be Schelling points (a.k.a. focal points).

I don’t develop any of these approaches to justification in much detail.

Beware! Some parts of this post are relatively hard to read, in part because it is about questions and ideas that I don’t have a proper framework for, yet.

A brief recap of safe Pareto improvements

I’ll give a brief recap (with some reconceptualization, relative to the original SPI paper) of safe Pareto improvements here. Without any prior reading on the subject, I doubt that this post will be comprehensible (or interesting). Probably the most accessible introductions to the concepts are currently: the introduction of my paper on safe Pareto improvements with Vince Conitzer, and Sections 1 and 2 of Tobias Baumann’s blog post, “Using surrogate goals to deflect threats”.

Imagine that you and Alice play a game G (e.g., a normal-form game) against each other, except that you only play this game indirectly: you both design artificial agents or instruct some human entities that play the game for you. We will call these the agents. (In the SPI paper, these were called the “representatives”.) We will call you and Alice the principals. (In the SPI paper, these were called the “original players”.) We will sometimes consider cases in which the principals self-modify (e.g., change their utility functions), in which case the agents are the same as (or the “successor agents” of) the principals.

We imagine that the agents have great intrinsic competence when it comes to game playing. A natural (default) way to indirectly play G is therefore for you and Alice to both instruct your agents by giving them your utility function (and perhaps any information about the situation that the agents don’t already have), and then telling them to do the best they can. For instance, in the case of human agents, you might set up a simple incentive contract and have your agent be paid in proportion to how much you like the outcome of the interaction.

Now imagine that you and Alice can (jointly or individually) intervene in some way on how the agents play the game, e.g., by giving them alternative utility functions (as in the case of surrogate goals) or by barring them from playing this or that action. Can you and Alice improve on the default?

One reason why this is tricky is that both with and without the intervention, it may be very difficult to assign an expected utility to the interaction between the agents due to equilibrium selection problems. For example, imagine that the base game G is some asymmetric version of the Game of Chicken and imagine that the intervention results in the agents playing some different asymmetric Game of Chicken. In general, it seems very difficult to tell which of the two will (in expectation) result in a better outcome for you (and for Alice).

Roughly, a safe Pareto improvement on the base game G is an intervention on the interaction between the agents (e.g., committing not to take certain action, giving them alternate objectives, etc.) that you can conclude is good for both players, without necessarily resolving hard questions about equilibrium selection. Generally, safe Pareto improvements rest on relatively weak, qualitative assumptions about how the agents play different games.

Here’s an example of how we might conclude something is Pareto-better than the default. Imagine we can intervene to make the agents play a normal-form game G’ instead of G. Now imagine that G’ is isomorphic to G, and that each outcome of G’ is Pareto-better than its corresponding outcome in G. (E.g., you might imagine that G is just the same as G’, except that each outcome has +1 utility for both players. See the aforementioned articles for more interesting examples.) Then it seems plausible that we can conclude that this intervention is good for both players, even if we don’t know how the agents play G or G’. After all, there seems to be no reason to expect that equilibrium selection will be more favorable (to one of the players) in one game relative to the other.

Introducing the problem: the SPI justification gap 

I’ll now give a sketch of a problem in justifying the use of SPIs. Imagine (as in the previous “recap” section) that you and Alice play a game against each other via your respective agents. You get to decide what game the agents play. Let’s say you have four options G₀,G₁,G₂,G₃ for games that you can let the agents play. Let’s say G₀ is some sort of default that G₁ is a safe Pareto improvement on G₀. Imagine further that there are no other SPI relations between any pair of these games. In particular, G₂ and G₃ aren’t safe Pareto improvements on G₀, and G₁ isn’t an SPI on either G₂ or G₃.

For a concrete example, let’s imagine that G₀ is some sort of threat game with aligned delegation (i.e., where the agents have the same objectives as you and Alice, respectively).

	Threaten	Not threaten
Give in	0,1 0,1	1,0 1,0
Not give in	-3,-3 -3,-3	1,0 1,0

The black numbers are the agent’s utility functions; the blue numbers are your and Alice’s utility functions. (Since the agents are aligned with you and Alice, the black numbers are the same as the corresponding blue numbers.)

(You might object that, while this game has multiple equilibria, giving in weakly dominates not giving in for the row player and therefore that we should strongly expect this game to result in the (Give in, Threaten) outcome. However, for this post, we have to imagine that the outcome of this game is unclear. One way to achieve ambiguity would be to imagine that (Not give in, Not threaten) is better for the row player than (Give in, Not threaten), e.g., because preparing to transfer the resource takes some effort (even if the resources are never in fact demanded and thus never transferred), or because there’s some chance that third parties will observe whether the row player is resolved to resist threats and are thus likely to threaten the row player in the future).)

Then let’s imagine that G₁ is the adoption of a surrogate goal and thus the deflection of any threats to that surrogate goal. Let’s therefore represent G₁ as follows:

	Threaten surrogate	Not threaten
Give in	0,1 0,1	1,0 1,0
Not give in	-3,-3 1,-3	1,0 1,0

Note that the agent’s utility functions are isomorphic to those in G₀. The only difference (highlighted by bold font) is that (Not give in, Threaten surrogate) is better for the principals (you and Alice) than the corresponding outcome (Not give in, Threaten surrogate) in G₀.

Let’s say G₂ simply consists in telling your agent to not worry so much about threats against you being carried out. Let’s represent this as follows:

	Threaten	Not threaten
Give in	0,1 0,1	1,0 1,0
Not give in	-1,-3 -3,-3	1,0 1,0

Finally, let’s say G₃ is a trivial game in which you and Alice always receive a payoff of ⅓ each.

Clearly, you should now prefer that your agents play G₁ rather than G₀. Following the SPI paper, we can offer straightforward (and presumably uncontroversial) justifications of this point. If furthermore you’re sufficiently “anchored” on the default G₀, then G₁ is your unique best option from what I’ve told you. For example, if you want to ensure low worst-case regret relative to the default you might be compelled to choose G₁. (And again, if for some reason you were barred from choosing G₂, G₃, then you would know that you should choose G₁ rather than G₀.) The SPI paper mostly assumes this perspective of being anchored on the default.

But now let’s assume that you approach the choice between G₀, G₁, G₂, and G₃ from the perspective of expected utility maximization. You just want to choose whichever of the six games is best. Then the above argument shows that G₀ should not be played. But it fails to give a positive account of which of the four games you should choose. In particular, it doesn’t make much of a positive case for G₁, other than that G₁ should be preferred over G₀. From an expected utility perspective, the “default” isn’t necessarily meaningful and there is no reason to play an SPI on the default.

In some sense, you might even say that SPIs are self-defeating: They are justified by appeals to a default, but then they themselves show that the default is a bad choice and therefore irrelevant (in the way that strictly dominated actions are irrelevant).

The problem is that we do want a positive account of, say, surrogate goals. That is, we would like to be able to say in some settings that you should adopt a surrogate goal. The point of surrogate goals is not just a negative claim about aligned delegation. Furthermore, in practice whenever we would want to use safe Pareto improvements or surrogate goals, we typically have lots of options, i.e., our situation is more like the G₀ versus … versus G₃ case than a case of a binary choice (G₀ versus G₁). That is, in practice we also have lots of other options that stand in no SPI relation to the default (and for which no SPI on the default is an SPI on them). (For example, we don’t face a binary choice between aligned delegation with and without surrogate goal. Within either of these options we have all the usual parameters – “alignment with whom?”, choice of methods, hyperparameters, and so on. We might also have various third options, such as the adoption of different objectives, intervening on the agent’s bargaining strategy, etc.) (Of course, there might also be multiple SPIs on the default. See Section 6 of the SPI paper and the discussion of the SPI selection problem in the “related issues” section below.)

For the purpose of this post, I’ll refer to this as the SPI justification gap. In the example of surrogate goals, the justification gap is the gap between the following propositions:

A) We should prefer the implementation of surrogate goals over aligned delegation.

B) We should implement surrogate goals.

The SPI framework justifies A very well (under various assumptions), including from an expected utility maximization perspective. In many settings, one would think B is true. But unless we are willing to adopt a “default-relative” decision making framework (like minimizing worst-case regret relative to the default), B doesn’t follow from A and the SPI framework doesn’t justify B. So we don’t actually have a compelling formal argument for adopting surrogate goals and in particular we don’t know what assumptions (if any) are needed for surrogate goals to be fully compelling (assuming that surrogate goals are compelling at all).

Now, one immediate objection might be: Why do we even care about whether an SPI is chosen? If there’s something even better than an SPI (in expectation), isn’t that great? And shouldn’t we be on the lookout for interventions on the agents’ interaction that are even better than SPIs? The problem with openness to non-SPIs is basically the problem of equilibrium selection. If we narrow our decision down to SPIs, then we only have to choose between SPIs. This problem might be well-behaved in various ways. For instance, in some settings, the SPI is unique. In other settings even the best-for-Alice SPI (among SPIs that cannot be further safely Pareto-improved upon) might be very good for you. Cf. Section 6 on “SPI selection problem section” in the original SPI paper. Meanwhile, if we don’t narrow our interventions on G down to SPIs on G, then in many cases the choice between interventions on G is effectively just choosing a strategy for G, which may be difficult. In particular, it may be that if we consider all interventions on G, then we also need to consider interventions that we only very slightly benefit from. For example, in the case of games with risks of conflict (like the Demand Game in the original SPI paper), all SPIs that cannot be further safely improved upon eliminate conflict outcomes (and don’t worsen the interaction in other ways). Whereas, non-safe Pareto improvements (i.e., an intervention that is good for both players in expectation relative to some probabilistic beliefs about game play) may also entail a higher probability of conceding more (in exchange for a decreased risk of conflict).

Below, I will first relate the justification gap problem to some other issues in the SPI framework. I will then discuss a few approaches to addressing the SPI justification gap. Generally, I think compelling, practical arguments can be made to bridge the SPI justification gap. But it would be great if more work could be done to provide more rigorous arguments and conditions under which these arguments do and don’t apply.

I should note that in many real-world cases, decisions are not made by expected utility maximization (or anything very close to it) anyway. So, even if the justification gap cannot be bridged at all, SPIs could remain relevant in many cases. For instance, in many cases interventions have to be approved unanimously (or at least approximately unanimously) by multiple parties with different values and different views about equilibrium selection. If no intervention is approved, then the default game is played. In such cases, SPIs (or approximate SPIs) on the default may be the only interventions that have a good chance at universal approval. For instance, an international body might be most likely to adopt laws that are beneficial to everyone while not changing the strategic dynamics much (say, rules pertaining to the humane treatment of prisoners of war). (In comparison, they might be less likely to adopt improvements that change the strategic dynamics – say, bans of nuclear weapons – because there will likely be someone who is worried that they might be harmed by such a law.) Another real-world example is that of “grandfathering in” past policies. For instance, imagine that an employer changes some of its policies for paying travel expenses (e.g., from reimbursing item by item to paying a daily allowance in order to decrease administrative costs). Then the employer might worry that some of its employees will prefer the old policy. In practice, the employer may wish to not upset anyone and thus seek a Pareto improvement. In principle, it could try to negotiate with existing employees to find some arrangement that leaves them happy while still enabling the implementation of the new policy. (For instance, it could raise pay to compensate for the less favorable reimbursement policy.) But in many contexts, it may be easier to simply allow existing employees to stay on the old policy (“grandfathering in” the old policy), thus guaranteeing to some extent that every employee’s situation will improve.

Some related issues

For clarification, I’ll now discuss a few other issues and clarify how they relate to the SPI justification gap, as discussed in this post.

First, the SPI justification gap is part of a more general question: “Why SPIs/surrogate goals?” As described above, earlier work gives, in my view, a satisfactory answer to a different aspect of the “Why SPIs?” question. Specifically, SPIs are persuasive in case of binary comparison: if we have a choice between two options G and G’ and G’ is an SPI on G, then we have good enough reason to choose G’ over G. This post is specifically about an issue that arises when we move beyond binary comparisons. (More generally, it’s about issues that arise if we move beyond settings where one of the available options is an SPI on all the other available options.)

Second, the justification gap is a separate problem from the SPI selection problem (as discussed in Section 6 of the SPI paper). The SPI justification gap arises even if there is just one SPI (as in the above G₀,….,G₃ example). Conversely, if we can in some way bridge the SPI justification gap (i.e., if we can justify restricting attention to interventions that are SPIs), then the SPI selection problem remains. The problems also seem structurally different. The justification gap isn’t intrinsically about reaching multiplayer agreement in the way that SPI selection is. That said, some of the below touches on SPI selection (and equilibrium selection) here and there. Conversely, we might expect that approaches to SPI selection would also relate to SPI justification. For instance, some criteria for choosing between SPIs could also serve as criteria that favor SPIs over non-SPIs. For example, maximizing minimum utility improvement both induces (partial) preferences over SPIs and preferences for SPIs over non-SPIs.

Second, the safe Pareto improvement framework assumes that when deciding whether to replace the default with something else, we restrict attention to games that are better for both players. (This is a bit similar to imposing a “voluntary participation” or “individual rationality” constraint in mechanism design.) In the context of surrogate goals this is motivated by worries of retaliation against aggressive commitments. From the perspective of an expected utility maximizer, it’s unclear where this constraint comes from. Why not propose some G that is guaranteed to be better for you, but not guaranteed to be better for the other player than the default? This is a different question than the question we ask in this post. Put concisely, this post is about justifying the “safe” part, not about justifying the “Pareto” part. (At least in principle, we could consider “safe my-utility improvements” (which are explicitly allowed to be bad for the opponent relative to the default) and ask when we are justified in specifically pursuing these over other interventions. I’m here considering safe Pareto improvements throughout, because the concept of safe Pareto improvement is more widely known and discussed.) Nevertheless, I think the problem of justifying the Pareto constraint – the constraint that the other player is (“safely” or otherwise) not made worse off than the default – is similar and closely related to the SPI justification gap. Both are about a “safety” or “conservativeness” constraint that isn’t straightforwardly justifiable on expected-utility grounds. See also “Solution idea 3” below, which gives a justification of safety that is closely related to the Pareto requirement (and justifications thereof).

Solution idea 1: Pessimistic beliefs about one’s ability to choose non-SPIs

Consider again the choice between G₀,….,G₃, where G₁ is an SPI on G₀ and there are no other SPI relations. Probably the most straightforward justification for SPIs would be a set of beliefs under which the expected utility of G₀ is higher than the expected utility of any of the games G₂ and G₃. Specifically, you could have a belief like the following, where G₀ is the default: If I can’t prove from some specific set of assumptions (such as “isomorphic games are played isomorphically”, see the SPI paper) that u(G) > u(G₀), then I should believe E[u(G)] < E[u(G₀)]. (I might hold this either as a high-level belief or it might be that all my object-level beliefs satisfy this.) Of course, in some sense, this is just a belief version of decision principles like minimizing regret relative to the default. Michael Dennis et al. make this connection explicit in the appendix of “Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design” as well as some ongoing unpublished work on self-referential claims.

Intuitively, non-SPI pessimism is based on trust in the abilities of the agents that we’re delegating to. For instance, in the case of threats and surrogate goals, I might be pessimistic about telling my agent that the opponent is likely bluffing, because I might expect that my agent will have more accurate beliefs than me about whether the opponent is bluffing. (On the other hand, if I can tell my agent that my opponent is bluffing in a way that is observable to the opponent, then such announcements are a sort of credible commitment. And if I can commit in this way and my agent cannot commit, then it might very well be that I should commit, even if doing so in some sense interferes with my agent’s competence.) One might be pessimistic about G₂ for similar reasons.

I don’t think such pessimistic beliefs are plausible in general, however. One argument against “non-SPI pessimism” is that one can give “almost SPIs”, about which I would be optimistic, and about which we need to be optimistic if we wish to apply SPIs in the real world (where exact SPIs are hard to construct). For example, take the Demand Game example from the SPI paper and decrease Player 1’s payoffs of the outcome (DL,RL) by 0.00001. You can’t make a clean SPI argument anymore, but I’d still in practice be happy with the approach proposed by the SPI paper. Or take the above example (the choice between G₀, G₁, G₂, G₃), imagine that as a better version of G₃ you can also choose a guaranteed payoff of 0.99. If you want the Pareto part, you may imagine that Alice also receives 0.99 in this case. Rejecting this option seems to imply great optimism about G₀. (You need to believe that Alice’s agent will play Not Threaten with probability >99%.) Such optimism seems uncompelling. (We could construct alternative games G₀ in which such optimism seems even less compelling.) Maybe the example is even more powerful if we take a third party’s perspective that wants to improve both players’ utility (and imagine again that a certain payoff of 0.99 for both players is available). After all, if G₀ is better than 0.99 for one player, 0.99 is much better than G₀ for the other player.

Solution idea 2: Decision factorization

(I suspect this section is harder to read than the others. There seem to be more complications to these ideas than those in the other sections. I believe the approach in this section to be the most important, however.)

One intuitive justification for the application SPIs is that the decision of whether to use the SPI/surrogate goal can be made separately from other decisions. This is quite explicit in the original surrogate story: Commit to the surrogate goal now – you can still think about, say, whether you should commit to not give in later on. If right now you only decide whether to use a surrogate goal or not, then right now you face a binary decision where one option is an SPI on the other, which is conceptually unproblematic. If instead, you decide between all possible ways of instructing your agents at once, you have lots of options and the SPI concept induces only a partial order on these options. In some cases a decision may be exogenously factorized appropriately, e.g., you might be forced to only decide whether to use surrogate goals on day 1, and to make other decisions on day 2 (or even have someone else make other decisions on day 2). The challenge is that in most real-world cases, decisions aren’t externally factorized.

Here’s a toy model, inspired by surrogate goals. Let’s say that you play a game in which you choose two bits, so overall you have four actions. However, imagine that you choose the two bits in sequence. Perhaps when thinking about the first bit, you don’t yet know what second bit you will choose. Let’s say that each sequence of bits in turn gives rise to some sort of game – call these games G₀₀, G₀₁, G₁₀, and G₁₁, where the first bit encodes the first choice and the second encodes the second. Intuitively you might imagine that the first choice is the choice of whether to (immediately) adopt a surrogate goal, and the second choice is whether to make some commitment to resist unfair outcomes. Importantly, you need to imagine that if you adopt the surrogate goal, then your choice on the second day will be made on the basis of your new utility function, the one that includes the surrogate goal. Anyway, you don’t need to have such a specific picture in mind.

Now, let’s say that you find that G₁₀ is an SPI on G₀₀ and that G₁₁ is an SPI on G₀₁. Imagine further that the choice of the second bit is isomorphic between the case where the first bit chosen was 1 and the case where the second bit chosen was 0, that is: in G₁ (the state of affairs where the first bit was chosen to be 1 and the second bit is not yet chosen) you will choose 0 if and only if you choose 0 in G₀. Then overall, you can conclude that G₁ is an SPI on G₀. Since in the first time step you only choose between 0 and 1, you can conclude by SPI logic from the above that you should choose 1 at the first time step. (Of course, the decision in G₁ will still be difficult.)

Here’s a non-temporal informal example of how factorization might work. Imagine that AliceCorp is building an AI system. One of the divisions of AliceCorp is tasked with designing the bargaining behavior of the AI system (e.g., by training the AI system on some set of bargaining scenarios). A different “SPI division” is tasked with assessing whether to modify the system’s utility function (e.g., by modifying the training rewards) to include surrogate goals. This SPI division is explicitly not supposed to induce a utility function that causes the system to give in less – influencing the bargaining policy is the job of the bargaining division after all. Importantly, the bargaining division either doesn’t know about the possibility of surrogate goals or doesn’t care about it. (For example, imagine that the bargaining division is paid in proportion to how well the AI system ends up handling bargaining situations in the real world. Then it is important that if a surrogate goal is implemented, the bargaining division receives low incentive pay if a surrogate threat is carried out.) If AliceCorp happens to be set up in this way, then – ignoring SPI selection, loads of implementation details, etc. – the SPI division faces a binary choice between implementing the SPI and not implementing the SPI.

Again, the main idea here is that we factorize our decision into multiple decisions and then we consider these decisions separately. In particular, the decision of whether to adopt the SPI or not becomes an easy binary decision.

Note that the factorization argument requires some structure on the set of possible games that might be played. The structure is what allows you to make the decision about the SPI in isolation, without considering your other choices. (For instance, adopting a surrogate goal doesn’t stop you from committing to never give in later on.) The argument doesn’t work if you just choose between four arbitrary games (as in the G₀ versus … G₃ case), one of which is an SPI on one of the other ones.

Also, note that depending on the details of the settings, the separation of the different decisions seems to be necessary to make the story compelling. For example, in the surrogate goal story, it’s important to first adopt surrogate goals and only then decide whether to make other commitments. If you, say, self-modify all at once to have a surrogate goal and to never give in, then the opponent should treat this as if you had just committed to never give in without the surrogate goal. There are other cases in which the order does not matter. For example, imagine that you choose 0 versus 1 and also choose between A, B, and C. Then you play some game whose payoffs are determined in some complex way by the A/B/C choice. Additionally, the first number (0 vs. 1) is simply added to each payoff in the resulting game. Choosing 1 is an SPI over choosing 0, and there’s no particular problem with choosing 0 versus 1 and A versus B versus C at the same time. (And there’s also no need, it would seem, to have separate “departments” make the two different choices.) So in this case it doesn’t matter whether the factorization is somehow (e.g., temporarily) (self-)imposed on the decision maker. It still matters that the factorization exists, though. For instance, if choosing the first number to be 1 precludes choosing the letter A, then we can’t necessarily make a case for choosing 1 over 0 anymore.

In a recent draft, DiGiovanni et al. (2024) provide a justification of safe Pareto improvements from an expected utility maximization standpoint. (See also their less technical blog post: “Individually incentivized safe Pareto improvements in open-source bargaining”.) I think, and Anthony DiGiovanni tentatively agrees, that one of its underlying ideas can be viewed as factorizing the choice set (which in their setting is the set of computer programs). In particular, their Assumptions 5 and 9 each enable us to separate out the choice of “renegotiation function” (their mechanism for implementing SPIs) from other decisions (e.g., how hawkish to be), so that we can reach conclusions of the form, “whatever we choose for the ‘other stuff’, it is better to use a renegotiation function” (see their Proposition 2 and Theorem 3). Importantly their results don’t require the factorization to manifest in, say, the temporal order of decisions. So in that way it’s most similar to the example at the end of the preceding paragraph. They also allow the choice of the “other stuff” to depend on the presence of negotiation, which is different from all examples discussed in this post. Anyway, there’s a lot more to their paper than this factorization and I myself still have more work to do to get a better understanding of it. Therefore, a detailed discussion of this paper is beyond the scope of this post, and the discussion in this section is mostly naive to the ideas in their paper.

I like the factorization approach. I think this is how I usually think of surrogate goals working in practice. But I nonetheless have many questions about how well this justification works. Generally, the story assumes many aspects about the agent’s reasoning as given and I’d be interested in whether these assumptions could be taken to be variable. Here are some specific concerns.

1) In some cases, the story assumes that decisions are factorized and/or faced in some fixed specific order that is provided exogenously. Presumably, the SPI argument doesn’t work in all orders. For example, in the “two-bits story” let’s say you first have to choose {11,00} versus {10,01} and then choose within the respective set (as opposed to first choosing {10,11} versus {00,01} and then within the chosen set). Then neither of the two decisions individually can be resolved by an SPI argument. What if the agent can choose herself in what order to think about the different decisions?

There are lots of ideas to consider here. For example, maybe agents should first consider all possible ordered factorizations of their decision and then use a factorization that allows for an SPI. But this all seems non-trivial to work out and justify. For instance, in order to avoid infinite regress, many possible lines of meta-reasoning have to be suppressed. But then we need a justification for why this particular form of meta-reasoning (how can I decompose my decision problems in the best possible way?) should not be suppressed. Also, how do we arrive at the judgment that the SPI-admitting factorization is best? It seems that this to some extent presupposes some kind of pro-SPI judgment.

2) Relatedly, the argument hinges to some extent on the assumption that there is some point in time at which the opportunity to implement the SPI expires (while relevant other decisions – the ones influenced by the SPI – might still be left to be made later). For example, in the two-bit story, the SPI can only be implemented on the first day, and an important decision is made on the second day. In some real-world settings this might be realistic.* But in some cases, it might be unrealistic. That is, in some cases the decision of whether to adopt the SPI can be delayed for as long as we want.

As an example, consider the following variant of the two-bits story. First, imagine that in order for choosing the first bit to be 1 to be an SPI over choosing it to be 0, we do need to choose the bits sequentially. For instance, imagine that the first bit determines whether the agent implements a surrogate goal and the second bit determines whether the agent commits to be a more aggressive bargainer. Now imagine that the agent has 100 time steps. (Let’s assume that the setting doesn’t have any “commitment race”-type dynamics – there’s no benefit to commit to an aggressive strategy early in order to make it so that the opponent sees your commitment before they make a commitment. E.g., you might imagine that a report of your commitments is only sent to opponents after all 100 time steps have passed.) In each time step, the agent gets to think a little. It can then make some commitment about the two bits. For example, it might commit to not choose 11, or commit to choose 1 as the first bit. Intuitively, we want the agent to at some point choose the first bit, and then choose the second bit in the last step. Perhaps in this simple toy model you might specifically want the agent to choose the first bit immediately, but in most practical settings, figuring out whether a surrogate goal could be used and figuring out how to best apply it takes some time. In fact, one would think that in practice there’s almost always some further thinking that could be done to improve the surrogate goal implementation. If there’s SPI selection, then you might want to wait to decide what SPI to go for. This would suggest that the agent would push the surrogate goal implementation back to the 99th time step. (Because sequential implementation is required, you cannot push implementation back to the 100th time step – on the 100th time step, you need to choose whether to commit to an aggressive bargaining policy while already having adopted the surrogate goal.)

The problem is that if you leave the SPI decision to the final time step, this leaves very little time for making the second decision (the decision of whether to adopt an aggressive bargaining policy). (You might object: “Can’t the agent think about the two decisions in parallel?” But one big problem with this is that before adopting a surrogate goal (before setting the first bit to be 1), the agent has different goals than it has after adopting the surrogate goal. So, if the agent anticipates the successful implementation of surrogate goals, it has little interest in figuring out whether future threats would be credible. In general, the agent would want to think thoughts that make its future self less likely to give in. (No, this doesn’t violate the law of total probability.))

One perspective on the above is that even once you (meta-)decided to first decide whether or how to implement the surrogate goal, you still face a ternary choice at each point: A) commit to the surrogate goal; B) commit against the use of the surrogate goal; and C) wait and decide A versus B later. The SPI framework tells you that A > B. But it can’t compare A and C.

Solution idea 3: Justification from uncertainty about one another’s beliefs

Here’s a third idea. Contrary to the other two approaches, this one hinges much more on the multi-agent aspect of SPIs. Roughly, this justification for playing SPIs on the default is as follows: Let’s say it’s common knowledge that the default way of playing is good (relative to a randomly selected intervention, for instance). Also, if G’ is an SPI on G, then by virtue of following from a simple set of (chosen-to-be-)uncontroversial assumptions, it is common knowledge that G’ is better than G. So, everyone agrees that G’ would be better for everyone than G, and everyone knows that everyone else agrees that G’ would be better for everyone than G, and so on. In contrast, if G’ is not an SPI on G and your belief that G’ is better than G is based on forming beliefs about how the agents play G (which might have many equilibria), then you don’t know for sure whether the other players would agree with this judgment or not. This is an obstacle to the use of “unsafe Pareto improvements” for a number of reasons, which I will sketch below. Compare the Myerson–Satterthwaite theorem, which suggests that Pareto-improving deals often fail to be made when the parties have access to private information about whether any given deal is Pareto-improving. Relative to this theorem, the following points are pre-theoretical.

• First and most straightforwardly, if you (as one of the principals) know that G’ is better for both you and Alice than G, but you don’t know whether Alice (the other principal) knows that G’ is better (for her) than G, then – in contexts where Alice has some sort of say – you will worry that Alice will oppose G’. If there’s a restriction on the number of times that you can propose an intervention, this is an argument against proposing G’ from an expected utility maximization perspective.
  
  Here’s an extremely simple numeric example. Let’s say you get to propose one intervention and Alice has to accept or reject. If she rejects, then G is played. (Note that we thereby already avoid part of the justification problem – the default G is, to some extent, imposed externally in a way that it normally would not be.) Now let’s say that G’ is an SPI on G. Let’s say that you also think that some other G’’ is better in expectation for you than G, but you think there’s a 20% chance that Alice will disagree that G’’ is better than G. Then for you to propose G’’ rather than G’, it would need to be the case that 80% * (u(G’’) – u(G)) > u(G’) – u(G’’). (This assumes that you don’t take Alice’s judgment as evidence about how good G’’ is for you.) Thus, even if the SPI G’ is slightly worse for you than the non-SPI G’’, you might propose the SPI G’, because it is more likely to be accepted.
• As an extension of the above, you might worry that the other player will worry that any proposed non-SPI was selected adversarially. That is, if you propose playing G’ instead of G to Alice and G’ is not an SPI on G, then Alice might worry that you only proposed G’ because it disproportionately favors you at her cost. That is, Alice might worry that you have private information about the agents and that this information tells you that G’ is specifically good for you in a way that might be bad for her.
  
  Here’s again a toy example. Let’s imagine again that G is in fact played by default (which, again, assumes part of the justification gap away). Imagine that there is exactly one SPI on G. Let’s say that additionally there are 1000 other possible games G₁,…,G₁₀₀₀ to play that all look like they’re slightly better in expectation for both players than G. Finally, imagine that you receive private information that implies that one of the 1000 games is very good for you and very bad for the other player. Now let’s say that you’ve specifically observed that G₂₃₃ is the game that is great for you and bad for your opponent. If you now propose that you and the opponent play G₂₃₃, your opponent can’t tell that you’re trying to trick them. Conversely, if you propose G₈₁₆, then your opponent doesn’t know that you’re not trying to trick them. With some caveats, standard game-theoretic reasoning would therefore suggest that if you propose any of G₁,…,G₁₀₀₀, you ought to propose G₂₃₃. Knowing this, the other principal will reject the proposal. Thus, you know that proposing any of G₁,…,G₁₀₀₀ will result in playing G. If instead you propose the SPI on G, then the other principal has no room for suspicion and no reason to reject. Therefore, you should propose G.
  
  There are many complications to this story. For example, in the above scenario, the principals could determine at random which of the G_i to play. So if they have access to some trusted third party who can randomize for them, they could generate l at random and then choose to play G_i. (They could also use cryptography to “flip a coin by telephone” if they don’t have access to a trusted third party.) But then what if the games in G₁,…,G₁₀₀₀ aren’t Pareto improvements on average, and instead the agents also have private information about which of these games are (unsafe) Pareto improvements? There are lots of questions to explore here, too many to discuss in this post.
• As a third consideration, agents might worry that in the process of settling on a non-SPI, they reveal private information, and that revealing private information might have downsides. For example, if the default G is a Game of Chicken, then indicating willingness to “settle” for some fixed amount reveals a (lower bound on the) belief in the opposing agent’s hawkishness. It’s bad for you to reveal yourself to believe in the opponent’s hawkishness.
  
  Again, there are lots of possible complications, of course. For instance, zero-knowledge mechanisms could be used to bargain in a way that requires less information to be exchanged.

Solution idea 4: Safe Pareto improvements as Schelling points

To end on a simpler idea: Safe Pareto improvements may be Schelling points (a.k.a. focal points) in some settings. For instance, you might imagine a setting in which the above justifications don’t quite apply. Then SPIs might stand out by virtue of standing out in slight alterations of the setting in which the above justifications do apply.

It seems relatively hard to say anything about the Schelling points justification of safe Pareto improvements, because it’s hard to justify preferences for Schelling points in general. For instance, imagine you and Bob have to both pick a number between 1 and 10 and you both get a reward if you both pick the same number. Perhaps you should pick 1 (or 10) because it’s the smallest (resp. largest) number. If you know each other to be Christians, perhaps you should pick 3. Perhaps you should pick 7, because 7 is the most common favorite number. If you just watched a documentary about Chinese culture together, perhaps you should pick 8, because 8 is a lucky number in China. And so on. I doubt that there is a principled answer as to which of these arguments matters most. Similarly I suspect that in complex settings (in which similarly many Schelling point arguments can be made), it’s unclear whether SPIs are more “focal” than non-SPIs.

Acknowledgments

Discussions with Jesse Clifton, Anthony DiGiovanni and Scott Garrabrant and inspired parts of the contents of this post. I also thank Tobias Baumann, Vincent Conitzer, Emery Cooper, Michael Dennis, Alexander Kastner and Vojta Kovarik for comments on this post.

* Arguing for this is beyond the scope of this post, but just to give a sense: Imagine that AliceCorp builds an AI system. AliceCorp has 1000 programmers and getting the AI to do anything generally requires a large effort by people across the company. Now I think AliceCorp can make lots of credible announcements about what AliceCorp is and isn’t doing. For example, some of the programmers might leak information to the public if AliceCorp were to lie about how it operates. However, once AliceCorp has a powerful AI system, it might lose the ability to make some forms of credible commitments. For example, it might be that AliceCorp can now perform tasks by having a single programmer work in collaboration with AliceCorp’s AI system. Since it’s much easier to find a single programmer who can be trusted not to leak (and presumably the AI system can be made not to leak information), it’s now much easier for AliceCorp to covertly implement projects that go against AliceCorp’s public announcements.

(This post assumes that the reader is fairly familiar with the decision theory of Newcomb-like problems. Schwarz makes many of the same points in his post “On Functional Decision Theory” (though I disagree with him on other things, such as whether to one-box or two-box in Newcomb’s problem). Similar points have also been made many times about the concept of updatelessness in particular, e.g., see Section 7.3.3 of Arif Ahmed’s book “Evidence, Decision and Causality”, my post on updatelessness from a long time ago, or Sylvester Kollin’s “Understanding updatelessness in the context of EDT and CDT”. Preston Greene, on the other hand, argues explicitly for a view opposite of the one in this post in his paper “Success-First Decision Theories”. ETA: Similar points have also been made w.r.t. the “Why ain’cha rich?” argument, which is an argument for EDT (and other one-boxing theories) and against CDT based on Newcomb’s problem. See, for example, Bales (2018) and Wells (2019) (thanks to Sylvester Kollin and Miles Kodama, respectively, for pointing out these papers).)

I sometimes read the claim that one decision theory “outperforms” some other decision theory (in general or in a particular problem). For example, Yudkowsky and Soares (2017) write: “FDT agents attain high utility in a host of decision problems that have historically proven challenging to CDT and EDT: FDT outperforms CDT in Newcomb’s problem; EDT in the smoking lesion problem; and both in Parfit’s hitchhiker problem.” Others use some variations of this framing (“dominance”, “winning”, etc.), some of which I find less dubious because they have less formal connotations.

Based on typical usage, these words make it seem as though there was some agreed upon or objective metric to compare decision theories in any particular problem and that MIRI is claiming to have found a theory that is better according to that metric (in some given problems). This would be similar to how one might say that one machine learning algorithm outperforms another on the CIFAR dataset, where everyone agrees that ML algorithms are better if they correctly classify a higher percentage of the images, require less computation time, fewer samples during training, etc.

However, there is no agreed-upon metric to compare decision theories, no way to asses even for a particular problem whether one decision theory (or its recommendation) does better than another. (This is why the CDT-versus-EDT-versus-other debate is at least partly a philosophical one.) In fact, it seems plausible that finding such a metric is “decision theory-complete” (to butcher another term with a specific meaning in computer science). By that I mean that settling on a metric is probably just as hard as settling on a decision theory and that mapping between plausible metrics and plausible decision theories is fairly easy.

For illustration, consider Newcomb’s problem and a few different metrics. One possible metric is what one might call the causal metric, which is the expected payoff if we were to replace the agent’s action with action X by some intervention from the outside. Then, for example, in Newcomb’s problem, two-boxing “performs” better than one-boxing and CDT “outperforms” FDT. I expect that many causal decision theorists would view something of this ilk as the right metric and that CDT’s recommendations are optimal according to the causal metric in a broad class of decision problems.

A second possible metric is the evidential one: given that I observe that the agent uses decision theory X (or takes action Y) in some given situation, how big of a payoff do I expect the agent to receive. This metric directly favors EDT in Newcomb’s problem, the smoking lesion, and again a broad class of decision problems.

A third possibility is a modification of the causal metric. Rather than replacing the agent’s decision, we replace its entire decision algorithm before the predictor looks at and creates a model of the agent. Despite being causal, this modification favors decision theories that recommend one-boxing in Newcomb’s problem. In general, the theory that seems to maximize this metric is some kind of updateless CDT (cf. Fisher’s disposition-based decision theory). 

Yet another causalist metric involves replacing from the outside the decisions of not only the agent itself but also of all agents that use the same decision procedure. Perhaps this leads to Timeless Decision Theory or Wolfgang Spohn’s proposal for causalist one-boxing.

One could also use the notion of regret (as discussed in the literature on multi-armed bandit problems) as a performance measure, which probably leads to ratificationism.

Lastly, I want to bring up what might be the most commonly used class of metrics: intuitions of individual people. Of course, since intuitions vary between different people, intuition provides no agreed upon metric. It does, however, provide a non-vacuous (albeit in itself weak) justification for decision theories. Whereas it seems unhelpful to defend CDT on the basis that it outperforms other decision theories according to the causal metric but is outperformed by EDT according to the evidential metric, it is interesting to consider which of, say, EDT’s and CDT’s recommendations seem intuitively correct.

Given that finding the right metric for decision theory is similar to the problem of decision theory itself, it seems odd to use words like “outperforms” which suggest the existence or assumption of a metric.

I’ll end with a few disclaimers and clarifications. First, I don’t want to discourage looking into metrics and desiderata for decision theories. I think it’s unlikely that this approach to discussing decision theory can resolve disagreements between the different camps, but that’s true for all approaches to discussing decision theory that I know of. (An interesting formal desideratum that doesn’t trivially relate to decision theories is discussed in my blog post Decision Theory and the Irrelevance of Impossible Outcomes. At its core, it’s not really about “performance measures”, though.)

Second, I don’t claim that the main conceptual point of this post is new to, say, Nate Soares or Eliezer Yudkowsky. In fact, they have written similar things, see, for instance, Ch. 13 of Yudkowsky’s Timeless Decision Theory, in which he argues that decision theories are untestable because counterfactuals are untestable. Even in the aforementioned paper, claims about outperforming are occasionally qualified. E.g., Yudkowsky and Soares (2017, sect. 10) say that they “do not yet know […] (on a formal level) what optimality consists in”.) Unfortunately, most outperformance claims remain unqualified. The metric is never specified formally or discussed much. The short verbal descriptions that are given make it hard to understand how their metric differs from the metrics corresponding to updateless CDT or updateless EDT.

So, my complaint is not so much about these authors’ views but about a Motte and Bailey-type inconsistency, in which the takeaways from reading the paper superficially are much stronger than the takeaways from reading the whole paper in-depth and paying attention to all the details and qualifications. I’m worried that the paper gives many casual readers the wrong impression. For example, gullible non-experts might get the impression that decision theory is like ML in that it is about finding algorithms that perform as well as possible according to some agreed-upon benchmarks. Uncharitable, sophisticated skim-readers may view MIRI’s positions as naive or confused about the nature of decision theory.

In my view, the lack of an agreed-upon performance measure is an important fact about the nature of decision theory research. Nonetheless, I think that, e.g., MIRI is doing and has done very valuable work on decision theory. More generally I suspect that being wrong or imprecise about this issue (that is, about the lack of performance metrics in the decision theory of Newcomb-like problems) is probably not an obstacle to having good object-level ideas. (Similarly, while I’m not a moral realist, I think being a moral realist is not necessarily an obstacle to saying interesting things about morality.)

Acknowledgement

This post is largely inspired by conversations with Johannes Treutlein. I also thank Emery Cooper for helpful comments.

[I assume that the reader is familiar with Newcomb’s problem and the Prisoner’s Dilemma against a similar opponent and ideally the equilibrium selection problem in game theory.]

The trust dilemma (a.k.a. Stag Hunt) is a game with a payoff matrix kind of like the following:

	S	H
S	4,4	-1,3
H	3,-1	2,2

Its defining characteristic is the following: The Pareto-dominant outcome (i.e., the outcome that is best for both players) (S,S) is Nash equilibrium. However, (H,H) is also a Nash equilibrium. Moreover, if you’re sufficiently unsure what your opponent is going to do, then H is the best response. If two agents learn to play this game and they start out playing the game at random, then they are more likely to converge to (H,H). Overall, we would like it if the two agents played (S,S), but I don’t think we can assume this to happen by default.

Now what if you played the trust dilemma against a similar opponent (specifically one that is similar w.r.t. how they play games like the trust dilemma)? Clearly, if you play against an exact copy, then by the reasoning behind cooperating in the Prisoner’s Dilemma against a copy, you should play S. More generally, it seems that a similarity between you and your opponent should push towards trusting that if you play S, the opponent will also play S. The more similar you and your opponent are, the more you might reason that the decision is mostly between (S,S) and (H,H) and the less relevant are (S,H) and (H,S).

What if you played against an opponent who knows you very well and who has time to predict how you will choose in the trust dilemma? Clearly, if you play against an opponent who can perfectly predict you (e.g., because you are an AI system and they have a copy of your source code source code), then by the reasoning behind one-boxing in Newcomb’s problem, you should play S. More generally, the more you trust your opponent’s ability to predict what you do, the more you should trust that if you play S, the opponent will also play S.

Here’s what I find intriguing about these scenarios. In these scenarios, one-boxers might systematically arrive at a different (more favorable) conclusion than two-boxers. However, this conclusion is still compatible with two-boxing, or with blindly applying Nash equilibrium. In the trust dilemma, one-boxing type reasoning merely affects how we resolve the equilibrium selection problem, which the orthodox theories generally leave open. This is in contrast to the traditional examples (Prisoner’s Dilemma, Newcomb’s problem) in which the two ways of reasoning are in conflict. So there is room for implications of one-boxing, even in an exclusively Nash equilibrium-based picture of strategic interactions.

“Abstract”: Some have claimed that moral realism – roughly, the claim that moral claims can be true or false – would, if true, have implications for AI alignment research, such that moral realists might approach AI alignment differently than moral anti-realists. In this post, I briefly discuss different versions of moral realism based on what they imply about AI. I then go on to argue that pursuing moral-realism-inspired AI alignment would bypass philosophical and help resolve non-philosophical disagreements related to moral realism. Hence, even from a non-realist perspective, it is desirable that moral realists (and others who understand the relevant realist perspectives well enough) pursue moral-realism-inspired AI alignment research.

Different forms of moral realism and their implications for AI alignment

Roughly, moral realism is the view that “moral claims do purport to report facts and are true if they get the facts right.” So for instance, most moral realists would hold the statement “one shouldn’t torture babies” to be true. Importantly, this moral claim is different from a claim about baby torturing being instrumentally bad given some other goal (a.k.a. a “hypothetical imperative”) such as “if one doesn’t want to land in jail, one shouldn’t torture babies.” It is uncontroversial that such claims can be true or false. Moral claims, as I understand them in this post, are also different from descriptive claims about some people’s moral views, such as “most Croatians are against babies being tortured” or “I am against babies being tortured and will act accordingly”. More generally, the versions of moral realism discussed here claim that moral truth is in some sense mind-independent. It’s not so obvious what it means for a moral claim to be true or false, so there are many different versions of moral realism. I won’t go into more detail here, though we will revisit differences between different versions of moral realism later. For a general introduction on moral realism and meta-ethics, see, e.g., the SEP article on moral realism.

I should note right here that I myself find at least “strong versions” of moral realism implausible. But in this post, I don’t want to argue about meta-ethics. Instead, I would like to discuss an implication of some versions of moral realism. I will later say more about why I am interested in the implications of a view I believe to be misguided, but for now suffice it to say that “moral realism” is a majority view among professional philosophers (though I don’t know how popular the versions of moral realism studied in this post are), which makes it interesting to explore the view’s possible implications.

The implication that I am interested in here is that moral realism helps with AI alignment in some way. One very strong version of the idea is that the orthogonality thesis is false: if there is a moral truth, agents (e.g., AIs) that are able to reason successfully about a lot of non-moral things will automatically be able to reason correctly about morality as well and will then do what they infer to be morally correct. On p. 176 of “The Most Good You Can Do”, Peter Singer defends such a view: “If there is any validity in the argument presented in chapter 8, that beings with highly developed capacities for reasoning are better able to take an impartial ethical stance, then there is some reason to believe that, even without any special effort on our part, superintelligent beings, whether biological or mechanical, will do the most good they possibly can.” In the articles “My Childhood Death Spiral”, “A Prodigy of Refutation” and “The Sheer Folly of Callow Youth” (among others), Eliezer Yudkowsky says that he used to hold such a view.

Of course, current AI techniques do not seem to automatically include moral reasoning. For instance, if you develop an automated theorem prover to reason about mathematics, it will not be able to derive “moral theorems”. Similarly, if you use the Sarsa algorithm to train some agent with some given reward function, that agent will adapt its behavior in a way that increases its cumulative reward regardless of whether doing so conflicts with some ethical imperative. The moral realist would thus have to argue that in order to get to AGI or superintelligence or some other milestone, we will necessarily have to develop new and very different reasoning algorithms and that these algorithms will necessarily incorporate ethical reasoning. Peter Singer doesn’t state this explicitly. However, he makes a similar argument about human evolution on p. 86f. in ch. 8:

The possibility that our capacity to reason can play a critical role in a decision to live ethically offers a solution to the perplexing problem that effective altruism would otherwise pose for evolutionary theory. There is no difficulty in explaining why evolution would select for a capacity to reason: that capacity enables us to solve a variety of problems, for example, to find food or suitable partners for reproduction or other forms of cooperative activity, to avoid predators, and to outwit our enemies. If our capacity to reason also enables us to see that the good of others is, from a more universal perspective, as important as our own good, then we have an explanation for why effective altruists act in accordance with such principles. Like our ability to do higher mathematics, this use of reason to recognize fundamental moral truths would be a by-product of another trait or ability that was selected for because it enhanced our reproductive fitness—something that in evolutionary theory is known as a spandrel.

A slightly weaker variant of this strong convergence moral realism is the following: Not all superintelligent beings would be able to identify or follow moral truths. However, if we add some feature that is not directly normative, then superintelligent beings would automatically identify the moral truth. For example, David Pearce appears to claim that “the pain-pleasure axis discloses the world’s inbuilt metric of (dis)value” and that therefore any superintelligent being that can feel pain and pleasure will automatically become a utilitarian. At the same time, that moral realist could believe that a non-conscious AI would not necessarily become a utilitarian. So, this slightly weaker variant of strong convergence moral realism would be consistent with the orthogonality thesis.

I find all of these strong convergence moral realisms very implausible. Especially given how current techniques in AI work – how value-neutral they are – the claim that algorithms for AGI will all automatically incorporate the same moral sense seems extraordinary and I have seen little evidence for it¹ (though I should note that I have read only bits and pieces of the moral realism literature).²

It even seems easy to come up with semi-rigorous arguments against strong convergence moral realism. Roughly, it seems that we can use a moral AI to build an immoral AI. Here is a simple example of such an argument. Imagine we had an AI system that (given its computational constraints) always chooses the most moral action. Now, it seems that we could construct an immoral AI system using the following algorithm: Use the moral AI to decide which action of the immoral AI system it would prevent from being taken if it could only choose one action to be prevented. Then take that action. There is a gap in this argument: perhaps the moral AI simply refuses to choose the moral actions in “prevention” decision problems, reasoning that it might currently be used to power an immoral AI. (If exploiting a moral AI was the only way to build other AIs, then this might be the rational thing to do as there might be more exploitation attempts than real prevention scenarios.) Still (without having thought about it too much), it seems likely to me that a more elaborate version of such an argument could succeed.

Here’s a weaker moral realist convergence claim about AI alignment: There’s moral truth and we can program AIs to care about the moral truth. Perhaps it suffices to merely “tell them” to refer to the moral truth when deciding what to do. Or perhaps we would have to equip them with a dedicated “sense” for identifying moral truths. This version of moral realism again does not claim that the orthogonality thesis is wrong, i.e. that sufficiently effective AI systems will automatically behave ethically without us giving them any kind of moral guidance. It merely states that in addition to the straightforward approach of programming an AI to adopt some value system (such as utilitarianism), we could also program the AI to hold the correct moral system. Since pointing at something that exists in the world is often easier than describing that thing, it might be thought that this alternative approach to value loading is easier than the more direct one.

I haven’t found anyone who defends this view (I haven’t looked much), but non-realist Brian Tomasik gives this version of moral realism as a reason to discuss moral realism:

Moral realism is a fun philosophical topic that inevitably generates heated debates. But does it matter for practical purposes? […] One case where moral realism seems problematic is regarding superintelligence. Sometimes it’s argued that advanced artificial intelligence, in light of its superior cognitive faculties, will have a better understanding of moral truth than we do. As a result, if it’s programmed to care about moral truth, the future will go well. If one rejects the idea of moral truth, this quixotic assumption is nonsense and could lead to dangerous outcomes if taken for granted.

(Below, I will argue that there might be no reason to be afraid of moral realists. However, my argument will, like Brian’s, also imply that moral realism is worth debating in the context of AI.)

As an example, consider a moral realist view according to which moral truth is similar to mathematical truth: there are some axioms of morality which are true (for reasons I, as a non-realist, do not understand or agree with) and together these axioms imply some moral theory X. This moral realist view suggests an approach to AI alignment: program the AI to abide by these axioms (in the same way as we can have automated theorem provers assume some set of mathematical axioms to be true). It seems clear that something along these lines could work. However, this approach’s reliance on moral realism is also much weaker.

As a second example, divine command theory states that moral truth is determined by God’s will (again, I don’t see why this should be true and how it could possibly be justified). A divine command theorist might therefore want to program the AI to do whatever God wants it to do.

Here are some more such theories:

• Social contract
• Habermas’ discourse ethics
• Universalizability / Kant’s categorical imperative
• Applying human intuition

Besides pointing being easier than describing, another potential advantage of such a moral realist approach might be that one is more confident in one’s meta-ethical view (“the pointer”) than in one’s object-level moral system (“one’s own description”). For example, someone could be confident that moral truth is determined by God’s will but be unsure that God’s will is expressed via the Bible, the Quran or something else, or how these religious texts are to be understood. Then that person would probably favor AI that cares about God’s will over AI that follows some particular interpretation of, say, the moral rules proposed in the Quran and Sharia.

A somewhat related issue which has received more attention in the moral realism literature is the convergence of human moral views. People have given moral realism as an explanation for why there is near-universal agreement on some ethical views (such as “when religion and tradition do not require otherwise, one shouldn’t torture babies”). Similarly, moral realism has been associated with moral progress in human societies, see, e.g., Huemer (2016). At the same time, people have used the existence of persisting and unresolvable moral disagreements (see, e.g., Bennigson 1996 and Sayre-McCord 2017, sect. 1) and the existence of gravely immoral behavior in some intelligent people (see, e.g., Nichols 2002) as arguments against moral realism. Of course, all of these arguments take moral realism to include a convergence thesis where being a human (and perhaps not being affected by some mental disorders) or a being a society of humans is sufficient to grasp and abide by moral truth.

Of course, there are also versions of moral realism that have even weaker (or just very different) implications for AI alignment and do not make any relevant convergence claims (cf. McGrath 2010). For instance, there may be moral realists who believe that there is a moral truth but that machines are in principle incapable of finding out what it is. Some may also call very different views “moral realism”, e.g. claims that given some moral imperative, it can be decided whether an action does or does not comply with that imperative. (We might call this “hypothetical imperative realism”.) Or “linguistic” versions of moral realism which merely make claims about the meaning of moral statements as intended by whoever utters these moral statements. (Cf. Lukas Gloor’s post on how different versions of moral realism differ drastically in terms of how consequential they are.) Or a kind of “subjectivist realism”, which drops mind-independence (cf. Olson 2014, ch. 2).

Why moral-realism-inspired research on AI alignment might be useful

I can think of many reasons why moral realism-based approaches to AI safety have not been pursued much: AI researchers often do not have a sufficiently high awareness of or interest in philosophical ideas; the AI safety researchers who do – such as researchers at MIRI – tend to reject moral realism, at least the versions with implications for AI alignment; although “moral realism” is popular among philosophers, versions of moral realism with strong implications for AI (à la Peter Singer or David Pearce) might be unpopular even among philosophers (cf. again Lukas’ post on how different versions of moral realism differ drastically in terms of how consequential they are); and so on…

But why am I now proposing to conduct such research, given that I am not a moral realist myself? The main reason (besides some weaker reasons like pluralism and keeping this blog interesting) is that I believe AI alignment research from a moral realist perspective might actually increase agreement between moral realists and anti-realists about how (and to which extent) AI alignment research should be done. In the following, I will briefly argue this case for the strong (à la Peter Singer and David Pearce) and the weak convergence versions of moral realism outlined above.

Strong versions

Like most problems in philosophy, the question of whether moral realism is true lacks an accepted truth condition or an accepted way of verifying an answer or an argument for either realism or anti-realism. This is what makes these problems so puzzling and intractable. This is in contrast to problems in mathematics where it is pretty clear what counts as a proof of a hypothesis. (This is, of course, not to say that mathematics involves no creativity or that there are no general purpose “tools” for philosophy.) However, the claim made by strong convergence moral realism is more like a mathematical claim. Although it is yet to be made precise, we can easily imagine a mathematical (or computer-scientific) hypothesis stating something like this: “For any goal X of some kind [namely the objectively incorrect and non-trivial-to-achieve kind] there is no efficient algorithm that when implemented in a robot achieves X in some class of environments. So, for instance, it is in principle impossible to build a robot that turns Earth into a pile of paperclips.” It may still be hard to formalize such a claim and mathematical claims can still be hard to prove or disprove. But determining the truth of a mathematical statement is not a philosophical problem, anymore. If someone lays out a mathematical proof or disproof of such a claim, any reasonable person’s opinion would be swayed. Hence, I believe that work on proving or disproving this strong version of moral realism will lead to (more) agreement on whether the “strong-moral-realism-based theory of AI alignment” is true.

It is worth noting that finding out whether strong convergence is true may not resolve metaphysical issues. Of course, all strong versions of moral realism would turn out false if the strong convergence hypothesis were falsified. But other versions of moral realism would survive. Conversely, if the strong convergence hypothesis turned out to be true, then anti-realists may remain anti-realists (cf. footnote 2). But if our goal is to make AI moral, the convergence question is much more important than the metaphysical question. (That said, for some people the metaphysical question has a bearing on whether they have preferences over AI systems’ motivation system – “if no moral view is more true than any other, why should I care about what AI systems do?”)

Weak versions

Weak convergence versions of moral realism do not make such in-principle-testable predictions. Their only claim is the metaphysical view that the goals identified by some method X (such as derivation from a set moral axioms, finding out what God wants, discourse, etc.) have some relation to moral truths. Thinking about weak convergence moral realism from the more technical AI alignment perspective is therefore unlikely to resolve disagreements about whether some versions of weak convergence moral realism are true. However, I believe that by not making testable predictions, weak convergence versions of moral realism are also unlikely to lead to disagreement about how to achieve AI alignment.

Imagine moral realists were to propose that AI systems should reason about morality according to some method X on the basis that the result of applying X is the moral truth. Then moral anti-realists could agree with the proposal on the basis that they (mostly) agree with the results of applying method X. Indeed, for any moral theory with realist ambitions, ridding that theory of these ambitions yields a new theory which an anti-realist could defend. As an example, consider Habermas’ discourse ethics and Yudkowsky’s Coherent Extrapolated Volition. The two approaches to justifying moral views seem quite similar – roughly: do what everyone would agree with if they were exposed to more arguments. But Habermas’ theory explicitly claims to be realist while Yudkowsky is a moral anti-realist, as far as I can tell.

In principle, it could be that moral realists defend some moral view on the grounds that it is true even if it seems implausible to others. But here’s a general argument for why this is unlikely to happen. You cannot directly perceive ought statements (David Pearce and others would probably disagree) and it is easy to show that you cannot derive a statement containing an ought without using other statements containing an ought or inference rules that can be used to introduce statements containing an ought. Thus, if moral realism (as I understand it for the purpose of this paper) is true, there must be some moral axioms or inference rules that are true without needing further justification, similar to how some people view the axioms of Peano arithmetic or Euclidean geometry. An example of such a moral rule could be (a formal version of) “pain is bad”. But if these rules are “true without needing further justification”, then they are probably appealing to anti-realists as well. Of course, anti-realists wouldn’t see them as deserving the label of “truth” (or “falsehood”), but assuming that realists and anti-realists have similar moral intuitions, anything that a realist would call “true without needing further justification” should also be appealing to a moral anti-realist.

As I have argued elsewhere, it’s unlikely we will ever come up with (formal) axioms (or methods, etc.) for morality that would be widely accepted by the people of today (or even among today’s Westerners with secular ethics). But I still think it’s worth a try. If it doesn’t work out, weak convergence moral realists might come around to other approaches to AI alignment, e.g. ones based on extrapolating from human intuition.

Other realist positions

Besides realism about morality, there are many other less commonly discussed realist positions, for instance, realism about which prior probability distribution to use, whether to choose according to some expected value maximization principle (and if so which one), etc. The above considerations apply to these other realist positions as well.

Acknowledgment

I wrote this post while working for the Foundational Research Institute, which is now the Center on Long-Term Risk.

---

^{1. There are some “universal instrumental goal” approaches to justifying morality. Some are based on cooperation and work roughly like this: “Whatever your intrinsic goals are, it is often better to be nice to others so that they reciprocate. That’s what morality is.” I think such theories fail for two reasons: First, there seem to many widely accepted moral imperatives that cannot be fully justified by cooperation. For example, we usually consider it wrong for dictators to secretly torture and kill people, even if doing so has no negative consequences for them. Second, being nice to others because one hopes that they reciprocate is not, I think, what morality is about. To the contrary, I think morality is about caring things (such as other people’s welfare) intrinsically. I discuss this issue in detail with a focus on so-called “superrational cooperation” in chapter 6.7 of “Multiverse-wide Cooperation via Correlated Decision Making”. Another “universal instrumental goal” approach is the following: If there is at least one god, then not making these gods angry at you may be another universal instrumental goal, so whatever an agent’s intrinsic goal is, it will also act according to what the gods want. The same “this is not what morality is about” argument seems to apply. ↩}

^{2. Yudkowsky has written about why he now rejects this form of moral realism in the first couple of blog posts in the “Value Theory” series. ↩}

I’ve written about the question of which decision theories describe the behavior of approaches to AI like the “Law of Effect”. In this post, I would like to discuss GOLEM, an architecture for a self-modifying artificial intelligence agent described by Ben Goertzel (2010; 2012). Goertzel calls it a “meta-architecture” because all of the intelligent work of the system is done by sub-programs that the architecture assumes as given, such as a program synthesis module (cf. Kaiser 2007).

Roughly, the top-level self-modification is done as follows. For any proposal for a (partial) self-modification, i.e. a new program to replace (part of) the current one, the “Predictor” module predicts how well that program would achieve the goal of the system. Another part of the system — the “Searcher” — then tries to find programs that the Predictor deems superior to the current program. So, at the top level, GOLEM chooses programs according to some form of expected value calculated by the Predictor. The first interesting decision-theoretical statement about GOLEM is therefore that it chooses policies — or, more precisely, programs — rather than individual actions. Thus, it would probably give the money in at least some versions of counterfactual mugging. This is not too surprising, because it is unclear on what basis one should choose individual actions when the effectiveness of an action depends on the agent’s decisions in other situations.

The next natural question to ask is, of course, what expected value (causal, evidential or other) the Predictor computes. Like the other aspects of GOLEM, the Predictor is subject to modification. Hence, we need to ask according to what criteria it is updated. The criterion is provided by the Tester, a “hard-wired program that estimates the quality of a candidate Predictor” based on “how well a Predictor would have performed in the past” (Goertzel 2010, p. 4). I take this to mean that the Predictor is judged based the extent to which it is able to predict the things that actually happened in the past. For instance, imagine that at some time in the past the GOLEM agent self-modified to a program that one-boxes in Newcomb’s problem. Later, the agent actually faced a Newcomb problem based on a prediction that was made before the agent self-modified into a one-boxer and won a million dollars. Then the Predictor should be able to predict that self-modifying to one-boxing in this case “yielded” getting a million dollar even though it did not do so causally. More generally, to maximize the score from the Tester, the Predictor has to compute regular (evidential) conditional probabilities and expected utilities. Hence, it seems that the EV computed by the Predictor is a regular EDT-ish one. This is not too surprising, either, because as we have seen before, it is much more common for learning algorithms to implement EDT, especially if they implement something which looks like the Law of Effect.

In conclusion, GOLEM learns to choose policy programs based on their EDT-expected value.

Acknowledgements

This post is based on a discussion with Linda Linsefors, Joar Skalse, and James Bell. I wrote this post while working for the Foundational Research Institute, which is now the Center on Long-Term Risk.

In an efficient market, one can expect that most goods are sold at a price-quality ratio that is hard to improve upon. If there was some easy way to produce a product cheaper or to produce a higher-quality version of it for a similar price, someone else would probably have seized that opportunity already – after all, there are many people who are interested in making money. Competing with and outperforming existing companies thus requires luck, genius or expertise. Also, if you trust other buyers to be reasonable, you can more or less blindly buy any “best-selling” product.

Several people, including effective altruists, have remarked that this is not true in the case of charities. Since most donors don’t systematically choose the most cost-effective charities, most donations go to charities that are much less cost-effective than the best ones. Thus, if you sit on a pile of resources – your career, say – outperforming the average charity at doing good is fairly easy.

The fact that charities don’t compete for cost-effectiveness doesn’t mean there’s no competition at all. Just like businesses in the private sector compete for customers, charities compete for donors. It just happens to be the case that being good at convincing people to donate doesn’t correlate strongly with cost-effectiveness.

Note that in the private sector, too, there can be a misalignment between persuading customers and producing the kind of product you are interested in, or even the kind of product that customers in general will enjoy or benefit from using. Any example will be at least somewhat controversial, as it will suggest that buyers make suboptimal choices. Nevertheless, I think addictive drugs like cigarettes are an example that many people can agree with. Cigarettes seem to provide almost no benefits to consumers, at least relative to taking nicotine directly. Nevertheless, people buy them, perhaps because smoking is associated with being cool or because they are addictive.

One difference between competition in the for-profit and nonprofit sectors is that the latter lacks monetary incentives. It’s nearly impossible to become rich by founding or working at a charity. Thus, people primarily interested in money won’t start a charity, even if they have developed a method of persuading people of some idea that is much more effective than existing methods. However, making a charity succeed is still rewarded with status and (the belief in) having had an impact. So in terms of persuading people to donate, the charity “market” is probably somewhat efficient in areas that confer status and that potential founders and employees intrinsically care about.

If you care about investing your resource pile most efficiently, this efficiency at persuading donors offers little consolation. On the contrary, it even predicts that if you use your resources to found or support an especially cost-effective charity, fundraising will be difficult. Perhaps you previously thought that, since your charity is “better”, it will also receive more donations than existing ineffective charities. But now it seems that if cost-effectiveness really helped with fundraising, more charities would have already become more cost-effective.

There are, however, cause areas in which the argument about effectiveness at persuasion carries a different tone. In these cause areas, being good at fundraising strongly correlates with being good at what the charity is supposed to do. An obvious example is that of charities whose goal it is to fundraise for other charities, such as Raising for Effective Giving. (Disclosure: I work for REG’s sister organization FRI and am a board member of REG’s parent organization EAF.) If an organization is good at fundraising for itself, it’s probably also good at fundraising for others. So if there are already lots of organizations whose goal it is to fundraise for other organizations, one might expect that these organizations already do this job so well that they are hard to outperform in terms of money moved per resources spent. (Again, some of these may be better because they fundraise for charities that generate more value according to your moral view.)

Advocacy is another cause area in which successfully persuading donors correlates with doing a very good job overall. If an organization can persuade people to donate and volunteer to promote veganism, it seems plausible that they are also good at promoting veganism. Perhaps most of the organization’s budget even comes from people they persuaded to become vegan, in which case their ability to find donors and volunteers is a fairly direct measure of their ability to persuade people to adopt a vegan diet. (Note that I am, of course, not saying that competition ensures that organizations persuade people of the most useful ideas.) As with fundraising organizations, this suggests that it’s hard to outperform advocacy groups in areas where lots of people have incentives to advocate, because if there were some simple method of persuading people, it’s very likely that some large organization based on that method would have already been established.

That said, there are many caveats to this argument for a strong correlation between fundraising and advocacy effectiveness. First off, for many organizations, fundraising appears to be primarily about finding, retaining and escalating a small number of wealthy donors. For some organizations, a similar statement might be true about finding volunteers and employees. In contrast, the goal of most advocacy organizations is to persuade a large number of people.¹ So there may be organizations whose members are very persuasive in person and thus capable of bringing in many large donors, but who don’t have any idea about how to run a large-scale campaign oriented toward “the masses”. When trying to identify cost-effective advocacy charities, this problem can, perhaps, be addressed by giving some weight to the number of donations that a charity brings in, as opposed to donation sizes alone.² However, the more important point is that if growing big is about big donors, then a given charity’s incentives and selection pressures for survival and growth are misaligned with persuading many people. Thus, it becomes more plausible again that the average big or fast-growing advocacy-based charity is a suboptimal use of your resource pile.

Second, I stipulated that a good way of getting new donors and volunteers is to simply persuade as many people of your general message as possible, and then hope that some of these will also volunteer at or donate to your organization. But even if all donors contribute similar amounts, some target audiences are more likely to donate than others.³ In particular, people seem more likely to contribute larger amounts if they have been involved for longer, have already donated or volunteered, and/or hold a stronger or more radical version of your organization’s views. But persuading these community members to donate works in very different ways than persuading new people. For example, being visible to the community becomes more important. Also, if donating is about identity and self-expression, it becomes more important to advocate in ways that express the community’s shared identity rather than in ways that are persuasive but compromising. The target audiences for fundraising and advocacy may also vary a lot along other dimensions: for example, to win an election, a political party has to persuade undecided voters, who tend to be uninformed and not particularly interested in politics (see p. 312 of Achen and Bartel’s Democracy for Realists); but to collect donations, one has to mobilize long-term party members who probably read lots of news, etc.

Third, the fastest-growing advocacy organizations may have large negative externalities.⁴ Absent regulations and special taxes, the production of the cheapest products will often damage some public good, e.g., through carbon emissions or the corruption of public institutions. Similarly, advocacy charities may damage some public good. The fastest way to find new members may involve being overly controversial, dumbing down the message or being associated with existing powerful interests, which may damage the reputation of a movement. For example, the neoliberals often suffer from being associated with special/business interests and crony capitalism (see sections “Creating a natural constituency” and “Cooption” in Kerry Vaughan’s What the EA community can learn from the rise of the neoliberals), perhaps because associating with business interests often carries short-term benefits for an individual actor. Again, this suggests that the fastest-growing advocacy charity may be much worse overall than the optimal one.

Acknowledgements

I thank Jonas Vollmer, Persis Eskander and Johannes Treutlein for comments. This work was funded by the Foundational Research Institute (now the Center on Long-Term Risk).

---

^{1. Lobbying organizations, which try to persuade individual legislators, provide a useful contrast. Especially in countries with common law, organizations may also attempt to win individual legal cases. ↩}

^{2. One thing to keep in mind is that investing effort into persuading big donors is probably a good strategy for many organizations. Thus, a small-donor charity that grows less quickly than a big-donor charity may be be more or less cost-effective than the big-donor charity. ↩}

^{3. One of the reasons why one might think that drawing in new people is most effective is that people who are already in the community and willing to donate to an advocacy org probably just fund the charity that persuaded them in the first place. Of course, many people may simply not follow the sentiment of donating to the charity that persuaded them. However, many community members may have been persuaded in ways that don’t present such a default option. For example, many people were persuaded to go vegan by reading Animal Liberation. Since the book’s author, Peter Singer, has no room for more funding, these people have to find other animal advocacy organizations to donate to. ↩}

^{4. Thanks to Persis Eskander for bringing up this point in response to an early version of this post. ↩}