Tag: philosophy

Deontological values and moral trade

On By CasparIn GeneralLeave a comment

[For this post, I’ll assume some familiarity with the concept of moral trade and the distinction between consequentialist and deontological values.]

In earlier work, I claimed that (in the specific context of ECL) if you are trying to benefit someone’s moral view as part of some cooperative arrangement, only the consequentialist aspects of their moral values are relevant to you. That is, if you want to act cooperatively toward Alice’s moral values, then you need to consider only the consequentialist components of Alice’s value system. For instance, you need to ask yourself: Would Alice wish for there to be fewer lies? You don’t need to ask yourself whether Alice considers it a moral imperative not to lie herself (except insofar as it relates to the former question).

While I still believe that there’s some truth to this claim, I now believe that the claim is straightforwardly incorrect. In short, it seems plausible that, for example, Alice’s moral imperative not to lie extends to actions by others that Alice brings about via trade. (Furthermore, it seems plausible that this is the case even if Alice doesn’t consider it a moral imperative to, for example, donate money to fund fact checkers in a distant country. That is, it seems plausible that this is the case even if Alice doesn’t care about lies in a fully consequentialist way.) I’ll give further intuition pumps for the relevance of deontological constraints below. I’ll take a more abstract perspective in the section right after giving the examples.

Related work. Toby Ord’s article on moral trade also has a section (titled “Consequentialism, deontology, and virtue ethics”) that discusses how the concept of moral trade interacts with the distinction between deontological and consequentialist ethics. (He also discusses virtue ethics, which I ignore to keep it simple.) However, he doesn’t go into much detail and seems to make different points than this post. For instance, he argues that even deontologists are often somewhat consequentialist (which my earlier writing also emphasizes). He also makes at least one point that is somewhat contrary to the claims in this article. I will discuss this briefly below (under example P1).

Examples. I’ll now give some examples of situations in which it seems intuitively compelling that someone’s deontological duties propagate through a trade relationship (P1–3). I’ll vary both the duties and the mode of trade. I’ll then also provide two negative examples (N1,2), i.e., examples where deontological norms arguably don’t propagate through trade. The examples are somewhat redundant. There’s no need to read them all!

The examples will generally consider the perspective of the deontologist’s trading partner who is uncertain about the deontologist’s views (rather than the deontologist herself). I’m taking the deontologist’s trading partner’s perspective because I’m interested in how to deal with other’s deontological views in trade. I’ll assume that one cannot simply ask the deontologist, because this would shift all difficulty to the deontologist and away from the deontologist’s trading partner.

P1: Say you have a friend Alice. Alice follows the following policy: Whenever someone does something that’s good for her moral views, Alice pays them back in some way, e.g., monetarily. For instance, Alice is concerned about animal welfare. Whenever she learns that hardcore carnivore Bob eats a vegetarian meal, she sends him a dollar. (There are lots of practical game-theoretic difficulties with this – can vegetarians all earn free money from Alice by claiming that they are vegetarian only to be nice to her? – but let’s ignore these.)

Now, let’s say that Carol is considering what actions to take in light of Alice’s policy. Carol comes up with the following idea. Perhaps she should put up posters falsely claiming that local cows graze from heavily polluted pastures. Let’s say that she is sure that this has a positive impact on animal welfare (i.e., that she has reason to not be concerned about backlash, etc.). But now let’s say that Alice once said that she wouldn’t want to lie even when doing so has positive consequences. Meanwhile, she also isn’t a “truth maximizer”; she argued that the cause of correcting others’ inconsequential lies out in the world isn’t worthwhile. Should Carol expect to receive payment?

To me it seems plausible that Carol shouldn’t or at least she should be doubtful about whether she will be paid. (For what it’s worth, it seems that Claude 3 agrees.) Given Alice’s stance on lying, Alice surely wouldn’t want to put up the posters herself. It seems plausible to me that Alice then also wouldn’t want to pay Carol to put up such posters (even if the payment is made in retrospect).

Here’s one way to think about it. Imagine that Alice has written a book, “What I’m happy to pay people to do”. The book contains a list with items such as “raise awareness of animal welfare conditions under factory farming”, “eating less meat”, etc. Would we expect the book to contain an item, “put up misleading posters that cause people to avoid meat”? Again, I would imagine that the answer is no. Putting such an item in the book is directly causing others to lie on Alice’s behalf. More specifically, it’s causing others to lie in pursuit of Alice’s goals. This doesn’t seem so different from Alice lying herself. Perhaps when Carol tries to benefit Alice’s moral views, she should act on what Carol would predict Alice to have put in the book (even if Alice never actually writes such a book).

Interestingly, Toby Ord’s article on the subject contains a hint in the opposite direction: “[I]t is possible that side constraints or agent-relative value could encourage moral trade. For example, someone might think that it is impermissible for them to lie in order to avoid some suffering but that it wouldn’t be impermissible to convince someone else to make this lie in order to avoid the suffering.”

P2: Let’s say Alice and Bob are decision-theoretically similar enough that they would cooperate with each other in a one-shot Prisoner’s Dilemma (even under somewhat asymmetric payoffs). Let’s say that Alice could benefit Bob but have to lie in order to do so. Conversely, let’s say that Bob could benefit Alice, but would have to steal in order to do so. Alice believes that one (deontologically) ought not to steal, but thinks that lying is in principle acceptable in and of itself. Conversely, Bob believes that one (deontologically) ought not to lie, but thinks that stealing is acceptable.

Ignoring the deontological constraints, Alice would benefit Bob to make it more likely that Bob benefits Alice. But now what if they take the deontological constraints into account? Would Alice still benefit Bob to make it more likely that Bob benefits Alice? Again, I think it’s plausible that she wouldn’t. By lying in order to benefit Bob, Alice makes it likely that Bob would steal. In some sense Alice makes Bob steal in service of Alice’s goals. It seems intuitive that Alice’s deontological constraints against stealing should still apply. It would be a strange “hack” if deontological constraints didn’t apply in this context; one could circumvent deontological constraints simply by trading one’s violations with others. (It’s a bit similar to the plot of “Strangers on a Train”, in which a similar swap is proposed to avoid criminal liability.) (Again, compare Toby Ord’s comment, as discussed in P1.)

P3: The previous examples illustrate the propagation of negative duties (duties of the “you shalt not…” variety). I’ll here give an example of how positive duties might also transfer.

The country of Charlesia is attacked by its neighbor. Charlesia is a flourishing liberal democracy, so Alice considers it an ethical imperative to protect Charlesia. In order to do so, Alice hires a group of mercenaries to fight on Charlesia’s side. Since time is of the essence, the mercenaries aren’t given a proper contract that specifies their exact objectives, rules of engagement and such.

Two weeks later, the mercenaries find themselves marching through the warzone in Charlesia. They encounter a weak old man who is asking to be evacuated. Doing so would distract significant resources from the mercenaries’ military mission. The mercenaries and their equipment are very expensive. So, clearly, if Alice wanted to spend money in order to help weak old people, she wouldn’t have spent the money on hiring the mercenaries. Thus, if Alice is a pure consequentialist, she wouldn’t want the mercenaries to help the weak old man. So if the mercenaries wanted to act in mercenary fashion (i.e., if they wanted to do whatever Alice wants them to do), should they leave the weak old man to die?

Not necessarily! I think it is entirely plausible that Alice would want the mercenaries to act according to common-sense, not fully consequentialist ethics. That is, it seems plausible that Alice would want the mercenaries to observe the moral obligation to help the old man, as Alice would presumably do if she was in the mercenaries’ place. As in the other cases, it seems intuitive that the mercenaries are in some sense acting on Alice’s behalf. So it seems that Alice would want the mercenaries to act somewhat similar to how she would act.

N1: Let’s say Alice from P1 wants to pay Bob to eat more vegetarian meals. Unfortunately, Bob’s social group believes that “real men eat meat”. Therefore, Bob says that if he becomes a vegetarian (or reducetarian), he’ll have to lie to his friends about his diet. (If this lie is not consequential enough to be ethically relevant, you can add further context. For instance, you might imagine that Bob’s lies will cause his friends to develop less accurate views about the healthiness of different diets.) Let’s say that Alice observes deontological norms of honesty. Does this mean that Bob should reason that becoming a vegetarian wouldn’t be a successful cooperative move towards Alice?

It’s unclear, but in this case it seems quite plausible that Bob can benefit Alice by becoming a closet vegetarian. It seems that the lie is more of a side effect – Bob doesn’t lie for Alice, he lies for himself (or in some sense for his social group). So it seems that (according to conventional ethical views) Alice doesn’t bear much of a responsibility for the lie. So in this case, it seems plausible to me that Alice’s deontological constraint against lying isn’t (strongly) propagated through the trade relationship between Alice and Bob.

N2: The inhabitants of a sparsely populated swampland want to found a state. (Founding a state can be viewed as a many-player trade.) Founding a state would require agreement from Alice – she’s the only accountant in the swampland and the envisioned state would require her to keep an eye on finances. (That said, lots of other inhabitants of the swampland are also individually necessary for the state, and only a small fraction of the state’s tax revenue would come from Alice.) Unfortunately, Alice can’t be present for the founding meeting of the state. Thus, in designing the state, the remaining inhabitants of the swampland have to make guesses about Alice’s interests.

After some discussion, over 90% of the resources of the state have been allocated, all to issues that Alice is known to be on board with: infrastructure, a medical system, social security, educatio, etc. The state policies also include a few measures that Alice agrees with and that only a minority of other inhabitants of the swampland care about, including a museum, a concert hall and a public library.

One of the last issues to be sorted out is law enforcement and in particular the legal system. In the past, vigilante justice has ruled the land. Now a formal set of laws are to be enforced by a police. Everyone (including Alice) agrees that this will reduce crime, will make punishments more humane, and will make it less likely that innocents are unfairly punished. Most inhabitants of the swampland agree that there should be a death penalty for the most severe crimes. However, Alice holds that all killing is unethical. In the past, when she carried out vigilante justice herself, she strictly avoided killings, even when doing so meant letting someone get away with murder.

If the members of the founding meeting propose a state policy that involves the death penalty, must they expect that Alice will refuse to perform accounting for the state?

Again, I think that under many circumstances it’s reasonable for the others to expect that Alice will not block the state. It seems that if people like Alice, then founding states (or other large social arrangements) with Alice would simply be too difficult. Perhaps hermits would reject the state, but agents who function in societies can’t be so fussy.

Conceptualization via a model of deontology. I’ll here propose a simple model of deontological ethics to get a better grasp on the role of deontological norms in trade. I won’t immediately use this model for anything, so feel free to skip this section! 

Consider the following model of deontological values. Let’s say your actions can result in consequences via different types of “impact paths”. Consequentialists don’t care about the impact path – they just care about the consequence itself. Deontologists generally care about the consequences, but also care about the impact path and depending on the type of the impact path, they might care more or less about the consequence. For instance, if an outcome is brought about via inaction, then a deontologist might care less about it. Similarly, impact paths that consist of long causal chains that are hard to predict might matter less to deontologists. Meanwhile, deontologists care a lot about the “empty” impact path, i.e., about cases where the action itself is the consequence. For example, deontologists typically want to minimize lies that they tell themselves much more than they care about not acting in a way that causes others to lie. Which impact paths matter how much is up to the individual deontological view and different deontologists might disagree. (I’m not sure how good of a model of deontological values this is. It’s certainly a very consequentialist perspective. (Cf. Sinnott-Armstrong (2009) [paywalled].) It’s also quite vague, of course.)

In this model of deontological views, trade is simply one type of impact path by which we can bring about outcomes. And then the different types of trades like tit for tat, signing contracts, ECL, and other forms of acausal trade are specific subtypes of impact paths. Different deontologists have different views about the paths. So in particular, there can be deontological views according to which bringing something about by trade is like bringing it about by inaction. But I think it’s more natural for deontologists to care quite a bit about at least some simple impact paths via trade, as argued above.

If impact paths via trade matter according to Alice’s deontological views, then Alice’s trading partners need to take Alice’s deontologist views into account (unless they simply receive explicit instructions from Alice about what Alice wants them to do).

Implications for ECL. (For this section, I assume familiarity with ECL (formerly MSR). Please skip this section if you’re unfamiliar with this.)

Do deontological views matter for how we should do ECL? In principle, it clearly does matter. For instance, consequentialists who otherwise don’t abide by deontological norms (even for instrumental reasons) should abide by deontological norms in their implementation of ECL.

That said, I think in practice there are lots of reasons why following deontological views via ECL might not matter so much, especially because even absent the above consideration ECLers might already take deontological norms into account: (Perhaps this is also a reason why neither I nor anyone else to my knowledge pointed out that my earlier writing on ECL was wrong to claim that deontological views of trading partners can be ignored.)

  • Consequentialists typically argue that consequentialism itself already implies that we should follow deontological norms in practice. For instance, consequentialists might say that in practice lies (especially consequential lies) are eventually found out with high enough probability that the risk of being found out typically outweighs the potential benefits of getting away with the lie. It is also sometimes argued that to apply consequentialism in practice, one has to follow simple rules (since assessing all the different possible consequences of an action is intractable) and that the rules proposed by deontologists are rules that consequentialists should follow in lieu of trying to calculate the consequences of all of their actions. If you agree with these sorts of views, then in particular you’d hold that consequentialist ECLers should already abide by deontological norms anyway, even if they only consider the consequentialist aspects of the ethical views they’re trying to benefit. Of course, there might be deontological norms that aren’t justifiable on consequentialist grounds. The propagation of these norms through trade would be relatively important.
  • My sense is that pure consequentialists are rare. My sense is that most EAs are, if anything, more likely than others to abide by standard ethical norms (such as not lying). (Sam Bankman-Fried is commonly brought up as an example of an overly consequentialist EA.) In any case, if you yourself already subscribe to lots of deontological norms, then importing deontological norms via ECL makes less of a difference. Again, you might of course specifically not subscribe to some specific popular deontological norms. If so, then ECL makes these norms more relevant to you.
  • Even without deontological views in the picture, ECL often pushes toward more deontological-norm-abiding behavior. For instance, ECL suggests a less adversarial posture towards people with other value systems.

Some research questions. I should start by saying that I assume there’s already some literature on the relation between deontological views and trade in the ethics literature. In traditional trade contexts (e.g., paying the baker so that she bakes us bread), this seems like a pertinent issue. For instance, I assume deontologists have considered ethical consumerism. I haven’t tried to review this literature. I would imagine that it mostly addresses the normative ethical dimension, rather than, say, the game-theoretic aspects of it. There’s also a question to what extent the moral trade context is fundamentally different from the traditional economic trade context.

An immediate obstacle to researching the role of deontological views in moral trade is that we don’t have toy models of deontological ethics. For trading on consequentialist grounds, we have some good economic toy models (agents investing in different types of interventions with comparative advantages, see, for example, the ECL paper). Are there corresponding toy models to get a better grasp on trading when deontological constraints are involved? Perhaps the simplest formal model would be that deontologists only agree to a trade if agreeing to the deal doesn’t increase the total number of times that their norms are violated, but that’s arguably too strong.

In real-world trades, one important question is to what extent and how a trading partner’s deontological constraints propagate beyond actions taken specifically to benefit that trading partner. The positive examples above (P1–3) consider cases where agent 1 faces a single choice solely to benefit agent 2 and then ask whether agent 2’s deontological views restrict that choice. Meanwhile, the negative examples above (N1,2) involve more complicated interactions between multiple decision problems and between the interests of multiple actors. So, in general what happens if agent 1 also faces other decisions in which she pursues exclusively her own goals – do the deontological views apply to those actions as well in some way (as in N1)? What should happen in multi-party trade (as in N2)? Let’s say agent 1 makes choices in order to benefit both agent 2 and agent 3, and that only agent 2 has deontological ethics. Then do agent 2’s deontological views fully apply to agent 1’s actions? Or is it perhaps sufficient for agent 2’s participation in the trade to not increase the number of violations by agent 1 of agent 2’s duties? (If so, then what is the right counterfactual for what trade(s) happen?) Of course, these are in part (descriptive) ethical questions. (“How do deontologists want their deontological views to propagate through trade?”) I wonder whether a technical analysis can nonetheless provide some insights. 

Acknowledgments. I thank Emery Cooper and Joseph Carlsmith for helpful comments and discussions.

On pragmatist critiques of self-locating beliefs

On By CasparIn General2 Comments

I assume some familiarity with basic concepts in the area of anthropics (a.k.a. self-locating beliefs or imperfect recall) and decision theory.

A pragmatist maxim of action relevance

Consider the following form of pragmatism, which I think is close to Peircian pragmatism:

The pragmatist maxim of action relevance: You should only ask yourself questions that are (in principle) decision-relevant. That is, you should only ask yourself a question Q if there is some (hypothetical) decision situation where you take different actions depending on how you answer Q.

I am aware that this is quite vague and has potential loopholes. (For any question Q, does it count as a decision situation if someone just asks you what you think about Q? We have to rule this out somehow if we don’t want the maxim to be vacuous.) For the purpose of this post, a fairly vague notion of the maxim will suffice.

The action relevance maxim rules out plenty of definitional questions. E.g., it recommends against most debates on the question, “If a tree falls in a forest and no one is around to hear it, does it really make a sound?”. (It does allow asking the question, “What meaning should we give to the word ‘sound’?”) Importantly, it allows asking normative ethical questions such as, “should we kill murderers or put them in jail?”. By allowing hypothetical scenarios, you can still ask questions about time travel and so forth. Perhaps it has some controversial implications for when and why one should discuss consciousness. (If you already fully understand how, say, a biological system works, then how will it matter for your actions whether that system is conscious? I can only think of ethical implications – if a system is conscious, I don’t want to harm it. Therefore, consciousness becomes primarily a question of what systems we should care about. This seems similar to some eliminativist views, e.g., that of Brian Tomasik.) That said, for every definitional question (what is “sound”, “consciousness”, etc.), there are pragmatically acceptable questions, such as, “What is a rigorous/detailed definition of ‘consciousness’ that agrees with (i.e., correctly predicts) our intuitions about whether a system is conscious?”.) I think the implications for anthropics / self-locating beliefs are also controversial. Anyway, I find the above pragmatist maxim compelling.

It is worth noting that there are also other pragmatist principles under which none of the below applies. For example, Eliezer Yudkowsky has a post titled “Making Beliefs Pay Rent in Anticipated Experiences”. Self-locating beliefs anticipate experiences. So even without action relevance they “pay rent” in this sense.

Successful pragmatist critiques of anthropics

I think some pragmatist critiques of anthropics are valid. Here’s the most important critique that I think is valid: (It uses some terms that I’ll define below.)

If you have non-indexical preferences and you think updateless decision theory – i.e., maximizing ex ante expected utility (expected utility from the perspective of the prior probability distribution) – is the only relevant normative decision criterion, then the philosophical question of what probabilities you should assign in scenarios of imperfect recall disappears.

First, some explanations and caveats: (I’ll revisit the first two later in this post.)

  • By non-indexical preferences I mean preferences that don’t depend on where in the scenario you are. For example, in Sleeping Beauty, an indexical preference might be: “I prefer to watch a movie tonight.” This is indexical because the meaning of “tonight” differs between the Monday and Tuesday instantiations of Beauty. The Tuesday instantiation will want Beauty to watch a movie on Tuesday night, while the Monday instantiation will not care whether Beauty watches a movie on Tuesday night.
  • I’ll leave it to another post to explain what I consider to be a “philosophical” question. (Very roughly, I mean: questions for which there’s no agreed-upon methodology for evaluating proposed answers and arguments.) I’ll give some questions below about self-locating beliefs that I consider to be non-philosophical, such as how to compute optimal policies.
  • Of course, scenarios of imperfect recall may also involve other (philosophical) issues that aren’t addressed by UDT (the maxim of following the ex ante optimal policy) and that don’t arise in, say, Sleeping Beauty or the absent-minded driver. For example, we still have to choose a prior, deal with problems of game theory (such as equilibrium selection), deal with infinities (e.g., as per infinite ethics) and so on. I’m not claiming that UDT addresses any of these problems.

As an example, optimizing ex ante utility is sufficient to decide whether to accept or reject any given bet in Sleeping Beauty. Also, one doesn’t need to answer the question, “what is the probability that it is Monday and the coin came up Heads?” (On other questions it is a bit unclear whether the ex ante perspective commits to an answer or not. In some sense, UDTers are halfers: in variants of Sleeping Beauty with bets, the UDTers expected utility calculations will have ½ in place of probabilities and the calculation overall looks very similar to someone who uses EDT and (double) halfing (a.k.a. minimum-reference-class SSA). On the other hand, in Sleeping Beauty without bets, UDTers don’t give any answer to any question about what the probabilities are. On the third hand, probabilities are closely tied to decision making anyway. So even (“normal”, non-UDT) halfers might say that when they talk about probabilities in Sleeping Beauty, all they’re talking about is what numbers they’re going to multiply utilities with when offered (hypothetical) bets. Anyway: Pragmatism! There is no point in debating whether UDTers are halfers or not.)

Some other, less central critiques succeed as well. In general, it’s common to imagine purely definitional disputes arising about any philosophical topic. So, if you show me a (hypothetical) paper titled, “Are SIA probabilities really probabilities?”, I will be a little skeptical of the paper’s value.

(There are also lots of other possible critiques of various pieces of work on anthropics that are outside the scope of this post. Arguably too many papers rely too much on intuition pumps. For instance, Bostrom’s PhD thesis/book on anthropics is sometimes criticized for this (anonymous et al., personal communication, n.d.). I also think that anthropic arguments applied to the real world (the Doomsday argument, the simulation argument, arguments from fine tuning, etc.) often don’t specify that they use specific theories of self-locating beliefs.)

The defense

I now want to defend some of the work on anthropics against pragmatist critiques. The above successful critique already highlights, to some extent, three caveats. Each of these gives rise to a reason why someone might think about how to reason de se (from “within” the scenario, “updatefully”) about games of imperfect recall:

  1. Indexical preferences. Ex ante optimization (UDT) alone doesn’t tell you what to do if you have indexical preferences, because it’s not clear how to aggregate preferences between the different “observer moments”. Armstrong (2011) shows a correspondence between methods of assigning self-locating beliefs (SIA, etc.) and methods of aggregating preferences across copies (average and total utilitarianism). That’s a great insight! But it doesn’t tell you what to do. (Perhaps it’s an argument for relativism/antirealism: you can choose whatever way of aggregating preferences across observer moments you like, and so you could also choose whatever method of assigning self-locating beliefs that you like. But even if you buy into this relativist/antirealist position, you still need to decide what to do.)
  2. Rejecting updatelessness (e.g., claiming it’s irrational to pay in counterfactual mugging). If the ex ante optimal/updateless choice is not the unambiguously correct one, then you have to ask yourself what other methods of decision making you find more compelling.
  3. Asking non-philosophical questions of what procedures work. One might want to know which kinds of reasoning “work” for various notions of “work” (satisfying the reflection principle; when used for decision making: avoiding synchronic or diachronic Dutch books, being compatible with the ex ante optimal/updatelessness policy). Why?
    • I’m sure some philosophers do this just out of curiosity. (“Non-minimum reference class SSA seems appealing. I wonder what happens if we use it to make decisions.” (It mostly doesn’t work.))
    • But there are also lots of very practical reasons that apply even if we fully buy into updatelessness. In practice, even those who buy fully into updatelessness talk about updating their probabilities on evidence in relatively normal ways, thus implicitly assigning the kind of self-locating beliefs that UDT avoids. (For example, they might say, “I read a study that caused me to increase my credence in vitamin D supplements being beneficial even in the summer. Therefore, I’ve ordered some vitamin D tablets.” Not: “From my prior’s perspective, the policy of ordering vitamin D tablets upon reading such and such studies is greater than the policy of not taking vitamin D tablets when exposed to such studies.”) Updating one’s beliefs normally seems more practical. But if in the end we care about ex ante utility (UDT), then does updating even make sense? What kind of probabilities are useful in pursuit of the ex ante optimal/updateless policy? And how should we use such probabilities?

      As work in this area shows: both SIA probabilities (thirding) and minimum-reference class SSA probabilities (double halfing) can be useful, while non-minimum-reference class SSA probabilities (single halfing) probably aren’t. Of the two that work, I think SIA actually more closely matches intuitive updating. (In sufficiently large universes, every experience occurs at least once. Minimum-reference class SSA therefore makes practically no updates between large universes.) But then SIA probabilities need to be used with CDT! We need to be careful not to use SIA probabilities with EDT.

      Relatedly, some people (incl. me) think about this because they wonder how to build artificial agents that choose correctly in such problems. Finding the ex ante optimal policy directly is generally computationally difficult. Finding CDT+SIA policies is likely theoretically easier than finding an ex ante optimal policy (CLS-complete as opposed to NP-hard), and also can be done using practicable modern ML techniques (gradient descent). Of course, in pursuit of the ex ante optimal policy we need not restrict ourselves to methods that correspond to methods involving self-locating beliefs. There are some reasons to believe that these methods are computationally natural, however. For example, CDT+SIA is roughly computationally equivalent to finding local minima of the ex ante expected utility function.

Acknowledgments

I thank Emery Cooper, Vince Conitzer and Vojta Kovarik for helpful comments.

An information-theoretic variant of the adversarial offer to address randomization

On By CasparIn GeneralLeave a comment

[I assume the reader is familiar with Newcomb’s problem and causal decision theory. Some familiarity with basic theoretical computer science ideas also helps.]

The Adversarial Offer

In a recently (Open-Access-)published paper, I (together with my PhD advisor, Vince Conitzer) proposed the following Newcomb-like scenario as an argument against causal decision theory:

> Adversarial Offer: Two boxes, B1 and B2, are on offer. A (risk-neutral) buyer may purchase one or none of the boxes but not both. Each of the two boxes costs $1. Yesterday, the seller put $3 in each box that she predicted the buyer would not acquire. Both the seller and the buyer believe the seller’s prediction to be accurate with probability 0.75.

If the buyer buys one of the boxes, then the seller makes a profit in expectation of $1 – 0.25 * $3 = $0.25. Nonetheless, causal decision theory recommends buying a box. This is because at least one of the two boxes must contain $3, so that the average box contains at least $1.50. It follows that the causal decision theorist must assign an expected causal utility of at least $1.50 to (at least) one of the boxes. Since $1.50 exceeds the cost of $1, causal decision theory recommends buying one of the boxes. This seems undesirable. So we should reject causal decision theory.

The randomization response

One of the obvious responses to the Adversarial Offer is that the agent might randomize. In the paper, we discuss this topic at length in Section IV.1 and in the subsection on ratificationism in Section IV.4. If you haven’t thought much about randomization in Newcomb-like problems before, it probably makes sense to first check out the paper and only then continue reading here, since the paper makes more straightforward points.

The information-theoretic variant

I now give a new variant of the Adversarial Offer, which deals with the randomization objection in a novel and very interesting way. Specifically, the unique feature of this variant is that CDT correctly assesses randomizing to be a bad idea. Unfortunately, it is quite a bit more complicated than the Adversarial Offer from the paper.

Imagine that the buyer is some computer program that has access to a true random number generator (TRNG). Imagine also that the buyer’s source code plus all its data (memories) has a size of, say, 1GB and that the seller knows that it has (at most) this size. If the buyer wants to buy a box, then she will have to pay $1 as usual, but instead of submitting a single choice between buying box 1 and buying box 2, she has to submit 1TB worth of choices. That is, she has to submit a sequence of 2^43 (=8796093022208) bits, each encoding a choice between the boxes.

If the buyer buys and thus submits some such string of bits w, the seller will do the following. First, the seller determines whether there exists any deterministic 1GB program that outputs w. (This is undecidable. We can fix this if the seller knows the buyer’s computational constraints. For example, if the seller knows that the buyer can do less than 1 exaFLOP worth of computation, then the seller could instead determine only whether there is a 1GB program that produces w with at most 1 exaFLOP worth of computation. This is decidable (albeit not very tractable).) If there is no such program, then the seller knows that the buyer randomized. The buyer then receives no box while the seller keeps the $1 payment. The buyer is told in advance that this is what the seller will do. Note that for this step, the seller doesn’t use her knowledge of the buyer’s source code other than its length (and its computational constraints).

If there is at least one 1GB program that outputs w deterministically, then the seller forgets w again. She then picks an index i of w at random. She predicts w[i] based on what she knows about the buyer and based on w[1…i-1], i.e., on all the bits of w preceding i. Call the prediction w’[i]. She fills the box based on her prediction w’[i] and the buyer receives (in return for the $1) the box specified by w[i].

Why the information-theoretic variant is interesting

The scenario is interesting because of the following three facts (which I will later argue to hold):

  1. The seller makes a profit off agents who try to buy boxes, regardless of whether they do so using randomization or not.
  2. CDT and related theories (such as ratificationism) assess randomizing to be worse than not buying any box.
  3. CDT will recommend buying a box (mostly deterministically).

I’m not aware of any other scenarios with these properties. Specifically, the novelty is item 2. (Our paper offers a scenario that has the other two properties.) The complications of this scenario – letting the agent submit a TB worth of choices to then determine whether they are random – are all introduced to achieve item 2 (while preserving the other items).

In the following, I want to argue in more detail for these three points and for the claim that the scenario can be set up at all (which I will do under claim 1).

1. For this part, we need to show two things:

A) Intuitively, if the agent submits a string of bits that uses substantially more than 1GB worth of randomness, then he is extremely likely to receive no box at all.

B) Intuitively, if the agent uses only about 1GB or less worth randomness, then the seller – using the buyer’s source code will likely be able to predict w[i] with high accuracy based on w[1…i-1].

I don’t want to argue too rigorously for either of these, but below I’ll give intuitions and some sketches of the information-theoretic arguments that one would need to give to make them more rigorous.

A) The very simple point here is that if you create, say, a 2GB bitstring w where each bit is determined by a fair coin flip, then it is very unlikely that there exists a program that deterministically outputs w. After all, there are many more ways to fill 2GB of bits than there are 1GB programs (about 2^(2^33) as many). From this one may be tempted to conclude that if the agent determines, say, 2GB of the TB of choices by flipping coin, he is likely to receive no box. But this argument is incomplete, because there are other ways to use coin flips. For example, the buyer might use the following policy: Flip 2GB worth of coins. If they all come up heads, always take box B. Otherwise follow some given deterministic procedure.

To make the argument rigorous, I think we need to state the claim information-theoretically. But even this is a bit tricky. For example, it is not a problem per se for w to have high entropy if most of the entropy comes from a small part of the distribution. (For example, if with probability 0.01 the seller randomizes all choices, and with the remaining probability always chooses Box 1, then the entropy of w is 0.01*1TB = 10GB (??), but the buyer is still likely to receive a box.) So I think we’d need to make a more complicated claim, of the sort: if there is no substantial part (say, >p) of the distribution over w that has less than, say, 2GB of entropy, then with high probability (>1-p), the agent will receive no box. 

B) Again, we can make a simple but incomplete argument: If of the 1TB of choices, only, say, 2GB are determined by random coin flips, then a randomly sampled bit is likely to be predictable from the agent’s source code. But again, the problem is that the random coin flips can be used in other ways. For example, the buyer might use a deterministic procedure to determine w (say, w=01010101…), but then randomly generate a number n (with any number n chosen with probability 2^-n, for instance), then randomly sample n indices j of w and flip w[j] for each of them. This may have relatively low entropy. But now the seller cannot perfectly predict w[i] given w[1…i-1] for any i.

Again, I think a rigorous argument requires information theory. In particular, we can use the fact that H(w) = H(w[0])+H(w[1]|w[0])+H(w[2]|w[0,1])+…, where H denotes entropy. If H(w) is less than, say, 2GB, then the average of H(w[i]|w[1…i-1]) must be at most 2GB/1TB = 1/500. From this, it follows immediately that with high probability, w[i] can be predicted with high accuracy given w[1…i-1].

2. This is essential, but straightforward: Generating w at random causes the seller to determine that w is determined at random. Therefore, CDT (accurately) assesses randomly generating w to have an expected utility near $-1.

3. Finally, I want to argue that CDT will recommend buying a box. For this, we only need to argue that CDT prefers some method of submitting w over not submitting a box. So consider the following procedure: First, assign beliefs over the seller’s prediction w’[0] of the first bit. Since there are only two possible boxes, for at least one of the boxes j, it is the case that P(w’[0]=j)<=½, where P refers to the probability assigned by the buyer. Let w[0] = j. We now repeat this inductively. That is, for each i given the w[1…i-1] that we have already constructed, the buyer sets w[i]=k s.t. P(w’[i]=k||w[1…i-1])<=½.

What’s the causal expected utility of submitting w thus constructed? Well, for one, because the procedure is deterministic (if ties are broken deterministically), the buyer can expect that she will receive a box at all. Now, for all i, the buyer thinks that if i is sampled by the seller for the purpose of determining which box to give to the buyer, then the buyer will in causal expectation receive $1.50, because the seller will predict the wrong box, i.e. w’[i] ≠ w[i], with probability at least ½.

Mozi, Mengzi, and Effective Altruism

On By Daniel KokotajloIn GeneralLeave a comment

[Author’s note: Another old paper makes it online!]

A while back I took a class on ancient chinese philosophy, taught by David Wong (who is great, by the way.) One of the things that Professor Wong noted was the similarity between modern-day ethical debates surrounding effective altruism and the ancient dispute between Mozi and Mengzi. I dug a little deeper and made a list of all the similarities I could think of; this is that list. Comments welcome:

https://docs.google.com/document/d/1xlGJ8_mV2vRB8ig1jIqsdWVPFcoNQXqYxBeBWc08NzE/edit?usp=sharing

I’m sufficiently impressed by these similarities that I think it is fair to say that Mozi and Mengzi really were talking about the same issues that feature prominently in debates about EA today; they were even making many of the same arguments and taking many of the same positions. I think this is really cool. If I ever teach intro to philosophy (or ancient philosophy, or non-western philosophy, or intro to ethics) in university, I intend to include a section on Mozi/Mengzi/EA.


Great Map of the Mind

On By Daniel KokotajloIn GeneralLeave a comment

We have all these theories and debates about parts of the mind; why not make a big map to show how they all fit together? 

Obviously, minds aren’t all the same. But having a map like this helps us organize our thoughts, to better understand our own minds and the minds we are trying to design and reason about. I’d love to see better versions, or elaborations of this one, or entirely different mind-designs.

If you want the original document: Get draw.io, then follow this link and click “Open with…” and then select draw.io. I’d love it if people spin off improved versions.