This page is an overview of the decision theory research we (Johannes Treutlein and Caspar Oesterheld) do. In particular, we discuss the decision theory of Newcomb-like scenarios and whether evidential decision theory is correct and if not, whether and how it can be fixed.

A short summary of our views

Overall, we think EDT gets almost everything right:

• “Reference class” Newcomb-like problems: One should cooperate in a prisoner’s dilemma against a copy and one-box in Newcomb’s problem.
• “Common cause” or medical Newcomb-like problems: We think EDT gets medical Newcomb problems, such as the Smoking lesion, right. We believe that most alleged medical Newcomb problems do not actually lead CDT and EDT to disagree, because of the tickle defense (Ahmed 2014, ch. 4). Perhaps it is possible to construct medical Newcomb-like problems where CDT and EDT actually disagree (i.e. where the tickle defense does not apply), such as Arif Ahmed’s “Betting on the Past”, the two-boxing gene, and the coin flip creation problem. However, in those problems, (the equivalent of) one-boxing can be defended for the same reasons it can be defended in reference class problems.
• Updatelessness: Some have argued that evidential decision theory gets counterfactual mugging (or similar problems such as Soares and Levinstein’s (2017, sect. 2) XOR blackmail) wrong. We, on the other hand, think that EDT’s behavior makes sense. We don’t think that reflective inconsistency is a conclusive argument against EDT, given that reflective consistency is at odds with other plausible principles.

However, there are also some challenges to EDT:

• Naturalized induction: EDT (and other non-logical decision theories) make their decisions based on some probability distribution P. In principle, Bayesian updating solves the problem of assigning these probabilities in the non-naturalized Cartesian case. However, two new problems arise in the naturalized case.
  • Building phenomenological bridges: Which collections of objects in the world model are instances of myself? The problem of translating between mental experiences and physics has been discussed at length in a few different fields and a few solutions have been proposed (see, e.g., the behaviorist approach to the BPB problem; this Brian Tomasik post; cf. the literatures surrounding “Mary the color scientist“, the computational theory of mind, computation in cellular automata, etc.). Unfortunately, it seems that statements about whether a particular physical process implements a particular algorithm can’t be objectively true or false. As an alternative, one could switch to a logical version of EDT. Unfortunately, switching from EDT to logical EDT has a lot of side-effects.
  • Anthropics: Anthropic decision theory (ADT) (Armstrong 2017), i.e. the way updateless decision theories automatically solve anthropics, seems satisfactory. In Sleeping Beauty-based anthropic decision problems, ADP’s behavior can be imitated by EDT+SSA (or CDT+SIA) (Schwarz 2015; Briggs 2010). However, without a precommitment to updatelessness, there seems to be no way to make EDT+SSA agree with ADT in an example by Conitzer (2015). Whereas it is apparent why one would want to be reflectively inconsistent in the counterfactual mugging, this is much less clear for these anthropic problems. Hence, we view Conitzer’s example as an open problem for EDT. A possible solution could be to assign reference classes according to the behaviorist approach to the BPB problem.
• Perhaps some version(s) of the 5 and 10 problem

Our work on decision theory

• Caspar Oesterheld (2015): Two-boxing, smoking and chewing gum in Medical Newcomb problems.
• Caspar Oesterheld (2016): Thoughts on Updatelessness.
• Caspar Oesterheld (2016): Environmental and Logical Uncertainty: Reported Environmental Probabilities as Expected Environmental Probabilities under Logical Uncertainty.
• Caspar Oesterheld (2017): Decision Theory and the Irrelevance of Impossible Outcomes.
• Johannes Treutlein (2017): Did EDT get it right all along? Introducing yet another medical Newcomb problem.
• Johannes Treutlein (2017): “Betting on the Past” by Arif Ahmed.
• Johannes Treutlein (2017): Anthropic uncertainty in the Evidential Blackmail.
• Caspar Oesterheld (2017): Are causal decision theorists trying to outsmart conditional probabilities?
• Caspar Oesterheld (2017): Multiverse-wide Cooperation via Correlated Decision Making. Only section 2 and parts of the appendix are about decision theory itself.
• Caspar Oesterheld (2017): Naturalized induction – a challenge for evidential and causal decision theory.
• Caspar Oesterheld (2017): A behaviorist approach to building phenomenological bridges.
• Decision Theory Research at FRI (2017). A two-part talk, describing two drafts of ours. Johannes talks about a wager for evidential decision theory and Caspar talks about decision theory and approval-directed agents. (slides)
• Caspar Oesterheld (2017): Futarchy implements evidential decision theory.
• Caspar Oesterheld (2018): Doing what has worked well in the past leads to evidential decision theory.
• Caspar Oesterheld (2018): A proof that every ex-ante-optimal policy is an EDT+SSA policy in memoryless POMPDs.
• Caspar Oesterheld (2018): The law of effect, randomization and Newcomb’s problem.
• Caspar Oesterheld (2018): Goertzel’s GOLEM implements evidential decision theory applied to policy choice
• Caspar Oesterheld and Vincent Conitzer (2019): Extracting Money from Causal Decision Theorists
• Caspar Oesterheld (2019): Robust Program Equilibrium. Theory and Decision 86(1), pp. 143–159.
• Caspar Oesterheld (2019): Approval-directed agency and the decision theory of Newcomb-like problems. Synthese Special Issue on Decision Theory and the Future of Artificial Intelligence.
• William MacAskill, Aron Vallinder, Caspar Oesterheld, Carl Shulman, and Johannes Treutlein: “The Evidentialist’s Wager“. The Journal of Philosophy 118 (6), pages 320–342, June 2021.
• Caspar Oesterheld and Vincent Conitzer: “Extracting Money from Causal Decision Theorists“. The Philosophical Quarterly 71 (4), pages 701–716, October 2021. Also presented at the IJCAI-PRICAI 2020 AI Safety workshop, GAMES 2020/1.
• Caspar Oesterheld, Abram Demski, and Vincent Conitzer (2021): “A theory of bounded inductive rationality“.
• James Bell, Linda Linsefors, Caspar Oesterheld, Joar Skalse (2021): Reinforcement Learning in Newcomblike Environments

Overview pages

• A comprehensive list of decision theories
• A survey of polls on Newcomb’s problem
• An overview of why we think that the Smoking Lesion doesn’t refute EDT
• An overview of introductions to the problem of naturalized agency
• Decision theory and artificial intelligence – an overview

Other valuable resources on decision theory

Here are some pieces on decision theory written by others that we find particularly valuable, although we may not agree with them entirely.

• Eliezer Yudkowsky (2008): Newcomb’s Problem and Regret of Rationality.
• Paul Almond (2010): On Causation and Correlation, Part 1: Evidential decision theory is correct.
• Arif Ahmed (2014): Evidence, Decision and Causality.
• Eliezer Yudkowsky (2010): Timeless Decision Theory.
• Stuart Armstrong (2017): Anthropic Decision Theory.

Acknowledgment

Some of our own work on decision theory – including the work necessary to create this summary page – was funded by the Foundational Research Institute (now Center on Long-Term Risk).