Futarchy is a meta-algorithm for making decisions using a given set of traders. For every possible action a, the beliefs of these traders are aggregated using a prediction market for that action, which, if a is actually taken, evaluates to an amount of money that is proportional to how much utility is received. If a is not taken, the market is not evaluated, all trades are reverted, and everyone keeps their original assets. The idea is that – after some learning and after bad traders lose most of their money to competent ones – the market price for a will come to represent the expected utility of taking that action. Futarchy then takes the action whose market price is highest.
For a more detailed description, see, e.g., Hanson’s (2007) original paper on the futarchy, which also discusses potential objections. For instance, what happens in markets for actions that are very unlikely to be chosen? Note, however, that for this blog post you’ll only need to understand the basic concept and none of the minutia of real-world implementation. The above description deliberately ignores and abstracts away from these. One example of such a discrepancy between standard descriptions of futarchy and my above account is that, in real-world governance, there is often a “default action” (such as, leave law and government as is). To keep the number of markets small, markets are set up to evaluate proposed changes relative to that default (such as the introduction of a new law) rather than simply for all possible actions. I should also note that I only know basic economics and am not an expert on the futarchy.
Traditionally, the futarchy has been thought of as a decision-making procedure for governance of human organizations. But in principle, AIs could be built on futarchies as well. Of course, many approaches to AI (such as most Deep Learning-based ones) already have all their knowledge concentrated into a single entity and thus don’t need any procedure (such as democracy’s voting or futarchy’s markets) to aggregate the beliefs of multiple entities. However, it has also been proposed that intelligence arises from the interaction and sometimes competition of a large number of simple subagents – see, for instance, Minsky’s book The Society of Mind, Dennett’s Consciousness Explained, and the modularity of mind hypothesis. Prediction markets and futarchies would be approaches to (or models of) combining the opinions of many of these agents, though I doubt that the human mind functions like either of the two. A theoretical example of the use of prediction markets in AI is MIRI’s logical induction paper. Furthermore, markets are generally similar to evolutionary algorithms.1
So, if we implement a futarchy-like system in an AI, what decision theory would that AI come to implement? It seems that the answer is EDT. Consider Newcomb’s problem as an example. Traders that predict one-boxing to yield a million and two-boxing to yield a thousand will earn money, since the agent will, in fact, receive a million if it one-boxes and a thousand if it two-boxes. More generally, the futarchy rewards traders based on how accurately they predict what is actually going to happen if the agent makes a particular choice. This leads the traders to estimate the value of an action as proportional to the expected utility conditional on that action since conditional probabilities are the correct way to make predictions.
There are some caveats, though. For instance, prediction markets only work if the question at hand can eventually be answered. Otherwise, the market cannot be evaluated. For instance, in Newcomb’s problem, one would usually assume that your winnings are eventually given and thus shown to you. But other versions of Newcomb’s problems are conceivable. For instance, if you are consequentialist, Omega could donate your winnings to your favorite charity in such a way that you will never be able to tell how much utility this has generated for you. Unless you simply make estimates – in which case the behavior of the markets depends primarily on what kind of expected value (regular or causal) you will use as an estimate –, you cannot set up a prediction market for this problem at all. An example of such a “hidden” Newcomb problem is cooperation via correlated decision making between distant agents.
Another unaddressed issue is whether the futarchy can deal correctly with other problems of space-time embedded intelligence, such as the BPB problem.
Notwithstanding the caveats, EDT seems to be an inherent the way the futarchy works. To get the futarchy to implement CDT, it would have to reward traders based on what the agent is causally responsible for or based on some untestable counterfactual (“what would have happened if I had two-boxed”). Whereas EDT arises naturally from the principles of the futarchy, other decision theories require modification and explicit specification.
I should mention that this post is not primarily intended as a futarchist argument for EDT. Most readers will already be familiar with the underlying pro-EDT argument, i.e., EDT making decisions based on what will actually happen if a particular decision is made. In fact, it may also be viewed as a causalist argument against the futarchy.2 Rather than either of these two, it is a small part of the answer to the “implementation problem of decision theory”, which is: if you want to create an AI that behaves in accordance to some particular decision theory, how should that AI be designed? Or, conversely, if you build an AI without explicitly implementing a specific decision theory, what kind of behavior (EDT or CDT or other) results from it?
Acknowledgment: This work was funded by the Foundational Research Institute (now the Center on Long-Term Risk).
1. There is some literature comparing the way markets function to evolution-like selection (see the first section of Blume and Easley 1992) – i.e., how irrational traders are weeded out and rational traders accrue more and more capital. I haven’t read much of that literature, but the main differences between the futarchy and evolutionary algorithms seem to be the following. First, the futarchy doesn’t specify how new traders are generated, because it classically relies on humans to do the betting (and the creation of new automated trading systems), whereas this is a central concern in evolutionary algorithms. Second, futarchies permanently leave the power in the hands of many algorithms, whereas evolutionary algorithms eventually settle for one. This also means that the individual traders in a futarchy can be permanently narrow and specialized. For instance, there could be traders who exploit a single pattern and rarely bet at all. I wonder whether it makes sense to combine evolutionary algorithms and prediction markets. ↩
2. Probably futarchist governments wouldn’t face sufficiently many Newcomb-like situations in which the payoff can be tested for the difference to be relevant (see chapter 4 of Arif Ahmed’s Evidence, Decision and Causality). ↩
The Universe from an Intentional Stance 
Isn’t a key difference between EDT and futarchy that a futarchy chooses a policy that it then sticks to for at least some period, rather than making each choice distinctly?
Consider Parfit’s Hitchhiker. EDT of course fails, since it will choose not to pay. However, if there is a futarchy market on what policy to follow while in the desert, the traders will of course “vote” for “pay him when you get to the city”, properly invoking pre-commitment. However, if the market simply chooses what to say to Parfit in the desert and the market is in the city after, the hiker dies. So the method of binding seems important here, as do the choices made available (e.g. Futarchy depends on its candidates).
In a more general case, if you had a market on “what decision theory should we use?” and the market was wise, it presumably would say that FDT produces better results than EDT, and self-modify, the same way a sufficiently wise EDT (or CDT) agent would do so, but only if such market exists. This suggests that Futarchy is very much a function of which markets are chosen. Of course, you could have a market on that…
LikeLiked by 1 person
Yes, the futarchy in politics is (usually) conceived of as making choices between more long-term policies. But to me it seems that whether one chooses policies or actions (or something in between) is orthogonal to the essence of EDT and the futarchy as theoretical ideas. (Well, not quite. Feedback is essential for markets, so it would be awkward to use the futarchy for a single policy choice. One could, however, “train” the market on some other decisions (potentially ones about entire policies or just whether to eat an apple or a banana) and then apply it to a policy choice.) It’s mostly a matter of whether precommitment is feasible or not. For instance, in Parfit’s Hitchhiker, both EDT and the futarchy precommit if they can, and both keep the money once they are in the city if they can.
Yes, it’s crucial that futarchy and EDT have the seme set of options. (In the post, I assume that the futarchy has a prediction market for every possible action of the decision problem.) My claim is that, in general, if EDT and a futarchy-based agent have the same set of options, they would choose the same action (assuming that the futarchy has “practiced” on sufficiently many sufficiently similar problems, etc.).
>In a more general case, if you had a market on “what decision theory should we use?” and the market was wise, it presumably would say that FDT produces better results than EDT, and self-modify, the same way a sufficiently wise EDT (or CDT) agent would do so, but only if such market exists.
Minor points (that you’re probably aware of):
-> EDT (and thus also idealized futarchy) would self-modify into “updateless EDT”. (This is under the assumption that the environment contains no objects that “punish” agents for their source code (rater than merely for the behavior implied by that source code). So, for example, there is no Newcomb-like problem in which you lose money if your source code is that of updateless EDT.) FDT hasn’t been formally specified, so it’s a bit hard to tell whether FDT and updateless EDT are the same. I suspect they’re not, though, because FDT is supposed to be updatelessless w.r.t. its own existence (see https://www.lesserwrong.com/posts/4MYYr8YmN2fonASCi/you-re-in-newcomb-s-box and http://lesswrong.com/lw/jrm/the_sin_of_updating_when_you_can_change_whether/ and http://lesswrong.com/lw/pft/naturalized_induction_a_challenge_for_evidential/ ), which standard versions of EDT and thus updateless EDT aren’t. Also, updateless EDT is relatively easy to formalize (modulo various issues related to self-locating beliefs), so one would expect that if that’s what anybody means with FDT/UDT/…, then there wouldn’t be as much discussion about how to formalize FDT/UDT/…
-> CDT will probably commit to something even farther away from FDT, because, e.g., it doesn’t cooperate with copies that aren’t created on the bases of the agent’s causal future. Thus, “updateless CDT” doesn’t do any acausal trade ( https://wiki.lesswrong.com/wiki/Acausal_trade ) with aliens.
LikeLike
I agree that prediction markets are based on EDT, but I see this as an argument against futarchy. I wrote about this here: http://lesswrong.com/lw/lm0/prediction_markets_are_confounded_implications/
In general, I am puzzled by your ongoing defence of EDT, and I wonder if we may be using the term to refer to different things.
You write that “Most readers will already be familiar with the underlying pro-EDT argument, i.e., EDT making decisions based on what will actually happen if a particular decision is made.”
In my conception of EDT, this sentence is trivially false. In fact, CDT is the decision theory that makes decisions based on what will happen if a particular decision is made, and this is the key difference between EDT and CDT. Therefore, this sentence suggests that we are probably using the EDT/CDT distinction to mean different things.
LikeLiked by 1 person
Thanks for your comment!
>I agree that prediction markets are based on EDT, but I see this as an argument against futarchy. I wrote about this here: http://lesswrong.com/lw/lm0/prediction_markets_are_confounded_implications/
Cool, seems like you’ve noticed this much earlier than I have. Sorry for overlooking your post. I’ve written a comment to it.
>In my conception of EDT, this sentence is trivially false. In fact, CDT is the decision theory that makes decisions based on what will happen if a particular decision is made, and this is the key difference between EDT and CDT. Therefore, this sentence suggests that we are probably using the EDT/CDT distinction to mean different things.
Interesting! I always thought that the “CDT makes the right predictions” view was mostly a one-boxer’s straw man of the two-boxer’s position, but it seems that the view is more common than I thought.
Let’s take Newcomb’s problem (though the same works for any other problem in which CDT and EDT disagree). Let’s say the predictor has 90% accuracy. Then after one-boxing, an agent gets 1M 90% of the time (or with 90% probability) and nothing 10% of the time. After two-boxing, an agent gets 1K 90% of the time and 1M+1K 10% of the time. I assume this is uncontroversial as it merely describes Newcomb’s problem. Consequently, the EDT-EVs of one- and two-boxing are 90%*1M=900K and 1K+10%*1M=101K, respectively.
(The causal decision theorist agrees with this. After having taken her action, the causal decision theorist will expect to receive a million with 10% probability and will make decisions based on this probability. Cf. Pearl’s distinction between “actions” and “acts” in section 4.1.1 of his book “Causality”.)
Now, let’s look at how well a causal decision theorist predicts itself to fare if it two-boxes:
E[two-box->payoff] = 1K + P(two-box->1M)*1M = 1K+P(1M)*1M,
since two-boxing is assumed to be causally independent of the content of the box. Thus, unless the prior of a million being under the opaque box P(1M) happens to be 10%, the CDT-EV differs from how much the CDT agent should actually expect to achieve.
LikeLike
Carl Shulman comments: “I have seen decision market descriptions where you use an RNG some portion of the time to assign policy randomly, and go with the conditional predictions for random assignment.
E.g. on Newcomb, you might use a d100: if it comes up 1 then you one-box, if it comes up 2 you two-box, and if it comes up 3-100 you one-box or two-box depending on prediction markets’ conditional forecasts for outcomes given the die coming up 1 or 2.
The results then will depend heavily on the details on whether Omega can predict your RNGs, and its conditional behavior in response to your using RNGs it can’t predict.”
LikeLike
That’s a good point. I hadn’t thought of that, though in other contexts, such as RL, it is quite easy to see how similar randomization may be used to yield more CDT-ish behavior.
I found a few relevant papers on decision markets and randomization, though the randomization is done in a way that differs slightly from the decision marekt you describe.
Theorem 2 in a paper by Chen et al. ( https://www.eecs.harvard.edu/~shnayder/papers/decision-wine11-full.pdf) suggests that (under certain assumptions) setting proper incentives requires that every action is taken with non-zero probability regardless of the predictions made. It seems that this already leads to a CDT-ish market. At least, if the current market bets were such that the futarchy agent would probably only take one box, then it seems traders would want to bet on two-boxing yielding 1M+1K. That is, assuming that the predictor fills boxes in dependence on the probability with which the futarchy agent one/two-boxes, i.e. that the market uses a RNG that the predictor cannot predict. So the market state consistent with EDT-EVs seems unstable.
Whereas the market you proposed seems to make systematically inaccurate predictions about what’s going to happen (because that’s not what it is actually betting on), it seems that markets of the above kind (if constructed according to theorem 3 of the paper) may actually make correct predictions and still not one-box for ratifiability-type reasons (i.e., “if I probably one-box, then two-boxing is actually better”).
I might look into this further because I don’t think this is the last word on either decision market manipulation or futarchy and Newcomb’s problem (and even so, ratifiability would be an interesting last word). For example, one thing to note about the Chen et al. paper is that the kind of manipulation they describe in the introduction (betting against the best option so that it receives zero probability) is actually much more harmful than the kind of manipulation they actually prove to be possible (reporting arbitrary beliefs on actions that will not be taken anyway). A more interesting kind of manipulation/non-truthful reporting is that described by Othman and Sandholm (2010, sect 3.2 https://pdfs.semanticscholar.org/b814/f7a98dc80301d2ce320f0ed60fdd279bdf35.pdf), but even there I think that alternative constructions might be able avoid the problem.
LikeLike
Pingback: The law of effect, randomization and Newcomb’s problem – The Universe from an Intentional Stance
Robin Hanson says ( https://twitter.com/robinhanson/status/965625971888959488 ): “Yes futarchy implements an evidential decision theory, not a causal decision theory. I thought that was obvious.”
LikeLike
Pingback: AI Alignment Prize: Round 2 due March 31, 2018 | Don't Worry About the Vase
Pingback: Goertzel’s GOLEM implements evidential decision theory applied to policy choice – The Universe from an Intentional Stance
Pingback: If I Was An Eccentric Trillionaire (niplav, 2023) – Veritas Reporters