The universal prior assigns zero probability to non-computable universes—for instance, universes that could only be represented by Turing machines in which uncountably many locations need to be updated, or universes in which the halting problem is solved in physics. While such universes might very likely not exist, one cannot justify assigning literally zero credence to their existence. I argue that it is of overwhelming importance to make a potential AGI assign a non-zero credence to incomputable universes—in particular, universes with uncountably many “value locations”.
Here, I assume a model of universes as sets of value locations. Given a specific goal function, each element in such a set could specify an area in the universe with some finite value. If a structure contains a sub-structure, and both the structure and the sub-structure are valuable in their own regard, there could either be one or two elements representing this structure in the universe’s set of value locations. If a structure is made up of infinitely many sub-structures, all of which the goal function assigns some positive but finite value to, then this structure could (if the sum of values does not converge) possibly only be represented by infinitely many elements in the set. If the set of value locations representing a universe is countable, then the value of said universe could be the sum over the values of all elements in the set (granted that some ordering of the elements is specified). I write that a universe is “countable” if it can be represented by a finite or countably infinite set, and a universe is “uncountable” if it can only be represented by an uncountably infinite set.
A countable universe, for example, could be a regular cellular automaton. If the automaton has infinitely many cells, then, given a goal function such as total utilitarianism, the automaton could be represented by a countably infinite set of value locations. An uncountable universe, on the other hand, could be a cellular automaton in which there is a cell for each real number, and interactions between cells over time are specified by a mathematical function. Given some utility functions over such a universe, one might be able to represent the universe only by an uncountably infinite set of value locations. Importantly, even though the universe could be described in logic, it would be incomputable.
Depending on one’s approach to infinite ethics, an uncountable universe could matter much more than a countable universe. Agents in uncountable universes might—with comparatively small resource investments—be able to create (or prevent), for instance, amounts of happiness or suffering that could not be created in an entire countable universe. For instance, each cell in the abovementioned cellular automaton might consist of some (possibly valuable) structure in of itself, and the cells’ structures might influence each other. Moreover, some (uncountable) set of cells might be regarded as an agent. The agent might then be able to create a positive amount of happiness in uncountably many cells, which—at least given some definitions of value and approaches to infinite ethics—would have created more value than could ever be created in a countable universe.
Therefore, there is a wager in favor of the hypothesis that humans actually live in an uncountable universe, even if it appears unlikely given current scientific evidence. But there is also a different wager, which applies if there is a chance that such a universe exists, regardless of whether humans live in that universe. It is unclear which of the two wagers dominates.
The second wager is based on acausal trade: there might be agents in an uncountable universe that do not benefit from the greater possibilities of their universe—e.g., because they do not care about the number of individual copies of some structure, but instead care about an integral over the structures’ values relative to some measure over structures. While agents in a countable universe might be able to benefit those agents equally well, they might be much worse at satisfying the values of agents with goals sensitive to the greater possibilities in uncountable universes. Thus, due to different comparative advantages, there could be great gains from trade between agents in countable and uncountable universes.
The above example might sound outlandish, and it might be flawed in that one could not actually come up with interaction rules that would lead to anything interesting happening in the cellular automaton. But this is irrelevant. It suffices that there is only the faintest possibility that an AGI could have an acausal impact in an incomputable universe which, according to one’s goal function, would outweigh all impact in all computable universes. There probably exists a possible universe like that for most goal functions. Therefore, one could be missing out on virtually all impact if the AGI employs Solomonoff induction.
There might not only be incomputable universes represented by a set that has the cardinality of the continuum, but there might be incomputable universes represented by sets of any cardinality. In the same way that there is a wager for the former, there is an even stronger wager for universes with even higher cardinalities. If there is a universe of highest cardinality, it appears to make sense to optimize only for acausal trade with that universe. Of course, there could be infinitely many different cardinalities, so one might hope that there is some convergence as to the values of the agents in universes of ever higher cardinalities (which might make it easier to trade with these agents).
In conclusion, there is a wager in favor of considering the possibility of incomputable universes: even a small acausal impact (relative to the total resources available) in an incomputable universe could counterbalance everything humans could do in a computable universe. Crucially, an AGI employing Solomonoff induction will not consider this possibility, hence potentially missing out on unimaginable amounts of value.
Acknowledgements
Caspar Oesterheld and I came up with the idea for this post in a conversation. I am grateful to Caspar Oesterheld and Max Daniel for helpful feedback on earlier drafts of this post.
The Universe from an Intentional Stance 
I like the reasoning in your other post on wagers (https://casparoesterheld.com/2018/03/31/three-wagers-for-multiverse-wide-superrationality/), but with this one I am extremely skeptical. I don’t think we can meaningfully imagine what it would mean for there to be infinite value locations in a finite space. And while you can make Pascalian arguments about how maybe that’s just a failure of imagination, I’d say that if we’re wrong about something so fundamental, we shouldn’t assume that our current concepts would survive the ontological crisis. Wagers only work if the unit of value given hypothesis 1 is the same unit of value as the value given hypothesis 2. But if we’re wrong about these infinite value locations, that may imply different solutions to infinite ethics, or different formalizations of how we want to use probabilities, etc. It’s not clear whether our current concepts can handle that.
So I’d be substantially more willing to buy a wager that we should reassess aggregative consequentialism, rather than that that there are universes with different mathematics from ours that somehow “matter” infinitely more than ours. “Matter” here is a term about my values and if I knew these universes existed, I wouldn’t be sure I’d want them to swamp everything that I understand in favor of things I can’t even properly imagine.
Sure, you can add that there’s a very small chance that the scenarios are still comparable or that the counterarguments are wrong, but this next-level conjunctive. If the resulting action-recommendation was “benign” and only changed our all-things-considered judgment about something by a few percentage points, sure, then the parts of my epistemological parliament who are sensitive to such considerations should have their say. But I wouldn’t want to have my entire epistemology be swamped by one wager, and if I understand it correctly, this action-recommenation may do that? Or what would an AI start to do if you give it aggregative consequentialism plus these non-zero priors? It seems like the wrong kind of fanaticism to buy this wager. I’d rather question our reasoning than accept it, even if I cannot point to a flaw that instantly blows it out of the water. 🙂
Or maybe you had in mind something more benign where the AI would have similar safeguards against fanatical wagers than people usually have?
LikeLike
Thanks for your comment! As I say in the post, whether this is a wager or not depends on one’s definition of value and how one handles infinities. I agree that when evaluating the wager, one needs to take into account all consequences of the hypothesis that the universes in question exist. For example, given such an ontology, the most plausible definition of value might be one according to which there is no wager. I am unsure whether this is the case, though.
Your main argument here seems to be that one cannot “meaningfully imagine what it would mean for there to be infinite value locations in a finite space.” I am not sure this is true. I feel like I can imagine a fractal, which consists of infinitely many self-similar structures in a finite space, for instance. If one values each structure independently of its size, then one must assign infinite value to the fractal. The fractal would not necessarily give rise to a wager, though, since there might only be countably many structures, and countable infinities do not suffice for there to be a wager.
The wager arises from the difference between a countable and an uncountable set. Since there is no surjective mapping from elements of a countable to an uncountable set, the uncountable set must somehow have “more” elements than the countable set. If one just abstractly considers sets of value locations, then it seems clear that there can hence be “more” value in the uncountable set. The question is whether there is actually a universe one would represent by such an uncountable set, or whether this is a useful abstraction in the first place. For instance, consider the cellular automaton with uncountably many cells I introduce in the post. One might care about the happenings in this automaton’s cells in a different way than one would care about the happenings in an automaton with only countably many cells. Maybe the interaction rules in such an automaton are about sets of cells and some property of such a set instead of individual cells. If one were to derive some measure of length in this automaton based on these sets of cells, then indeed the wager would imply that there must be uncountably many value locations in a finite region of space.
In general, uncountable infinities are hard to imagine, even more so than countable infinities. Intuitions about universes with such properties are either non-existent or unreliable and subject to specific framings. This does not mean that the wager does not work, but it means that it is hard to judge whether the wager works or not. I think there is a realistic possibility that after careful reflection and systematizing my intuitions about these issues, I would arrive at the conclusion that there can actually be a universe which I would represent as an uncountable set of value locations.
Regarding your more general arguments, I agree one should not buy the wager just from reading one blog post about it. Nevertheless, this does ring true of any wager, so there is still the need to think about this specific wager more on the object level. As to the consequences about AI, they heavily depend on the specific goal function. If it is not clear what the AI’s goal function would imply about incomputable universes, then it might be a good safety measure to exclude them from the AI’s prior. If one can be confident about the AI’s goal function even in the case of incomputable universes, then it might be good to include them. In either case, the real-world consequences are hard to estimate.
LikeLike
My post-logical induction picture is that we should only take Solomonoff induction’s statements seriously when they are about the next few observations that you will make. More “global” questions, such as whether the sequence of all observations is computable, are outside of the scope of Solomonoff induction, since it’s not selecting between hypotheses in ways that test whether they get these questions right. Relatedly, Solomonoff induction’s hypotheses don’t make predictions directly about these global questions; you have to ask for all the observations and then check whether they satisfy some property.
Questions like whether the universe is countable are even worse, since this is not a property of your observation stream, but instead is about what exists. You could try to read answers to this sort of question off of the programs that Solomonoff induction uses to model the world, but this is even more fraught.
Logical induction instead makes predictions explicitly about any statement that can be expressed in your language, and has a more sophisticated way of using evidence to update its probabilities of these statements. It would give probabilities intermediate between 0 and 1 to statements like your observation stream being computable or the universe being countable (once we cash that out in some suitable language).
There is still a question of which logical inductors have reasonable priors about such questions. It is not that bad though; every logical inductor dominates every other logical inductor, in a sense closely related to the dominance relationship between universal semimeasures.
LikeLiked by 1 person
Thanks for the comment! I was thinking about it in the way that the programs that predict the observations are the world models (and these world models must be computable). It makes sense to me though that such questions are better addressed by logical induction.
> It is not that bad though; every logical inductor dominates every other logical inductor, in a sense closely related to the dominance relationship between universal semimeasures.
Could you maybe elaborate on what you mean by this or point to some reading that will make it more clear?
LikeLike
> I was thinking about it in the way that the programs that predict the observations are the world models (and these world models must be computable).
Yeah, I also used to think along these lines, but now I have been convinced that it doesn’t make sense for a reasoner to try to hold an entire world in its head when the world may be bigger than the reasoner. Instead, the reasoner can use sentences to represent the world.
> Could you maybe elaborate on what you mean by this or point to some reading that will make it more clear?
We say that a (probability) measure $\mu$ dominates a (probability) measure $\nu$ if there is some constant $c$ such that for all events $A$, we have $\mu(A) \ge c \nu(A)$. We can think of this in terms of finite relative surprise; whatever observation we observe, we will receive at most $\log \frac{1}{c}$ additional bits of surprise if we believe $\mu$ relative to if we believe $\nu$. Thus, we may want to dominate all the probability measures in a large class. If we do, then we won’t end up being needlessly infinitely surprised. These issues are discussed more in connection with Solomonoff induction in Hutter’s work, e.g. in Rathmanner and Hutter, 2011 at . The notion of dominance is also used in the logical induction paper itself , in desideratum 8 and in section 4.6.
Regarding the particular claim I made, it refers to unpublished work. To state more clearly what I mean, taking the limit of a logical inductor as time goes to $\infty$ gives a measure on complete extensions of your theory (this is theorem 4.1.2 in Logical Induction, limits of logical induction also come up in section 4.6). The unpublished result is that given any two logical inductors, each of their limits dominates the other. There are also results relating to what happens at finite time, but these are more involved to state.
LikeLiked by 1 person