LikeLike

]]>LikeLike

]]>LikeLike

]]>LikeLike

]]>LikeLike

]]>Yeah, I also used to think along these lines, but now I have been convinced that it doesn’t make sense for a reasoner to try to hold an entire world in its head when the world may be bigger than the reasoner. Instead, the reasoner can use sentences to represent the world.

> Could you maybe elaborate on what you mean by this or point to some reading that will make it more clear?

We say that a (probability) measure $\mu$ dominates a (probability) measure $\nu$ if there is some constant $c$ such that for all events $A$, we have $\mu(A) \ge c \nu(A)$. We can think of this in terms of finite relative surprise; whatever observation we observe, we will receive at most $\log \frac{1}{c}$ additional bits of surprise if we believe $\mu$ relative to if we believe $\nu$. Thus, we may want to dominate all the probability measures in a large class. If we do, then we won’t end up being needlessly infinitely surprised. These issues are discussed more in connection with Solomonoff induction in Hutter’s work, e.g. in Rathmanner and Hutter, 2011 at . The notion of dominance is also used in the logical induction paper itself , in desideratum 8 and in section 4.6.

Regarding the particular claim I made, it refers to unpublished work. To state more clearly what I mean, taking the limit of a logical inductor as time goes to $\infty$ gives a measure on complete extensions of your theory (this is theorem 4.1.2 in Logical Induction, limits of logical induction also come up in section 4.6). The unpublished result is that given any two logical inductors, each of their limits dominates the other. There are also results relating to what happens at finite time, but these are more involved to state.

]]>> It is not that bad though; every logical inductor dominates every other logical inductor, in a sense closely related to the dominance relationship between universal semimeasures.

Could you maybe elaborate on what you mean by this or point to some reading that will make it more clear?

LikeLike

]]>Questions like whether the universe is countable are even worse, since this is not a property of your observation stream, but instead is about what exists. You could try to read answers to this sort of question off of the programs that Solomonoff induction uses to model the world, but this is even more fraught.

Logical induction instead makes predictions explicitly about any statement that can be expressed in your language, and has a more sophisticated way of using evidence to update its probabilities of these statements. It would give probabilities intermediate between 0 and 1 to statements like your observation stream being computable or the universe being countable (once we cash that out in some suitable language).

There is still a question of which logical inductors have reasonable priors about such questions. It is not that bad though; every logical inductor dominates every other logical inductor, in a sense closely related to the dominance relationship between universal semimeasures.

]]>Your main argument here seems to be that one cannot “meaningfully imagine what it would mean for there to be infinite value locations in a finite space.” I am not sure this is true. I feel like I can imagine a fractal, which consists of infinitely many self-similar structures in a finite space, for instance. If one values each structure independently of its size, then one must assign infinite value to the fractal. The fractal would not necessarily give rise to a wager, though, since there might only be countably many structures, and countable infinities do not suffice for there to be a wager.

The wager arises from the difference between a countable and an uncountable set. Since there is no surjective mapping from elements of a countable to an uncountable set, the uncountable set must somehow have “more” elements than the countable set. If one just abstractly considers sets of value locations, then it seems clear that there can hence be “more” value in the uncountable set. The question is whether there is actually a universe one would represent by such an uncountable set, or whether this is a useful abstraction in the first place. For instance, consider the cellular automaton with uncountably many cells I introduce in the post. One might care about the happenings in this automaton’s cells in a different way than one would care about the happenings in an automaton with only countably many cells. Maybe the interaction rules in such an automaton are about sets of cells and some property of such a set instead of individual cells. If one were to derive some measure of length in this automaton based on these sets of cells, then indeed the wager would imply that there must be uncountably many value locations in a finite region of space.

In general, uncountable infinities are hard to imagine, even more so than countable infinities. Intuitions about universes with such properties are either non-existent or unreliable and subject to specific framings. This does not mean that the wager does not work, but it means that it is hard to judge whether the wager works or not. I think there is a realistic possibility that after careful reflection and systematizing my intuitions about these issues, I would arrive at the conclusion that there can actually be a universe which I would represent as an uncountable set of value locations.

Regarding your more general arguments, I agree one should not buy the wager just from reading one blog post about it. Nevertheless, this does ring true of any wager, so there is still the need to think about this specific wager more on the object level. As to the consequences about AI, they heavily depend on the specific goal function. If it is not clear what the AI’s goal function would imply about incomputable universes, then it might be a good safety measure to exclude them from the AI’s prior. If one can be confident about the AI’s goal function even in the case of incomputable universes, then it might be good to include them. In either case, the real-world consequences are hard to estimate.

LikeLike

]]>So I’d be substantially more willing to buy a wager that we should reassess aggregative consequentialism, rather than that that there are universes with different mathematics from ours that somehow “matter” infinitely more than ours. “Matter” here is a term about my values and if I knew these universes existed, I wouldn’t be sure I’d want them to swamp everything that I understand in favor of things I can’t even properly imagine.

Sure, you can add that there’s a very small chance that the scenarios are still comparable or that the counterarguments are wrong, but this next-level conjunctive. If the resulting action-recommendation was “benign” and only changed our all-things-considered judgment about something by a few percentage points, sure, then the parts of my epistemological parliament who are sensitive to such considerations should have their say. But I wouldn’t want to have my entire epistemology be swamped by one wager, and if I understand it correctly, this action-recommenation may do that? Or what would an AI start to do if you give it aggregative consequentialism plus these non-zero priors? It seems like the wrong kind of fanaticism to buy this wager. I’d rather question our reasoning than accept it, even if I cannot point to a flaw that instantly blows it out of the water. 🙂

Or maybe you had in mind something more benign where the AI would have similar safeguards against fanatical wagers than people usually have?

LikeLike

]]>