Impossibility Results in AI: A Survey

Limits of deduction are limits on our capability to achieve perfect certainty in facts. The vast majority of listed results here use Lawvere’s fixed-point theorem [59, 110] as the basis of proofs, i.e., more specifically, a combination of Cantor’s diagonalization and liar’s paradox.

The most basic results here are Gödel’s incompleteness theorems [40] addressing unprovability and Turing’s work [90] covering undecidability. In addition to the aforementioned that cover the processing, Gregory Chaitin provided additional incompleteness results [20] that cover input sizes measured by Kolmogorov’s complexity. Chaitin’s incompleteness theorem states the existence of a limit on any formal system to prove Kolmogorov complexity of strings beyond some length. There are even conjectures that information complexity might be the source of incompleteness [16, 20] whereby the theorems of finitely stated theories cannot be significantly more complex than the theory itself.

Rice’s theorem [77] is a generalization of Turing’s undecidability of the halting property of programs to any sufficiently complex property. This makes it an ideal tool for finding and proving deduction limitations in AI, within its assumptions. Löb’s theorem [64], informally put, states that a formal consistent system cannot, in general, prove its own soundness.

Regarding formal semantics, Tarski’s undefinability theorem [86] states that truth in a formal system cannot be defined within that system. Measuring semantic information yield by deduction is also poised by paradoxes. Bar-Hilel-Carnap paradox [19] in classical semantic information theory entails that contradiction conveys maximal information. Hintikka’s scandal of deduction [44] points to the fact that the information yield of truthful sentences is zero, since their information is already contained in the premises.

There is vast work in impossibility theorems in beliefs, which is an extension of Gödel’s work. Huynh and Szentes [46] have demonstrated irreconcilability between two notions of self-belief. In References [2, 13] the paradox of self-reference was extended to the games with two or more players that yield impossible beliefs.

Deductive impossibility results have been found in machine learning as well regarding the properties of learning algorithms and processes given the data. Authors in References [11, 76] have shown that learnability can be undecidable and unprovable for certain problems. Finding if an encodable learning algorithm always underfits a dataset is undecidable [81], even with unlimited training time.

Inference devices covered in the series of papers by Wolpert [98–100] are an extension from pure deductive systems to the general notion of inference devices, which covers deduction, induction (prediction, retrodiction), observation, and control while the device itself is embedded in a physical universe. Additional limitations are found both in a logical and stochastic sense. For example, limits are found on strong and weak inference, control, self-control, and mutual control between the two distinguishable devices. The limits are also put on prediction, retrodiction, observation, and knowledge. This work was extended in Reference [25] to find the constraints on modeling systems from within systems using the frame of relativity. The concluding impossibility theorem states that the universe cannot be completely modeled from within the universe.

Observability is the ability to infer the state of a black-box system from its input/output data. Identifiability (i.e., parameter and/or structural identifiability) is a special case of observability for constant elements of the system whereby we only need to infer the values of those constant elements. There are limits to both observability and identifiability, and the limits are caused by non-injective mappings [12], which inevitably lose information.

Identifiability and observability are important for the control over systems. Controllability in control theory can be of two kinds: State and output. State controllability is the ability to control the inner state of the system. Output controllability is the ability to control the output of the system. State uncontrollability is the dual of unobservability. That is, state controllability is impossible without observability [52, 53].

Good regulator theorem [23] relates output controllability and identifiability (modeling), but only in a sense of optimal control, not sufficient control. It states that maximally simple among optimal regulators must behave as an image of the controlled system under a homomorphism. Sufficiently good regulators need not be optimal, and the generalization of such theorem would be interesting.

Law of requisite variety [8] states that variety in outputs can only be reduced by the state complexity of the controlling mechanism. Information theory state-control limit [88] says that only up to information observed from the system can be used to reduce the entropy of the system.

In the absence of additional biases, general nonlinear independent component analysis has an infinity of solutions that are indistinguishable [47]. Similar is shown for unsupervised learning of disentangled representations [63]. There are also well-known non-identifiability limits to causal discovery from the data [73].

Limits on induction constrain our ability to infer latent factors. Here, we will ignore the problem of indistinguishability by just looking at equivalence classes of models indistinguishable in the limit. In this case, there is a possibility to learn the true equivalence class asymptotically. But, given some prefix of experience, there may be a multitude of candidate classes. This is pointed out in Hume’s problem of induction and Goodman’s new riddle of induction [38]. No free lunch theorems (NFL), by Wolpert [97, 102], are the basic building blocks underlying the formalization of these limitations. They were first formulated in general supervised learning and optimization, which were subsequently unified through that framework. No free lunch theorems state that under uniform distribution over induction problems, all induction algorithms perform equally [101]. At the heart of NFL formalization is the independence of (search/learning) algorithm performance from the uncertain knowledge of the true problem at hand. That independence is materialized in the inner product formula of those two in describing the probability of attaining a performance value over the unknown problem. There are, however, free lunches if more structure is imposed on the problem, i.e., “there is no learning without bias, there is no learning without knowledge” [26]. For example, there are free lunches in continuous spaces [9], in quantum machine learning by utilizing entanglement [82, 95], in coevolutionary problems [103], under certain regularity assumptions in meta-learning [36, 79, 80], and under the assumption of Occam’s razor validity [57].

Goodhart-Strathern’s law [37, 84] and Campbell’s law [17] deal with the difficulties and the inability in finding expressible proxy numerical measures for success that are well aligned with inexpressible/unknown-explicitly experiential measure of success. A similar sentiment is expressed in Reference [83] where a more detailed explanation is given for the observed difficulties, all stemming from the unpredictability of solution-routes to hard or even unknown problems. Metric is a model of an imagined success, but shallow and not with perfect alignment.

In games with uncertainty in opponent’s payoffs, it is impossible to predict the behavior of perfectly rational agents due to the feedback loop emanating from their own decisions that influence opponent’s behavior [34]. Placing further restrictions on the assumptions can regain predictability. In economic situations, further limits relating to rationality, predictability, and control were proved in Reference [56]. Therein, (i) logical limits were set to forecasting the future, (ii) non-convergence of Bayesian forecasting in infinite-dimensional space was demonstrated, and (iii) impossibility of computer perfectly forecasting economy if agents know its forecasting program. These results are related to the already mentioned results in deduction-related impossibility theorems for Wolpert’s inference devices and regarding beliefs in games.

Anticodifiability thesis [65, 66, 89] is a conjecture in moral philosophy that states that universal morality cannot be codified in a way that would be aligned in all circumstances with our inexpressible/unknown-explicitly experiential moral intuition.

Tradeoff limits constrain our attempts to achieve perfect outcomes. Examples include impossibility theorems in clustering [54], fairness [55, 78], and social choice theory (SCT) [7]. In many situations, we have to choose with respect to multiple criteria simultaneously. Often, it is the case that there is no ideal point that simultaneously optimizes all the criteria, in other words, achieves maximal possible hypervolume indicator [39].

In social choice theory, there are results such as Arrow’s impossibility theorem [7], which states there must be a tradeoff that forces choosing only a strict subset of desirable properties in voting mechanisms. In moral theory, there are different problems regarding population ethics [6] where all total orderings entail some problematic properties that contradict our intuitions. Solutions have been proposed for automated systems that search for impossibility theorems in SCT regarding rankings of objects [35].

Intractability limits divide possibility-in-principle and practically impossible due to the resource limitations. There are three types of intractability impossibility results: asymptotic, physics-based, and human-centered.

Asymptotic intractabilities fall neatly under the complexity theory [1, 51]. That research field is simply too rich to expand on it here. We shall only highlight the probably approximately correct (PAC) learning framework [91] by Valiant that defines the border between efficient (polynomial time) and inefficient learnability. Of our interest is also the intractability of bottom-up ethics [14, 75], which stems from the game-theoretic nature of ethics. Lin et al. in Reference [61] have dealt with possible reasons and situations when deep learning works efficiently and when does it necessitate an exponential number of parameters in the number of variables. They prove various “no-flattening theorems” that show loss of efficiency when approximating deep networks with shallow ones. Calude et al. have extended the latter work in Reference [15] by working on sensitive and robust Boolean functions, whereby especially parity function poses exponential difficulties for learning by multilayered perceptrons with single hidden layer.

Physics-based limits put bounds on physically implementable computation and intelligence. No-go theorems state constraints for certain implementation approaches. Currently, these limits (e.g., Reference [62]) are quite loose and/or specific so we do not go into their details. One exception is the work of Wolpert we have previously mentioned [98–100]. That research is quite general and is an extension of previous mathematical results by embedding computational agent into the universe within which it utilizes resources for computation.

There is an area of human-centered limits that does not seem to be well researched and measured. Humans, as agents of finite capabilities, have strict limits with regard to explainability, comprehensibility [27], and all other aspects. One of the commonly mentioned impossibilities is unsurveyability [10] in the context of mathematical proofs.