Facticity as the amount of self-descriptive information in a data set

We will follow the standard textbook of Hopcroft, Motwani and Ullman for the basic definition of a Turing machine (TM)([19]). $\mathcal{T}$ will be the set of all possible descriptions according to this formalism.

An example: the self-delimiting code for ”11001110” is the concatenation of ”000”,”1”,”100” and”11001110”. Note that self-delimiting code in this sense is prefix-free: strings of different length get a different prefix. Throughout this paper we will assume a reference universal Turing machine $\overline{U}$ with prefix-free indexes has been chosen. If $\overline{U}(\overline{\imath}p)=x$, then $\overline{\imath}$ is the prefix-free index of a Turing machine $T_{i}$ to be emulated by $\overline{U}$ and $p$ is a input string for $T_{i}$. Note that the input string $p$ is not prefix-free, a feature that is essential for the results in this paper. In short: $\overline{U}(\overline{\imath}p)=T_{i}(p)=x$ All definitions in this paper refer to the preselected machine $\overline{U}$ which we will refer to as $U$. Without loss of generality we will suppose that there is a minimal Turing machine $T_{0}$ that simply is empty and does not compute anything. The index of $T_{0}$ is the empty string $\varepsilon$.

Proof: This follows from the fact that $U$ is universal. $\Box$

The length of the optimal code defines the classical Kolmogorov complexity:

We define:

This is in fact a one-part code optimization variant $K_{1}$ of Kolmogorov complexity that forces all complexity of the information to be stored in the index of the Turing machine. It is useful to distinguish a two-part code optimization variant:

This version balances the information over an index $i$ and a program $p$ for $x$. Here $\overline{\imath}$ is the self-delimiting code of an index and $U$ is a universal Turing machine that runs program $p$ after interpreting $\overline{\imath}$ and $\varepsilon$ is the empty string.The reason to use $\overline{\imath}$ lies in the fact that it allows us to separate the concatenation $\overline{\imath}p$ into its constituent parts, $i$ and $p$. Here $i$ can be seen as capturing the regular (structural [28], meaningful [1], model [18], effective [17]) part of the string $x$, where $p$ describes the irregular part.

The following lemmas show that two-part code is really more expressive than its one-part variants.

Proof: Suppose $K_{2}(x)>K(x)$. We have $K_{2}(x)=\min_{i,p}\{|\overline{\imath}|+|p|:U(\overline{\imath}p)=x\}$ , $K(x)=\min_{j,\varepsilon}\{|\overline{\jmath}|:U(\overline{\jmath}\varepsilon)=x\}$ i.e. $|\overline{\imath}|+|p|>|\overline{\jmath}|$, but then $K_{2}(x)=\min_{j,\varepsilon}\{|\overline{\jmath}|+0:U(\overline{\jmath}\varepsilon)=x\}=K(x)$. $\Box$

Proof: We have $K_{2}(x)=\min_{i,p}\{|\overline{\imath}|+|p|:U(\overline{\imath}p)=x\}$, $C(x)=\min_{q}\{|q|:U(q)=x\}$. Note that both $C$ and $K$ are defined with respect to the same prefix-free Turing machine $U$, which implies that $\overline{\imath}p=q$. $\Box$

The elegance of the introduction of an empty machine is illustrated by:

Proof: Suppose $x$ is random. In this case it cannot be compressed and the empty machine $T_{0}$ with index $\varepsilon$ is the best model of length $0$. We have $U(\overline{\varepsilon}x)=x$ and thus $K_{2}(x)=|x|+1$. $\Box$

There exists a so-called universal distribution $m$ with $m(x)=2^{-(K(x)+O(1))}$. This distribution dominates any recursive distribution by a multiplicative constant [23].

We will follow the standard textbook of Cover and Thomas for the basic definitions of Information theory ([11]).

The following lemma ([11], lemma 5.8.1) is important:

For a system of messages $S$ with $\{s_{i}:s_{1},s_{2},...,s_{n}\}$, the Shannon entropy is defined as

Here $p(s_{i})$ is the probability of message $s_{i}$. The following concept is useful:

Stochastic binary strings define the connection between Shannon information and Kolmogorov complexity:

Proof: We use a result from [11] (Theorem 5.4.2.). Since $S=\{0,1\}$ is a stationary stochastic source the expected code length per symbol is $H(S)$, which gives $kH(S)$ as optimal compression length. Since $x$ is stochastic we have $H(x)+|\overline{\imath}|=K(x)$, where $i$ is the index of a program that prints $x$ on the basis of an optimal code. Since $|\overline{\imath}|$ is constant in the limit we have $H(x)=K(x)$. $\Box$

An useful lemma is:

Proof: Consider the set of binary strings $X^{n}_{k}$ of length $n$ with $k$ zeros. These strings can be enumerated using an index of length $\log_{2}{n\choose k}$. A vanishing fraction of these indexes is itself compressible. A program of $p$ constant length transforms the indexes in to the original strings giving $K(x)\leq\log_{2}{n\choose k}+|p|$ for elements of $X^{n}_{k}$. The majority of strings in $X^{n}_{k}$ will be typical: $K(x)=\log_{2}{n\choose k}+|p|$. In the limit the contribution of $|p|$ vanishes. $\Box$

These limit results are rather rough and can be refined once we have formulated a definite measure for the model information in a string.

In this paragraph I investigate the relation between complexity $K_{2}(x)$ and factictiy $\varphi(x)$.

The distinction between stochastic and non stochastic strings is important. The following lemma describes the behavior of the stochastic strings.

Proof: According to lemma 5 there is always an optimal code for which 1) the two longest codewords have the same length 2) two of the longest words differ only in the last bit and correspond to the two least likely symbols. For a binary stochastic string the probability of the least likely blocks of length $k$ will be determined by their density of either ones or zeros. Thus, for a stochastic string that is long enough, the only relevant model parameters are an integer of length $\log|x|$ giving the total number of ones in the string and $k$ giving the size of the maximal block length. From these two parameters we can estimate the density. A program of constant length $c$ gives an optimal code. This approach collapses when sampling the strings with block-size $k$ when some random longest word blocks do not occur in $x$ and the optimal code has to be adapted. $H^{-1}(s)$ estimates the smallest probability of either zeros or ones in the string. $(H^{-1}(s))^{k}$ is the lowest probability for a block of length $k$. The probability that this block is not selected is $1-(H^{-1}(s))^{k}$. The probability that this block is not selected in $\frac{|x|}{k}$ trials is $\epsilon=(1-(H^{-1}(s))^{k})^{\frac{|x|}{k}}$. This is the probability that the string does not have an optimal model in the sense of lemma 5. $\Box$

Note that in the limit $K_{2}(x)$ approaches $|x|H(S)$. When $H(S)=1=s$ we have $H^{-1}(s)=1/2$, i.e. the probability that strings with high entropy have no small models goes to zero in the limit. When $H(S)=1/2=s$ we have $H^{-1}(s)\approx 0.1$. The probability that a stochastic string has a small model increases exponentially with length of the string, and for strings of fixed length polynomially with the complexity of the string. In the limit (long strings with small block-size) the cut-off will be sharp and act like a sort of percolation threshold. Above a certain complexity the size of optimal models for random strings will decrease rapidly. A defining family of curves is what one could call the:

The collapse probabilities, that are instantiations of theorem 2, specify the probability that a string of length $k2^{k}$ with a certain entropy between zero and one does not have a short optimal model when sampled with a block size $k$. The definition is derived from the so-called Coupon collectors problem that specifies the number of trials a collector of $n$ coupons has to make to collect all $n$ coupons as $n\log(n)$.

In general one could say that the factors $H(S)=s$ and $H^{-1}(s)$ specify a balance entropy and inverse entropy, that $2^{k}$ specifies the size of a space and $k$ a sampling granularity. The Collapse probabilities define the facticity curves directly as the following theorem shows:

Proof: A term of size $\log k$ specifies the string length $k2^{k}$ and the block size $k$. A term of length $k$ specifies the density of zeros. This allows us to estimate $H(S)$. From this we calculate an optimal coding for a string with $H(S)$. The term $\Phi(H(S))$ gives us the probability that some of the $2^{k}$ possible blocks do not occur in $x$, giving $2^{k}\Phi(H(S))$ as the size of the set of blocks that do occur in $x$. An optimal index for this set has complexity

A program of length $c$ computes an optimal model on the basis of these data. $\Box$

Proof: Direct consequence of theorem 3. Note that the maximal complexity of $\varphi(x)=2^{k}+k+\log k+c$ is reached for the value $\Phi(H(S))=1/2$. Take $k2^{k}=n$. $\Box$

Another important insight is that the density of a facticity plot is completely determined by the sampling probability distribution. If we use some simple high entropy process for the generation of examples all our strings will have small models and high entropy. The interesting plots are generated by computational procedures:

Proof: Direct consequence of Levin’s coding theorem: $m(x)=2^{-(K(x)+O(1))}$. The exponential decay of $m$ is up to a constant factor equal to the increase of density of binary strings with $K(x)$, so the exponential decay in probability is balanced by an exponential increase in density. $\Box$

Note that the universal distribution dominates any recursive distribution up to a multiplicative constant. Remember that the structure of a model is $K_{2}(x)=(\varphi(x)+2\log\varphi(x)+1)+\rho(x)$. Theorem 3 gives an optimal model size for $x$, but in general this size will not be reached because of the penalty of size $2\log\varphi(x)+1$. The search algorithm behind $K_{2}$ will settle for an optimal exchange between bits stored in the residual entropy and the facticity. Purely non-stochastic strings are found on the line $\varphi(x)=K_{2}(x)$. Their density diminishes exponentially with growing $K_{2}(x)$. Anthunes and Fortnow have proved the existence of so-called absolutely non-stochastic strings that encode the halting set for all smaller strings [4]. So, close to the upper bound $\varphi(x)=|x|/2$ there are still non-stochastic strings, although we will with very high probability never sample them. Note that we can transform every non-stochastic string to a mixed string by simply flipping some bits and adding a list of locations of the flipped bits. This representation is very inefficient. There will be a band of mixed optimal models close to the non-stochastic models.

We can give a taxonomy of strings under $K_{2}$ compression (See figure 1):

1. 1.

   Strings below the facticity threshold (c.f. theorem 3) are either:

   1. (a)

      Non-stochastic, i.e. the facticty is ’close’ to the Kolmogorov complexity $\varphi(x)\approx K_{2}(x)$

   2. (b)

      Purely stochastic, i.e. the density of the strings is a sufficient model (see theorem 2): $\varphi(x)=\log|x|+\log k+c$

   3. (c)

      Stochastic, i.e. they are under-sampled in the sense of theorem 3.

2. 2.

   Strings above the stochasticity threshold (c.f. theorem 3) are in principle non-stochastic. They are either:

   1. (a)

      absolutely non-stochastic, i.e. non-computable (c.f. [4]): $\varphi(x)\approx|x|/2$.

   2. (b)

      Computable, i.e. created by a non-deterministic process: $\varphi(x)\ll|x|/2$.

The following lemma is interesting:

Proof: $x$ is non-stochastic: $\varphi(x)=K_{2}(x)$, there are no data. Note that $|u|+2log|u|+1$ is the length of the self delimiting code for $U$, as soon as the model becomes longer than $\varphi(x)=2^{|u|+2log|u|+1}$ it is more efficient to prefix $U$ to the code as a general computational model and interpret the model as data.$\Box$

This proof is related to the proof of lemma 12. An immediate consequence of this is that the maximal number of meaningful models that is available for pure computational structures is limited by:

where $u$ is the length of the optimal index of the smallest universal Turing machine that exists and $u+\log u+\log\log u+\log\log\log u+....$ is the theoretical optimal length of its prefix-free index. Longer models will automatically be reinterpreted as data without structure. Only up to a certain limit there are non-stochastic objects, but there are mixed strings with longer model information.

For approximation of facticity we have theoretically the same limits as were proved in [2]: i.e. the facticity is in principle non-approximable in finite time. We may compress the data set, but there is no guarantee that the randomness deficiency of the model is also improved. In practice however, any learning or compression algorithm that allows us to estimate the balance between the data and the model code with reasonable accuracy can be used in real world data. Such algorithms include decision tree induction, nearest neighbor search, neural networks, grammar induction algorithms, standard compression algorithms etc. ([1], [3], [10]). The philosophical question why learning algorithms that in principle are faulty work reasonably well on real life data is still a matter of debate ([1], [18]).

Sawtooth plots are well-known in the literature and have been observed empirically in many different environments (e.g. [12], [13], [14], [22], Figure ). We can now tentatively interpret these plots as finding an optimal between two compression techniques: non-stochastic models and mixed models. Further research has to show whether these plots can indeed be explained by the theoretical framework developed in this paper. The plots have sharp cut-off points in the area where non-stochastic models collapse into mixed models and vice versa. The point that is in the literature referred to as the edge of chaos ([22]), seems to have no special meaning. It is a collection of ad hoc measurements in an area where the probability decays exponentially. Close to the point $\varphi(x)=|x|/2$ there will be so-called absolutely non-stochastic objects, that exist, but are never observed for strings of any size.

By nature deterministic processes can not generate new information. Information is associated with uncertainty, but in a deterministic process the future is known completely. On the other hand thermodynamic processes seem to increase information in an uncontrollable manner. The amount of information grows but the amount of facticity, or self-descriptive information, is minimal in the end. This suggests that, apart from growth of entropy there is a second useful way to classify processes in terms of the growth of facticity.

Let $x$ be some system that evolves over time and let $x_{t}$ be the binary description of $x$ system at time $t$: in terms of entropic and factic behavior over time we can distinguish the following five cases:

1. 1.

   Information discarding processes: $\frac{\Delta K(x_{t})}{\Delta t}<0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}<0$

2. 2.

   Self-organizing processes: $\frac{\Delta K(x_{t})}{\Delta t}<0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}\geq 0$

3. 3.

   Reversible processes:$\frac{\Delta K(x_{t})}{\Delta t}=0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}=0$

4. 4.

   Random processes: $\frac{\Delta K(x_{t})}{\Delta t}>0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}\leq 0$

5. 5.

   Factic processes: $\frac{\Delta K(x_{t})}{\Delta t}>0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}>0$

Let’s discuss these cases briefly:

1. 1.

   Information discarding processes both decrease entropy and meaningful information and thus violate the second law of thermodynamics and do not occur in closed systems. Standard computing is an example. Recursive functions are in general information discarding functions. When a computation is finished we are left with less entropy and meaningful information as before. Consider adding two numbers: $a+b=c$. Before the computation we have $\log a+\log b$ bits of information, after the computation only $\log(a+b)$.

2. 2.

   Self-organizing processes still decrease entropy and do not occur in closed systems. They reduce complexity and increase facticity. The growth of plants in a greenhouse, or bacteria on a petri dish are examples.

3. 3.

   Reversible computation in which we keep track of all information (including the meaningful information) is a borderline case. In principle such computations could be energy neutral in a closed system.

4. 4.

   Random processes like flipping a coin or diffusion of gases increase entropy but do not generate any new facticity. They are studied in the general theories of randomness and thermodynamics.

5. 5.

   Factic processes maintain a balance between model and ad-hoc information, i.e. their data sets can at any time both be interpreted as stochastic and non-stochastic. Processes that both increase entropy and meaningful information have, to my knowledge, not been studied well up till now. Still they deserve our attention: learning, game playing, the development of exchange rates, evolutionary processes and creative processes are factic.

It is clear that factic processes in closed finite domains (e.g. strings of length $n$) cannot increase model information indefinitely (c.f. lemma 12). The existence of absolutely non-stochastic uncomputable strings implies that the strings with the largest models will not be found in finite time. In the limit factic processes in finite domains will slowly die out and no increase of model size is possible, no matter how much computing time is spent. Factic processes defy some of our basic philosophical intuitions as the following theorem shows:

Proof: facticity is defined as

For factic processes we have $\frac{\Delta K(x_{t})}{\Delta t}>0$, $\frac{\Delta\varphi(x_{t})}{\Delta t}>0$. Suppose that at time $t$ an optimal description of $x_{t}$ is $T_{i,t}(p_{t})=x_{t}$ and that some time $t^{\prime}$ later we have $T_{i,t^{\prime}}(p_{t^{\prime}})=x_{t^{\prime}}$. Because of increasing entropy and facticity we have $K(i_{t^{\prime}})>K(i_{t})$ and $K(p_{t^{\prime}})>K(p_{t})$. For a process with a sufficient statistic the structural part $K(i)$ will stabilize after a certain period of time, pushing all the growth of information to the non-structural part $p$. With factic processes for which the domain can be expanded indefinitely this will never happen. $\Box$

This insight runs against central intuitions of scientific methodology. We always feel that no matter how complex a set of phenomena is, if we gather enough data, in the end we will be able to come up with a model that explains them. For factic processes this is per definition not the case. Any model that explains the data now, is sure to fail, at least partly, somewhere in the future. ²² 2 Note that when factic process reach the limit from theorem 15 they can have no purely non-stochastic models anymore. There will always be some residual entropy. In a certain way it is impossible to predict the development of a factic process, but the definition of facticity itself gives us two rules to guide us when predicting the evolution of a factic process:

• •

  Randomness aversion: A factic process that appears to be random will structure itself (the factor $H^{-1}(s)$.

• •

  Model aversion: A factic process is maximally unstable when it appears to have a regular model (the factor $H(S)=s$).

Note that this also implies that factic processes are disruptive: i.e. they are maximally unstable when they appear to have a model.