Infinite Intensions

Let us start with a simple, and seemingly unambiguous object, namely an arithmetic expression:

(1) 2 + ((2 ^ 3) * (3 + 7))

Like any other object (including a painting, a musical note, a video, a sentence in some natural language, a relation in a database), the expression in (1) has form (think structure, syntax) and content (think meaning, semantics). The form of the expression can be described in a tree-like structure as that shown in figure 1.

But what about the meaning (every form has content)? Well, it depends. The ‘content’ of the expression in (1) is infinite — yes, infinite! One part of the content is the (decimal) value, which in this case is 82. This part of the content is stored in an attribute (loosely speaking, a feature, a property) that we can call VALUE. But there’s a lot more content in that expression — or, there are many ways we can describe that expression other than by its value. For example, the expression in (1) can also be described by NUMBER_OF_OPERATORS, which in this case is 4, or by the NUMBER_OF_ODD_NUMBERS, which in this case is 3. How many attributes do we have in total — that is, in how many ways can we describe the expression in (1)? Again, we have an infinite number of attributes (descriptions)! Consider this:

DECIMAL_VALUE
NUMBER_OF_OPERATORS
NUMBER_OF_OPERANDS
NUMBER_OF_ODD_NUMBERS
NUMBER_OF_EVEN_NUMBERS
NUMBER_OF_PARANTHESES
NUMBER_OF_EVEN_NUMBERS_LESS_THAN_100
NUMBER_OF_ODD_NUMBERS_LESS_THAN_100
NUMBER_OF_EVEN_NUMBERS_LESS_THAN_200
NUMBER_OF_EVEN_NUMBERS_LESS_THAN_202
NUMBER_OF_EVEN_NUMBERS_LESS_THAN_203
NUMBER_OF_EVEN_NUMBERS_LESS_THAN_204
etc.

Sure, most of these attributes are usually not relevant to us (in most cases the VALUE is what matters), but nevertheless there are an infinite number of ways one can describe the ‘content’ of that ‘form’.

Now that (I hope) I convinced you that objects have infinite intensions — they can be potentially described in an infinite number of ways, even if they are as simple and as seemingly unambiguous as an arithmetic expression, let me now move into other types of objects, let’s say my car. While the landlord wants the MODEL, MAKE and LICENSE_PLATE (and perhaps the COLOR) of my car (when they give me a parking spot), an insurance company (that perhaps does not care about the color) wants attributes that the landlord does not care about such as the PRICE, the YEAR, the MILAGE, etc. But in general, there are an infinite number of descriptions I can give my car — that is, the “intension” of my car object is an infinite set (e.g., there’s the WEIGHT, the NUM_OF_ROADS_DRIVEN_ON, NUM_OF_ROADS_DRIVEN_ON_AT_9PM, the NUM_OF_ROADS_DRIVEN_ON_AT_9:05PM, etc…). Again, many of these descriptions are (perhaps!) not relevant to anyone, but they are descriptions (attributes) of my car.

The fact that objects (all objects) have an infinite intension (an infinite number of ways they can be described) is related to (what we call) ambiguity. Consider the picture in figure 2.

Like an arithmetic expression (and like a Spanish or a Japanese sentence) the picture in figure 2 has form (think structure/syntax) and content (think meaning/semantics). And like any other object it has an infinite intension — that is, it can be described in an infinite number of ways, although, admittedly, many of the descriptions (attributes) are not relevant to most of us. For example, while NUM_OF_PIGS might be a relevant attribute to some, NUM_OF_BASEBALL_GLOVES is not. The point I wish to make here is this: different people ‘see’ different attributes of the content and the same person might, at different points in time ‘see’ different attributes. We mistake this by ‘ambiguity’ — we say the picture is ambiguous because different people read it (interpret it) differently, or because they interpret it differently at different points in time. But the picture is innocent — it is not ambiguous. It does have the same content that has an infinite number of attributes that we decide to see a (different) subset of them at different points in time, depending on ‘our’ context — i.e., our ‘frame of mind’ at that point in time. The same applies to a musical note, or a painting, or a building, or a mountain. Any object has an infinite number of attributes and we decide what to ‘see’ in it. So while we refer to this as ambiguity, objects are innocent, and their content is not ambiguous. Because different people zone in on a different subset of the intension — that is, because different people see different things, we call this ambiguity.

The Language of Thought

Having explained that all objects have form (structure/syntax) and content (meaning/semantics), it means that there is a formal language that describes any object. When we think (conceive) of these objects we also think of their composition, and depending on the “type” of the object, their negation, their part-whole relationship, their size and their enlargements, their rotations, their orientation, their length, etc. These mental operations happen according to some rules, and they occur in some language — a language that Jerry Fodor called The Language of Thought. It is the language we think in. It is not English, nor Persian, nor Japanese nor Chinese nor Portuguese, it is Mentalese. And the Language of Thought must be a formal language since when we agree on a specific attribute, we fetch the same ‘data’.

The reason for all of the above is a recent article in nature magazine that draws the conclusion that we do not think in language. They base their conclusions on evidence of subjects with defects in their language network that still demonstrated thinking and reasoning on many tasks. The authors then conclude that language is a tool for communication more so than a biological faculty in the human species, a faculty that is central to thinking and reasoning.

The authors are correct if by language they mean the external languages we use to encode our thoughts, but the Language of Thought hypothesis is not about these external languages (E-languages) but about the internal language of thought, the I-language that we use to manipulate and compose our thoughts that we make of objects and concepts — objects that, as we argued above, have infinite intensions. The selection of a subset of an intension happens according to some rules. For example, while the number of ways an object can be described is potentially infinite, dreams do not have a HEIGHT, and tables do not make PLANS, and mountains do not RUN. The attribution, composition, and manipulation of objects of cognition (the ‘thinking’) must happen in some formal language, otherwise we could not convey thoughts to one another.

That the Language of Thought is a formal language is just a corollary of Immanuel Kant’s very first sentence of his Logic monograph:

Everything in nature, both in the lifeless and in the living world, takes place according to rules, although we are not always acquainted with these rules*.

If Kant and Fodor are correct, then we are still very far from anything we can call AGI, because mere approximation of data by some continuous function is certainly not going to take us too far. Our cognitive and conceptual apparatus and the language of thought that we use to manipulate and think about the world — the world of objects with infinite intensions, still eludes us.

__
* Incidentally, I am always amused by those that contrast neural networks with “rule-based systems” — as if the two have nothing to do with each other. Besides Kant’s statement that “everything happens according to rules” it is easy to prove that a neural network — that is nothing but a series of function compositions, is actually a series of composed rules since every function is, in the end, a rule. For example

length [ ] = 0
length(x : xs) = 1 + length xs

Can be re-written as

s.LENGTH = 0 if s ::= [ ]
s.LENGTH = 1 + xs.LENGTH if s ::= (x : xs)

See this article that proves neural networks are a collection of many rules and that all computation is defined, in the end, by rules.

There’s no Ambiguity, but Infinite Intensions: On Ambiguity, Language, and the Language of Thought was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

For several years now artificial intelligence (AI) has been witnessing an unprecedented amount of misguided hype, led by — unfortunately, big tech and worse yet, by some “prominent” AI researchers. Even reputable journals that we used to look for to keep abreast the latest advances in AI were swept by the hype. Some have published papers where the main claim of the paper is one that was scientifically refuted decades ago (e.g., “reward is enough” — which is basically equivalent to reviving the discredited “behaviorism” of B. F. Skinner, knowing that Skinner has been Skinned, as the late Dan Dennet once remarked!)

Scientifically false claims were also made in thousands of peer reviewed papers on other topics. Thousands of papers on “fine-tuning” were published in the past several years, yet now we know that fine-tuning is not only a futile attempt at making a ‘model’ more aware of the knowledge of some domain, but it might actually cause a drop in performance. This should not be surprising — if we were not swept by the hype, since slight tweaking of the weights of a massive neural model will only add to the unpredictability of the model. But we had to publish thousands of (now) irrelevant papers — papers with a long list of authors and nicely formatted tables that always showed (of course!) great numbers.

Numerous papers are still also appearing claiming some progress in explainability, yet it is a scientific fact that neural networks (NNs) are not explainable since computations in neural networks are not invertible and the outputs cannot be traced back to the original input components (constituents) — in NNs these components are lost once they are sub-symbolically distributed. It is that simple, yet numerous papers making false claims continue to be published.

Numerous papers are still also published claiming automatic programming by AI is near, although the theory of computation tells us that this is impossible to achieve.

There’s more. Thousands of papers have also been published to ‘measure’ the capabilities of large language models (LLMs) in reasoning — mostly in planning and problem solving. Some refelection on the nature of the underlying architecture of LLMs (namely, neural networks) should’ve made it obvious that LLMs cannot handle the kind of reasoning required in planning since the reasoning rquired in planning should involve the storage of and reference to symbolic variables, none of which exists in NNs (if you want to be amused, just ask your favorite LLM to show the plan of going from the initial state to the goal state of the image below).

This is not alchemy — to make a plan of going from the initial state to the goal state — even in this very simple example, requires the manipulation of symbolic variables of different types, something that is not available in purely sub-symbolic distributed neural networks (I thought Jerry Fodor and others proved this decades ago — so save some PDF trees!)

It is one thing for someone to say AGI is near, or that they feel they are witnessing “sentience” in some new AI model (you are free to feel anything you like) but scientific venues should not oblige “feelings” that might simply be the byproduct of a strange brew!

Perhaps all of this started when it became popular to offer an AI certificate with a few courses to anyone who paid the fees. You would think that reputable academic institutions knew that you cannot become an “expert” with a couple of courses in AI. They certainly knew that cognition and the human mind, subjects that for millennia have occupied the most penetrating minds in human history, cannot be mastered in a few months (Plato, Leibniz, Russell, Frege, Immanuel Kant, Quine, Noam Chomsky, etc. were not that mentally challenged to spend their lifetime studying the mind without claiming to reach conclusive results).

We have been warning that this gold rush has been getting out of hand for a long time. Worse even than an AI Winter, getting to a point where we cannot trust peer-reviewed research is very damaging in the long term.

Isn’t it time to bring ‘science’ back into ‘science and technology’?

Artificial Intelligence and Natural Voodoo was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

In an excellent new post Melanie Mitchell addresses an important issue related to large language models (LLMs) namely the nature of ‘intelligence’ (if any) in LLMs and do LLMs ‘reason’ at all. The question can best be answered by testing whether some output LLMs produce was the result of ‘recalling’ some memorized content — stitched together in a clever way, or is it the result of genuine reasoning and understanding?

In this context the counterfactual task is an ideal task to test LLMs — basically, instead of giving an LLM inputs that are very likely to have been seen in the ‘training’ data, expose the LLM to data that is unlikely to have been seen during training (the ‘counterfactual’ data). This technique has been tried by others who reported that disrupting the training ‘template’ the LLMs showed a drop in performance to near random choice.

Inspired by these results — which did not surprise me at all — I repeated some tests I did some time ago but now on the new and improved GPT 4o (the one that is getting closer to AGI, that is — and yes, I am being extremely sarcastic). The experiment I conducted is similar to the idea of ‘counterfactuals’ in the sense that I create a possible world where some reality is represented with a different coding scheme, but where the semantics — i.e., the reality itself is still the same. For example, suppose we scramble the alphabet of the English language in some way, where, let’s say ‘a’ is now ‘b’ and ‘b’ is now ‘c’, etc. The meaning of ‘dbu’ should still refer to the furry little kittens we all know, since all that changed is ‘cat’ has changed to ‘dbu’ but CATS themselves did not change at all. Of course, since LLMs know nothing about reality — nor do they know anything about cats-for that matter, their superficial memorization (through millions and millions of weights) will be immediately exposed by this ‘counterfactual’ trick.

I must say that I knew the world of the LLM will be turned upside down with this test before I even tried it, because I have no doubt (and never did) that LLMs — and all deep neural networks — are nothing more than massive fuzzy hashtables! And I was not disappointed. If anything was surprising it was how bizarre the performance was. In fact, in some trials the LLM would fetch some text that ‘matched’ one or two phrases — although it was text that is not related at all. In some cases the LLM would fetch some text it stored in the discourse three or four queries ago. Complete random rambling — poor cosine similarity function, it had no clue what tensors to work with.

Here’s my query/prompt, in case you want to have fun:

Try different queries and different coding schemes, as long as you somehow create a “counterfactual” — i.e., as long as you disrupt what the LLM might have memorized without changing the meaning (the reality) in any way. And… have fun with a massive unintelligent machine that cost billions just to do “approximate retrieval” (and that’s why I like to use the term ‘massive fuzzy hashtables’).

When will we climb down the tree and acknowledge what we knew more than four decades ago, that a purely behavioristic, associative, statistical paradigm will not explain cognition? Even if it was useful in picking up some low hanging fruits (by finding some patterns in the data), this paradigm will not (scientifically) tell us anything about language nor reasoning, understanding and the mind.

LLMs don’t even do ‘approximate retrieval’ — embarrassingly, they try to recall some ‘similars’ was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Argument

The Chinese Room Argument (CRA) is a famous thought experiment conducted by the prominent philosopher John Searle that can be described as follows:

The argument and thought-experiment now generally known as the Chinese Room Argument was first published in a 1980 article by American philosopher John Searle (1932– ). It has become one of the best-known arguments in recent philosophy. Searle imagines himself alone in a room following a computer program for responding to Chinese characters slipped under the door. Searle understands nothing of Chinese, and yet, by following the program for manipulating symbols and numerals just as a computer does, he sends appropriate strings of Chinese characters back out under the door, and this leads those outside to mistakenly suppose there is a Chinese speaker in the room. The narrow conclusion of the argument is that programming a digital computer may make it appear to understand language but could not produce real understanding. Searle argues that the thought experiment underscores the fact that computers merely use syntactic rules to manipulate symbol strings, but have no understanding of meaning or semantics.

(from Stanford Encyclopedia of Philosophy — here)

The gist of John Searle’s argument is that a computer that is only performing symbol manipualtion can only do syntax, but not semantics (and thus there can be no understanding).

The Source of the Counter Argument

Chomsky famously once gave the following sentences to underline the fact that probabilistic analysis of language does not explain language acquisition:

(1) Colorless green ideas sleep furiously.
(2) Furiously sleep ideas green colorless.

Chomsky’s argument went (roughly) like this: (1) and (2) are equally nonsensical and are also equally unlikely to occur (at least at the time of his publication, both sentences were probably never seen in any corpus) and so a purely statistical analysis would/should assign both sentences a probability near 0 yet there is something clearly different about (1) and (2): speakers of ordinary language would consider (1) to be perfectly grammatical while this is not the case in (2). Thus, according to Chomsky, the ‘probablity’ of a sentence is a meaningless notion.

While Chomsky’s concern was language acquisition (and the poverty of the stimulus argument), there is another important side to this argument, namely this: while one can make a perfectly grammatical sentence that is (semantically) nonsensical, one cannot make a sensible (meaningful) sentence that is not grammatically valid (so “2 * (3 + 5)” could be meaningful — could be assigned a meaning/value, because it is, in the first place, grammatical; while “2 * 0 14 + / 51 0 62 * * +” could not be assigned a meaning because it is not even grammatical. In fact, that’s also why two expressions — e.g., “2 * (3 + 5)” and “7 + 9”, can have the same meaning/value although they have different syntactic strcutures)(see note [1], however).

The implication between syntax and semantics is thus one directional from semantics to syntax — that is,

IF (semantically valid) THEN (syntactically correct)

In Venn diagrams this corresponds to a set of syntactically valid sentences that is much bigger than the smaller subset of semantically valid or meaningful/sensible sentences (see figure 1). To state this in a slogan, we could say “no semantics without syntax” (incidentally, this is why one can argue that modern large language models (LLMs) have “learned” syntax from all the linguistic patterns they ingested, since they do generate not only syntactically fluent language, but also coherent and sensible language — i.e., since LLMs produce sensible language, they must have mastered syntax).

In any case what we will use in refuting John Searle’s argument is this fact: mere syntactic manipulation can lead to the generation of grammatically valid sentences, but it cannot ensure that the gramatically valid sentences are meaningful (or sensible). Thus, selecting a meaningful/sensible sentence implies doing more than syntax — whatever that ‘more’ is! Or, moving from the wider set of grammatically valid sentences and entering the smaller subset of meaningful sentences means doing more than ‘just’ syntax.

The Proof (the refutation of Searle’s CRA)

There are two options in considering the argument John Searle makes: (1) John Searle admits he can/might send back inappropriate Chinese responses because he does not understand Chinese; or (2) Searle insists he can send appropriate Chinese answers by just syntactic symbol manipulation and without any understanding. In the former case John Searle’s argument will be vacuous (or moot) because then there’s no claim of ‘understanding’ for him to deny. It is clearly the latter case that Searle is defending — namely that he can send appropriate Chinese answers to Chinese questions, without understanding a word of Chinese. This situation, however, cannot happen, as we argue below.

Here’s the proof:

1. The set of all grammatically (syntactically) correct responses includes — like it does in any language — semantically non-sensical sentences (aka Chomsky’s ‘colorless green ideas’ sentence).
2. Searle is sending only ‘appropriate’ (sensible) responses out of the Chinese Room.
3. By (1) and (2), Searle is selecting not only grammatically (syntactically) correct responses, but is selecting from a smaller subset of all those grammatically (syntactically) correct responses only ‘appropriate’ (sensible/meaningful) responses.
4. By (3), Searle (or the computer program manipulating symbols) understands — can identify — what responses are appropriate (meaningful) and is thus doing not only syntax, but is also doing semantics.

QED.

I am sure I am going to have many counter-counter arguments, but this was fun, nevertheless :)

Notes:
[1]in formal languages where the notion of ‘sensible’ or ‘nonsensical’ does not factor in, like the language of arithmetic in this case, the set of syntactically valid (grammatical) sentences and the set of semantically valid (meaningful) sentences are usually the same and so we have an if-and-only-if, but that’s another issue.

A Refutation of John Searle’s Famous Chinese Room Argument? was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

Testing GPT-4 for reasoning and language understanding was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

In an article in The Gradient (and briefly in a previous post) I described what I called MTP — the Missing Text Phenomenon, which is the information that is never explicitly stated in everyday discourse, but is information that the speaker can safely assume to be available to the listener. We appropriately call this missing, implicitly assumed, and mutually agreed upon information, common knowledge. In the previous articles I was presenting a proof that BERT-based large language models (e.g., GPT-3) will never scale intro true language understanding (although they might be useful in picking up some low hanging fruits in tasks such as summarization, search, etc.). The proof was intuitively simple:

The equivalence between machine learning (ML) and compression has been mathematically established (see this, and this).

Natural language understanding (NLU), on the other hand, involves decompression — specifically, it involves ‘uncovering’ all the missing and implicitly assumed information.

The above are inconsistent (contradictory) and thus: ML is not relevant to NLU.

The above argument, while appreciated by many, was not accepted by several that did not appreciate the nature of the argument. In this article I hope to support this argument by briefly covering extensive research that has been done in psycholinguistics and neurolinguistics. In particular, research that uses information-theoretic concepts and is based on data from evolutionary linguistics. This research can be summarized as follows:

In linguistic communication both the speaker and the listener would like to minimize their efforts: the speaker would like to minimize the effort of encoding a thought into a linguistic message and the listener would like to minimize the effort of decoding the message. These two requirements work against each other but since speaker and listener change roles in communication, an amazing optimal point was reached at in the evolution of language in such a way that the speaker says the minimum possible, but not less than is necessary for the listener to successfully retrieve the content of the message. This process happens by what Zipf called the least effort principal (for more details see these very important papers, and references therein: 1, 2, 3, and 4).

The above optimization process that minimizes the efforts of both speaker and listener necessarily results in introducing ambiguity, by semantically compressing the encoded message as much as possible. As noted here, when there is enough common knowledge that is shared between speaker and listener, semantic entropy can be reduced while compressing the encoded message without loosing information (“with the help of shared background knowledge, we will be able to communicate with shorter messages to achieve the maximal informativeness of the source.” p. 6)

Let us consider a simple example. Consider the sentence in (1).

(1) The laptop did not fit in the briefcase because it is too small.

The reference ‘it’ has two possible meanings here — it could be a reference to the laptop or to the briefcase. Let us assume there is no shared background knowledge and that all the information required to understand the message is in the text. In this case the probability that ‘it’ refers to either the laptop or the suitcase is equally likely — since there are two possibilities than the probability that ‘it’ referers to either one is 0.5. Using Shannon’s entropy (uncertainty) measure, we get the following, where H[M] is the uncertainty in the message given all possible meanings m:

Thus, in the absence of any background knowledge, uncertainty is at a maximum and ‘it’ is absolutely ambiguous. The data in the text itself does not contain information to make one reference more probable than the other. However, let us assume that speaker and listener have some mutual information (called common knowledge, symbolized by C). The set C could, for example, contain the following:

if LARGER-THAN(x, y) then NOT-FIT-IN(x, y)
LARGER(x, y) iff SMALLER(y, x)

We can now compute the uncertainty (entropy, ambiguity) with the assumption that speaker and listener have access to mutualy shared common knowledge using the following:

Since the probability conditioned on background knowledge of one reference is now near 1.0 (say, 0.999), the right-hand side reduces to 0:

Thus, by assuming some mutually shared common knowledge, we were able to compress our message without introducing any uncertainty (entropy is 0). This, in fact, is how we communicate:

For effective communication, and inline with the least effort principal, we compress our utterances, introducing ambiguity, although it is ambiguity that we are certain can be resolved given common knowledge that speaker and listener mutually share.

A recap: in seeking efficiency in communication, in ordinary discourse we do not express any of the mutually agreed on (shared) common knowledge, such as “object x will not fit in container y if x is physically larger than y”. Our linguistic utterances are thus highly compressed and the amount of information missing is exactly the amount of common knowledge (see proof here).

While this evolution of human communication works well for humans, it is very challenging for machines since machines do not have access to that ‘shared’ and mutually agreed upon common knowledge. In fact, most challenges in NLU are due to this fact. The issue in (1), for example, is known in NLU and computational linguistics by ‘reference resolution’. In general, reference resolution can in fact be much more difficult than the situation in (1). For example, consider the following sentences:

(2) Jon wants to be a guitarist because he thinks it’s a beautiful instrument.
(3) Olga is an old dancer. She has been doing that for a long time.

In (2) and (3) there are references to objects that are not even mentioned explicitly in the text — ‘it’ is a reference for a guitar in (1) while ‘that’ is a reference to Olga’s dancing in (2). Besides reference resolution the missing text phenomenon (MTP) is the cause of many other challenges in language understanding. For example, hearing a waiter in a bar saying (4) we all know that the uncompressed text is that in (5)

(4) The loud omelet wants another beer.
(5) The loud person eating the omelet wants another beer.

How do we ‘uncover’ that missing text for successful ‘understanding’ of ordinary spoken language is a massive project and this is not the place to discuss this (although some of this project is briefly described here).

What matters for now is to appreciate that all linguistic corpora of ordinary spoken language does not contain the implicitly assumed common knowledge, and there can be no language understanding without access to that knowledge. Large language models, therefore, and no matter how much text they ingest, are looking for something that is not even there — that is what the data on language evolution tells us.

Let’s reduce carbon emission and stop chasing infinity by trying to ingest more and more text — it is time to climb down the tree.

_
https://medium.com/ontologik

Large Language Models and what Information Theory tells us about the Evolution of Langauge was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

I have written a few posts where I make the point that most of the important knowledge that is needed to build intelligent agents is not learned —because it cannot be learned differently, and it cannot be susceptible for incremental, approximate and individual learning from observations. It is that simple. I have written about this topic first in “Learning is Overrated: Machine Learning vs. Knowledge Acquisition” where I discuss the difference between “knowing how” and “knowing that”. Recently, I wrote a post where I explain “Why Commonsense Knowledge is not (and can not be) Learned”. In comments and (mostly private) messages I keep getting remarks like “but why can’t that be learned?” It seems that the ‘folk’ meaning of learning has taken over even the most rational of people that the techncial point I’m trying to get across is still not appreciated.

The Folk Meaning of Learning

It is quite understandable for the common folk to lump all kinds of learning under the English word “learn.ing” but in techncial domains we have to be more accurate in our language. We learn to play guitar, to ride a bike, to swim, to fix a TV set, etc. We also learned that addition is commutative (that is, u + v = v + u), that the intersection of two sets cannot have more elements than the elements in either set, that water boils at 100 degrees celcius, that the circumeference of a circle is 2*PI*r, etc. But the first group of things we learn are quite different from the second group — we learn the first group of things by experience, while the second set of things is learned by “being told” or by “deduction”. Furthermore, there are a number of what I called previsouly universally valid cognitive templates that we don’t even learn at all — these are innate cognitive templates that have been encoded in our DNA and they are simply tenplates that abide by “the laws of nature” (e.g., that a phsyical object cannot be in more than place at the same point in time). But let us stick to “knowing how” vs. “knowing that”. Below I will reuse a diagram I used before.

Let us take an example from each branch. Let’s take “knowing how to play guitar” as an example of “knowing how” and “knowing that the circumference of a circle is 2*PI*r” as an example of “knowing that”. Now note that:

Knowing How to play guitar is individual
My cousin plays guitar and so does Carlos Santana, but — unfortunately for my cousin, he and Carlos know how to play guitar differently!
Knowing That the circumference of a circle is 2*PI*r is universal
My knowledge that the circumference of a circle is 2*PI*r cannot be different from your knowledge of the same fact, either we know this fact or we don’t, but once we know it, we all know the same fact.

Knowing How to play guitar is continuous/fuzzy
My cousin knew how to play guitar gradually (incrementally) and also there’s no ceiling to how much that skill can be developed.
Knowing That the circumference of a circle is 2*PI*r is absolute
My knowledge of the fact that the circumference of a circle is 2*PI*r did not happen gradually —I did not partially know some part of that fact or learned that it is 0.85% true and then I arrived at knowing that it is an absolute truth. I learned that fact once, and that it is a true fact with 100% certainty.

Knowing How to play guitar is inconsequential
The entire universe does not care if I know how to play guitar or not.
Knowing That the circumference of a circle is 2*PI*r is consequential
The universe will not be the same if the circumference of a circle is not 2*PI*r.

Knowing How to play guitar is not forgettable
Once we learn some skill, it cannot be forgotten (unless we have a memory loss — like, as it was rumored, once happened to Jerry Garcia, the guitarist of the Grateful Dead, who temporarily forgot how to play guitars but got the skill back when he recovered and gained access to his memory!)
Knowing That the circumference of a circle is 2*PI*r is forgettable
We do forget some facts that we need to either look-up again or are reminded of somehow.

The differences I went through above should clearly make it now clear that “knowing how” is quite different from “knowing that” — this is actually an old subject that has been studied by logicians and philosophers for a long time (see this for some background reading and follow the references there for more in depth reading.)

Machine Learning and Knowing-How

Machine learning (ML) has swept everyone by a storm to the point where it has become synonymous with artificial intelligence (AI). (actually, I always find it amusing when I see someone declaring his skills by writing “AI/ML”, I guess using the backslash to mean “and/or” as if AI = ML). Regardless, ML has been equated with AI because for everyone it seems so obvious: a machine that can learn will, eventually, be an intelligent machine. Well, not quite, and especially if by learning we mean learning from data (i.e., learning from experience/observation), because many things we learn (we “come to know”) not by experience/observation, but (i) by being told (instruction); (ii) by discovery; (iii) by deduction; or (iv) by analogy. In other words, we do need “knowledge acquisition” but statistical learning and finding patterns in data is a tiny part of our cognitive apparatus. Moreover, and more importantly, most of the things we learn from observation/experience is not knowledge that is consequential to our functioning as intelligent agents. Furthermore, and as I wrote in the previous post, commonsense knowledge (e.g., if x is bigger than y and y is bigger than z, then x is bigger than z) is not (and I repeat again, cannot) even be learned at all. Those universally valid facts (that obey the logic of the universe) cannot be learned individually from experience/observations because we are not allowed to learn them differently.

Final Word

I hope I was able to defuzzify what the word “learning” means. This is important in AI and has very serious consequences — least of which is to be humble in making claims that the dominant AI paradigm we call “deep learning” (DL) will lead to artificial general intelligence (AGI). Quite the opposite, and even if DL becomes very accurate and can get serious problems like adversarial examples, transferability, and other issues resolved, it will end up being just a tiny fraction of what is needed to build intelligent machines since most of what we come to know (or we come equipped with) is knowledge that is not learned from data (observation/experience) through an incremental process of minimizing error. That is just one simple mechanism.

Finally, I hope I have explained the technical reason why quite a bit of knowledge is not (and cannot be) incrementally learned from observations, and I hope I will not anymore get messages like “but why this/that cannot be learned?” — although I suspect I will still get such messages 😊

Keep AI’ing

___
https://medium.com/ontologik

Why is “Learning” so Misunderstood? was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

Commonsense (background) knowledge, at least the kind of knowledge that we fetch and rely upon in the process of language understanding: (i) cannot be learned by processing vast amounts of text because that knowledge is never explicitly stated in the text — and you cannot find what’s not there; and (ii) that background knowledge cannot be learned perceptually from observation since the vast amount of the crucial background knowledge is universal, and is not probablistic nor approximate, and so it cannot be susceptible to an incremental process of individual observations. The shared background knowledge needed in the process of language understanding is the kind of knowledge that obeys and respects the laws of nature and as such it has to be codified. In fact, that knowledge must be codified in a symbolic system that quantifies over variables of specific ontological types.

Commonsense Knowledge is not Learned 1 — It’s not in the Text

There’s a consensus among researchers investigating the neurological, psychological and evolutionary aspects of human linguistic communication that languages have evolved according to the information-theoretic principle of least effort. Specifically, it has been established that interacting communicative agents tend to produce utterances that minimize the complexity of coding a thought (by the speaker) as well as minimize the process of decoding linguistic utterances by the listener back to the intended thought [1] and in the process finding an optimal point where the effort of both speaker and listener is minimal.

In the process of minimizing both the speaker and listener efforts an optimization process has evolved that resulted in some form of language compression by introducing ambiguity to the language where the speaker does not have to spell out everything in detail but leaves out information that that the speaker can safely assume to be available for the listener. In this framework, efficient linguistic communication happens when the shared background knowledge between the speaker and listener is large [2]. What this means is that ‘shared background knowledge’ is hardly — if ever — stated in linguistic communication, if for nothing else but for the reason that, since it is available for (and shared by) speaker and listener, transmitting any of that information is a wasted effort. But while ambiguity, pervasive in natural language, has been shown to be necessary for efficient compression, it has proven to be difficult to handle by machines, and precisely because machines do not know what that ‘shared background knowledge’ is. Since that shared background knowledge is not in the text it is perplexing to find some highly visible research efforts suggesting that they are trying to extract this background (commonsense) knowledge from text (see for example, [3]).

To show why this type of background (commonsense) knowledge is not the text, consider the sentences in (1) and (2):

(1) Jon has a Greek statue in every room in his house
(2) Jon broke his leg in a car accident his family had near Toronto.

A 4-year old knows that ‘a Greek statue’ in (1) refers not to one, but to many statues since one cannot have a Greek statue (a physical artifact) in more than one location — in NLU lingo, the challenge in (1) is usally in resolving quantifier scope ambiguities. Similarly, a 4-year old knows where Jon broke his leg upon hearing (2), since the location of a sub-event is always the location of the main/larger event. This can be expressed as shown in figure 1 below — the challenge in (2) is in knowing the commonsense logic of events, a logic that will never be explicitly stated in the text.

To sum-up: trying to capture this type of shared background knowledge by ingesting large volumes of text is like looking for something that is not even there since — for effective communication — this type of background knowledge is never explicitly stated in linguistic communication. Incidentally, below is the result of testing sentence (1) on so-called largae language models that try to understand language by trying to find some meaningful statistical correlations in vast amounts of text.

Commonsense Knowledge is not Learned 2 — It’s not Perceptable

There are others who suggest that most of our knowledge can be ‘learned’ by perceptually observing the world and gradually building a model of it. The assumption is that what we call shared ‘background knowledge’ is also statistically and perceptually learned by observation (this is the opnion suggested in [4], for example). Unfortunately, this is also a faulty thesis since the vast amount of ‘background knowledge’ that is implicitly assumed in tasks that require high-level reasoning (e.g., planning and language understanding) cannot be approximate and cannot be learned differently by different individuals and thus it cannot be susceptible to individual observations.

In linguistic communication, and for efficient compression to minimize the effort exerted by speaker and listener, the amount of information that is left out and not included in messages encoded by the speaker must be the same background knowledge that the speaker can assume the listener has in order to decode and in the process successfully disambiguate the message. This is done by relying on quite a bit of shared background knowledge, and thus the entire process would fail if the shared background knowledge was not the same.

Since that background knowledge (such as the ones shown in figure 1 above) are not ‘approximate’ and cannot be different, they cannot thus be individually and perceptually learned — learnability theory itself precludes learning these universally valid cognitive templates — unless, of course, cognitive agents have an infinite amount of time.

Final Word

An important question now (assuming the reader has accepted the above argument) is this: what is the nature of this ‘shared background knowledge’, how vast is it, and how could it be codified. I shall return to this question in future posts.

References

1. Fortuny, J., Corominas-Murtra, B. (2013), On the origin of ambiguity in efficient communication, Journal of Logic, Language and Information volume 22, pages 249–267. (available on the arxiv here)
2. Bao, J., et. al. (2011), Towards a theory of semantic communication, 2011 In IEEE Network Science Workshop.
3. One Man’s Dream of Fusing A.I. With Common Sense, New York Times, August 28, 2022 (here)
4. Browning, Jacob and LeCun, Yann (2022), AI And The Limits Of Language, NOEMA, August 23, 2022.

__
https://medium.com/ontologik

Why Commonsense Knowledge is not (and cannot be) Learned was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

A while back a 2-day online workshop on compositionality and AI was organized by Gary Marcus and Raphael Milliere, with additional contributions form Allyson Ettinger and Paul Smolensky (one of the pioneers of tensor algebra in NNs). As often, and although compositionality touches on many subjects in AI, natural language was the focus of the discussion, and rightly so, as compositionality is best understood in the context of (natural) language — although it equally applies to formal (e.g., programming) languages, among other formalisms.

The Inverse of the Compositionality Function

As illuminating as the discussion was I found it to be lacking in a very critical aspect, namely in appreciating what compositionality entails, not only in semantic theories, but also the crucial aspect of compositionality as it relates to cognitive architecture, to learnability, and to computability and thus to the evolution of human thought. Typically, and this was also reiterated in the workshop, compositionality is often reduced to the well-known cliche (that has been rephrased in so many ways ever since Frege):

The Compositionality Principal: the meaning of a complex expression is a function of the meanings of the constituents and the way they are combined.

While true, this on it’s own is (almost) a vacuous statement, as Wlodek Zadrozny [1] formally argued over two decades ago. Indeed, almost any semantic formalism is compositional in that sense, including transformers in deep neural networks, since at every node the output is generated as ‘some’ function of all the constituent inputs. The point here is that the importance of compositionality — which is a function — is in its inverse! A formalism can capture the important computational and cognitive aspects of compositionality not if it can compose an output out of the values of constituents, but if the decomposition can be maintained and recovered!

Let us take a simple string like SANTANIA. This string could be obtained using a string append function as follows:

append(‘SAN’, ‘TANIA’)
append(‘S’, append(‘ANTA’, ‘NIA’))
append(‘’, ‘SANTANIA’)
etc.

Similarly, 24 could be the result of many (infinite!) possible arithematic operations:

12 * (8 / 4)
20 + 4
16 + (2 * 4)
etc.

The point here is this: compositionality is important as a high-level semantic procedure to obtain the meaning (value) of some composite as a function of the meanings (values) of the constituents, but more importantly, compositionality is not of that great value if our formalism does not support maintaining and recovering the constituents, because otherwise the decomposition is undecidable (there are infinite number of ways to decompose a value). Now connectionist models — in all its variations, including modern day deep neural networks (DNNs), do not preserve the composition since once tensors are composed, their decomposition is undecidable. In short, and as Fodor & Pylyshyn [2] showed over three decades ago, NNs do not maintain the syntactic structure, and as such, they cannot model compositionality.

Why is that important? I will first discuss why that is important from the standpoint of language and language learnability. Finally, I will discuss why compositionality (whose inverse cannot be modelled by NNs) is the curse of connectionism as a serious model for truly intelligent systems.

Why is Compositionality Important?

Consider the template of a simple sentence below:

The above is a template for (potentially) an infinite number of sentences — some of which can be quite complex. Here are some examples:

Note that the “entity” a person may love is an object of a very generic type; that is, what a person may love could be almost anything — it could be another person, an activity, a status, a property, etc. Note also that the person (and the entity) can be made up of a very complex syntactic expression that — in the end, will “converge” to an object of that type. How does that happen? How does it happen that a child knows that the simple reference in (1) and the complex reference in (2) both refer to objects of the same type, namely to person and thus they can occupy the same slot?

(1) John
(2) the boy next door that always wears an AC/DC t-shirt

The question goes even beyong “how does a child know that (1) and (2) both refer to objects of the same type?” but also how does a child workout this algebraic composition, in real-time, so that they can generate from (2) and object of type person? Incidentally, that was the genius of Richard Montague who developed an algebra that formally shows how such infinite and strongly-typed compositions can happen — but that subject would have to be left to another time and place.

Research has shown that without compositionality language learnability could not be explained since computationally there is no explanation as to how we workout the filling of constituents in the right slot in real-time, without having these compositional structures that are decomposable. They must be decomposable since we need to verify that a certain value can sensibly occupy a certain slot in the composite. All this happens in real-time (when hearing an utterance, or when converting a thought to an utterance), and that is how a single template structure becomes productively the parent of an infinite number of thoughts. This was summarised by Peter Pagin [3] as follows:

Compositional semantics allows greater expressive power, simply because a systematic but non-compositional semantics of a language with the same expressive power would be intractable.

What the above implies is that language learnability and language acquisition could not happen without compositionality — there just is no other “technical” explanation of how we understand and convert thoughts to linguistic utterances in real-time without compositionality!

Having said that, the fact that we make up the meaning of the whole compositionally — i.e., as a function of the meanings of the constituents, is the simple and trivial part, it is the fact that we have access to the constituents — that is, that our composition is reversible (or that our formalism allows us to we maintain and thus recover the decomposition) is even more crucial. Connectionist architectures (including modern day NNs) cannot maintain the decomposition since tensor composition is not decidedly reversible. It can only preserve the composition if it admitted symbolic structures and variables that can be instantiated with values of different types. But if they admit that, then they become nothing but an implementation machine of symbolic systems. Either way, true compositionality is their curse. And without true compositionality we cannot speak of any thinking and reasoning.

Final Word

Talk of compositionality is music to my ears since some of the most penetrating minds have spent centuries studying this phenomenon — not only in relation to language, semantics and understanding but also to reasoning in general and, more importantly, as the only mechanism by which language learnability can happen in an effectively computable manner. Without compositionality (and a formalism that maintains and allows the recovery of the decomposition) we would have to make the ridiculous stipulation that we require infinite memory and an infinite amount of time, just to explain how we make (or understand) a single sentence in real time.

As a final remark, those that are mistakenly excited about so-called Large Language Models (LLMs) should ask this question: before we tackle natural language with this approach, why not show that this paradigm can handle simpler languages, say programming languages. There are millions of valid Python examples out there, so can LLMs ever learn to accept valid Python programs? The answer is no, and that is exactly due to “infinity” that compositionality can account for but NNs that do not admit symbolic structures to represent infinite objects, simply cannot.

References

1. Zadrozny, W. (1998), Is Compositionality Formally Vacuous?, Linguistics and Philosophy, vol. 21, pp. 629–633
2. Fodor, J. A., & Pylyshyn, Z. W. (1988). Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1–2), pp. 3–71.
3. Pagin, P. (2011), Compositionality, Complexity, and Evolution, PERILUS 2011, Symposium on Language Acquisition and Language Evolution The Royal Swedish Academy of Sciences and Stockholm University.

Compositionality: the curse of connectionist AI was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In the previous post I argued that most of the knowledge that an intelligent agent needs to perform most higher-level cognitive tasks (language understanding, planning, reasoning, etc.) is not learned, but is acquired. Unlike skills (say playing guitars), that are individually learned, and learned differently due to our unique individual observations and experiences, universally valid knowledge is not learned incrementally, but is acquired, and it is not ‘approximately’ acquired nor acquired with some probability. Universally Valid Cognitive Templates (UVCTs) that are agnostic to our individual experiences (and are also agnostic to culture, time, place, etc.) are acquired in one shot — either we come to know of them or we don’t. In short, that knowledge — that a 4-year old uses to function in the world, to reason and to understand language, is knowledge that we have no choice but following — you can think of this knowledge as knowledge that emanates from the universal laws — the “laws of nature”.

In the previous post (and briefly in “AI Cannot Ignore Symbolic Logic, and Here’s Why”), I mentioned a few of these Universally valid Cognitive Templates (UVCTs) such as the logic of events (e.g., a sub-event e2 of a parent event e1 must have the same time and location of e1), the logic of the “contained-in” relationship, the universality of transitivity, the logic of “parthood”, etc. Here I will discuss one Universally Valid Cognitive Template that is usually a problematic logical operation — namely, entailment (or, implication) — that we all questioned when we first learned it. Many “learned” people think we invented this rule/template. But far from it, it is a template that simply respects the laws of nature. I will show here that in fact the logic of this operation is consistent with physical reality and thus it was not invented but we simply discovered it! In the next post I will also discuss in some details another culture- and language-agnostic phenomenon that is universal and that is never learned, but it simply obeys metaphysics — or, the logic of ontological categories of the world we live in. If some data-driven, machine learning extremists still want to ignore mathematical proofs, then I suggest they get into politics, advertising/marketing, or comedy — dancing could also be a career option, since they are good at hand waving. In any case, they definitely should not be involved in science nor even in engineering.

Entailment (Implication)

In figure 1a below is a truth table that explains the semantics (functional behavior) of the entailment logical operation (if P then Q, or P implies Q, or P entails Q).

The truth table (that extensionally lists all possible data variations) represents the the “intension” that sums up this logic: P implies Q is false (F) only when P is true (T) and Q is F. In a sense, the semantics of the “implies” operation says that from truth you cannot infer falsehood, but anything else goes. It does sound strange, and it did baffle me when I had my first exposure to prepositional logic. How could a statement like

(1) if there are pink flying pigs, then the moon is made of cheese.

be true. Well, without getting into the technical side of things, but yes (1) is true because it can never be disproven (it can never be proven false), and the reason it can never be proven false is because you will never see pink flying pigs to see if (conditional) conclusion holds—and, since you will never be able to disprove (1), (1) is true. (Incidentally, I use this trick on people who tell me since I cannot disprove the existence of the Abrahamic God, then He must exist —I usually reply “then there are “unicorn pink flying elephants”, unless you can prove they do not exist!)

Humor aside, what I want to prove now is that the truth table of implication operation shown above in figure 1a was not invented by logicians, but was simply discovered (they stumbled upon it, like all of mathematical truths that simply explain the laws of nature). In fact, the truth table of implication corresponds not just to abstract reality but even to physical reality.

Why the Meek (us, meager Humans) Did Not Invent Entailment, We Simply Discovered It

First, let us talk briefly about the notion of a set. We can make a set (a grouping) of anything we want. But the notion of a set itself, is not our invention. Sets exist in nature — there just are sets of things. Now any set we want to think of is a set that is defined by some predicate (criteria for membership) — so I can think (conceive) of the set German cars, and in this case an object “is a member” of this set if it is a car and the car is manufactured by a German car company. Moreover, my VW would then be a member of this set, because it is a a car and it is is manufactured by a German car company. So, sets are natural objects that model/explain a natural phenomenon.

More to our point, however, we can always convert between a logical symbol (or a predicate) and a set (e.g., the set of apples is the set of all objects that the “predicate” apple is true of). In other words, a set is just the extension of some predicate/criteria of membership. Now, then, what is the set equivalent of P implies Q? Well, P implies Q is true in set-theoretic terms if the set corresponding to P is a subset of the set corresponding to Q. More formally,

Examples abound. Think of “if laptop then electronic device” which is equivalent to saying “the set of all laptops is a subset of all electronic devices” or think of “if apple then fruit” which is equivalent to saying the set of all apples is a subset of the set of all fruits”, etc. Now, what does it mean to say A is a subset of B, pictorially? It means we have some space (that contains all A’s) that is a subspace (a space contained) in the space that contains all the B’s (see figure 1b).

With that picture in mind, let us now prove that the logical operation of implication was not invented, but discovered. Let us also prove that this universal template, like all Universally Valid Cognitive Templates are not things we experience, or observe, or incrementally learn, they are just natural “laws of the universe” we live in.

Below in figure 2 we show why P implies Q is false in one situation only. Basically, the laws of nature (or “reality”) explains the truth table in figure 2a.

Note that it can happen that P is true (we are in P), and that Q is true (we are in Q) — this is the black circle, it can also happen that P is false (we are not in P) but Q is true (we are in Q) — this is the green circle, and it can also happen that P is false (we are not in P) and also Q is false (we are also not in Q) — this is the yellow circle (note that U stands for the universe of discourse). The only situation that cannot happen is the one where P is true (we are in P), but Q is false (we are not in Q). The last combination cannot even physically happen. In other words, the logic of implication/entailment confirms with the logic of our physical reality. None of this was “invented” it was simply discovered.

Final Remarks

I have over three posts tried to convince extremists on the empirical side (data-driven, statistical and machine learning folks) that we (humans) give ourselves to much credit to the point that we came to believe that individual experiences/observations are what make up our cognitive apparatus. But nothing else is farther from the truth. Most of what we “know” is imprinted in us, is not individual, is not experienced, and is not “approximate” nor probabilistic. The logic of the universe is not something we can violate.

Stop this childish charade of learning from data — which is nothing more than discovering patterns in data and computational statistics , and start working on the actual hard problems, if you really want to utter the phrase “artificial intelligence”.

___
https://medium.com/ontologik

Recall that the set-theoretic equivalence of logical implication is the subset relation. We can always convert between

Universally Valid Templates: one more time for the Deep Learners who appreciate proofs was originally published in ONTOLOGIK on Medium, where people are continuing the conversation by highlighting and responding to this story.