Wittgen's Bio-Neurological Motivation

This post and the following posts will discuss the motivation for the creation of Wittgen and advantages that may arise through using it and thinking along the lines of its paradigms. 

One of the tidbits of wisdom of the computer industry, attributed to Alan Perlis in 1982, says:. “Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy”. Wittgen is not intended to be a Turing tar-pit. There seems little point in using it for general purpose standard applications.

Wittgen belongs to a family of programming languages, environments, virtual or real computers conceived of with a tiny instruction set. Gilreath and Laplante (2003) provide a detailed description of the One Instruction Set Computer (OISC). Wittgen has only two instructions and one special variable. The retrieve instruction has one argument - the entire string it contains - whereas the assign instruction takes two: the content to assign and the variable name.

There are a number of ways of creating a turing-complete programming system from two instructions. The design of Wittgen is based on the fact that both the human brain and your mind clearly seem to include these two. In fact, it is questionable whether the brain, for example, contains anything other than the ability to store a memory and retrieve a specific memory based on some key or some other form of memory instigator. (This assertion is exclusive of task-specific areas of the brain such as vision processing.) If the brain is a computer, or at least the part of it that is capable of discursive reason or other interesting intellectual achievements, then it is important to understand how these capabilities are achieved. If the brain is a two-instruction computer based on assign and retrieve, then Wittgen should be a very interesting experiment.

The current understanding of the human brain sees it as a network of some 1011 neurons. The biological neural network is often modeled using Artificial Neural Networks (ANNs). There are a large variety of such models. The details of the communication between Neurons are continuously being discovered. For example, Baslow (2009) shows that frequency-encoded language is used. This certainly paints a more complex picture than the simple weighted aggregation envisioned by early ANNs. Nevertheless, it may be a terrible simplification, but the functionality of a neural network taken as a whole is that in response to a specific input pattern, it will produce an output pattern. This may be seen as a fixed key-value response except that the response (a) can be learnt and (b) can be robust in the face of noise or input pattern incompleteness.

Traditionally there has been a conflict between advocates of ‘connectionist’ architectures and symbolic or classic computer architectures. Connectionist is a term that includes ANNs but also includes technologies that bear little resemblance to biological neural networks. More recently there has been an attempt to combine the connectionist and symbolic systems. At first this approach was referred to as “Hybrid” but in the last few years the term of choice seems to be Neuro-Symbolic Computing. Sun (1997) and Bader and Hitzler (2005) provide surveys of the research being conducted into these hybrids. Examples are Garcez et al. (2009) that explore the implementation of modal and other logics as ANNs and Siegelmann (1994) and Neto et al. (2003) that create programming languages that are designed to be compiled into neural networks.

Neural-symbolic research addresses the question, among others, of how symbolic systems might be implemented in a neural network implementation as opposed to a von Neumann architecture. The brain seems to be only a (set of) neural network(s). On the other hand, there certainly is some symbolic processing going on. Discursive reasoning, conscious planning, mathematical manipulation and language use seem to be the manipulation of symbolic tokens according to symbolic inference rules (logical or otherwise). Moreover, all this symbolic activity seems to take place in a sequential manner. Certainly, when considering these activities using internal intuitive reflection, they take place in some chronological order; one step at a time. How does the neural network of the brain produce such sequential symbolic capabilities?

This question was already recognized in Rumelhart et al. (1987). The model described there is of a sequence of neuronal activations where at each step an input pattern is fed to a neural net putting it into an excited state, then the net finds the corresponding output pattern, it settles into an energy minimum before another input pattern is provided thus initiating the next step. The input and output can be symbolic tokens and the sequence of activations corresponds to what we imagine is discursive thinking. There is, therefore, no symbolic processor as such, but nevertheless sequential thinking is a sequence of associative memory symbolic lookups.

Wittgen, approximately, uses this model. When Wittgen code is running, the effect is a sequence of associative memory lookups. If you write Wittgen code to do formal logic, say, then the formal logic is implemented in a sequence of associative memory operations. Therefore Wittgen provides a plausible model for how formal logic - a sequential symbolic process - can be executed by an ANN or a brain. This is because it is well understood how a neural network can implement associative memory. Wittgen, provides more than just a plausible model for human symbolic reasoning. It is not just an hypothesis for how reasoning in general might happen; it is a working language in which plausible models can be built for how humans think about specific problems. If used correctly, it can be used to explore how logic, arithmetic, planning, hypothesis formation, debate, strategy and many more fields that form part of what we understand as human reasoning.

Wittgen in its current form does not actually need a neural net to function. While neural nets are very capable of implementing an associative memory, there are other, well known technologies both in software (hash tables for example) or hardware (CAM chips) that can implement the core key-value lookup. Such technologies are extremely fast (a few nanoseconds) while the human neural network seems to take a significant fraction of a second (1-100ms intuitively) to retrieve a single symbolic memory. Nevertheless, since associative memory is a plausible bio-neurological option, it forms the core of the language.

It is possible to create long procedures with tens of items in Wittgen. Similarly, it is possible to create nested retrieve instructions with tens of retrieve levels. However, if the goal is plausible modeling of human reason, this should be avoided. Procedures can always be made very short, with at most two steps - one which does some assign and the other that refers to the next procedure to execute. By following these guidelines it is possible to create processes that are very psychologically plausible. The human mind is capable of storing a very large number of items in memory, each triggered by a different key. However, it is very difficult to remember accurately a long train of steps or to remember exactly step one is up to. (This changes when a page with writing or a scratch pad is used.)

Seen as a model of human symbolic reasoning, the flow of the calculation is controlled by the fact that the only item that the subject must keep track of actively is “Doing Now” which is simply the thought: what am I doing now. All the rest is responsive rather than proactive. For example: “What I am doing now is adding whatever’s in (the top number) to whatever’s in (the bottom number)”, “What I am doing now is adding 5 to 3”, “adding 5 to 3 is 5 plus 3”, “answer is 8”, “What was the next step?” “next step is remember whatever’s in (the answer) as the sum” and finally “the sum is 8”. Thus, following Rumelhart et al.’s model, there is a sequence of simple memory access activation states with no complex serial flow controller guiding the process.

Why then, the need for a neural net, biological or artificial? Isn't the associative memory a sufficient explanatory mechanism? The answer is that without changing the core “syntax” of the model, a neural net can replace an associative memory and extend its power. Firstly, a neural net can learn. As the programmed procedure is activated, the connection between initial starting point and final result can become stored. Thus alternatives to the full calculation arise which are equivalent to “knowing the answer right away”. For example, we could write a multiplication procedure that works on adding the first number to itself as many times as is specified by the second number. However, once I do 7 x 6 the hard way lots of times, I might simply learn that 7 x 6 = 42. It seems very intuitive that this is the way we learn and increase proficiency at whatever we practice repeatedly.

Secondly, a neural network, while learning, might miss certain differences between similar examples. Missing these differences can present a significant advantage. While associative memory will always keep “apples” as a separate string from “oranges”, if the outcome of apples and oranges is often the same, a neural network learning process might easily apply results learned from apples to oranges. One could say that it failed to learn the significance of the difference between the two or one could say that it has learned to create analogies between the two. Search algorithms, when applied to significant quantities of data suffer from the fact that the complexity is O(nD) where D is the dimensionality of the problem. By failing to distinguish between cases that need not be distinguished, one could say that the dimensionality is being collapsed.

Thirdly, outside experience may create the memory assignments. Wittgen includes the ability to assign content to a variable but the database/memory need not start empty nor is there a reason that some other process may not also be writing to memory. The content of memory, the collection of assignments may include associations that neural networks are classically known for. For example, the database may include a neural net trained extensively on good chess moves. The net thus recognizes patterns and excellent moves given specific situations or partial situations (where some of the information, such as where pawns are on the other side of the board, is considered irrelevant). In that case the role of a Wittgen program is to build the fragmentary one-move response knowledge into a play flow. Wittgen is the glue or wires that turns static associative knowledge into a sequentially ordered strategy. Again, it does this without introducing specialized serial hardware or components.

These three considerations are the reason the neural network basis for Wittgen is not simply replaced by associative memory. That does not mean that for performance reasons, the actual implementation of Wittgen might use an associative memory technology that is not a neural network.

This post has presented what may be called the neural network motivation for Wittgen. The next few posts will explore the philosophical aspects of the language. These for another motive for the design of Wittgen.

----

References:

d’Avila Garcez, Arthur; Lamb, L. C; Gabbay D. M., (2009) Neural-Symbolic Cognitive Reasoning. Cognitive Technologies. Springer-Verlag, ISBN 978-3-540-73245-7, 2009.

Bader, Sebastian; Hitzler, Pascal, (2005). "Dimensions of Neural-symbolic Integration - A Structured Survey". arXiv:cs/0511042v1

Baslow, Morris H. (2009). "The Languages of Neurons: An Analysis of Coding Mechanisms by Which Neurons Communicate, Learn and Store Information." Entropy 2009, 11, 782-797; doi:10.3390/e11040782

Gilreath, William F.; Laplante, Phillip A. (2003). Computer Architecture: A Minimalist Perspective. Springer Science+Business Media. ISBN 978-1-4020-7416-5

Neto, J. P. ; Siegelmann H. T.; Costa J. F. (2003). "Symbolic processing in neural networks." Journal of the Brazilian Computer Society, 8(3)

Rumelhart, D.E.; Hinton, G.E.; McClelland, J.L.; Smolensky P., (1987). "Schemata and Sequential Thought Processes." In D. E. Rumelhart, J. L. McClelland, and the PDP Research Group (Eds.). Parallel Distributed Processing, Vol, 2: Psychological and Biological Models. Cambridge, MA: MIT Press.

Siegelmann, Hava T., (1994). "Neural Programming Language." AAAI-94 Proceedings.

Sun, R., (1997). "An Introduction to Connectionist Symbolic Integration." R. Sun and F. Alexandre. (eds) Connectionist Symbolic Integration. Lawrence Erlbaum Associates, Hillsdale, NJ, 1997

Using Wittgen To Do Logic

This post will continue introducing Wittgen by way of giving usage examples. However, this time, the focus is some of the components of human reason: logic and arithmetic. It also tackles the question of how a language with just two instructions that is missing almost all the elements common to all programming languages can still do anything its bigger brothers and sisters can.

One might imagine that the most basic construct that a programming language needs is the conditional branching operator: the “if” statement. It is not missing; it is simply that the retrieve can do the job on its own. Let’s say you want to terminate a loop when the variable Num is equal to 0. In a classic programming language you would write something like:

if (Num == 0) {

        do this ...

}

else {

        do that ...

}

Assume there is in Wittgen’s memory, a variable that is called “stop when 0”. If there is an assign in the loop such as: 

        Doing Now:=@stop when @Num}}}

First Wittgen evaluates @Num}. If this is “0”, the right side becomes “@stop when 0}”. If, say, Num is “2” then the right side becomes “@stop when 2}”. Now there is a variable called “stop when 0” but there is no variable called “stop when 2”. “stop when 0” might contain an assign to Doing Now that would do whatever we wanted to do if "Num" was "0". On the other hand, “stop when 2” does not exist, so any assign involving it would fail. We thus have a simple mechanism that implements the conditional execution construct. One might even argue that this memory assign operation is more fundamental than the “if” operation. Here is an interesting possibility: What we think of as “if” is just a choice between different memory access options.

The If construct is one of the most basic components of logic. Symbolic logic as a whole will be discussed in later posts but by now it should seem plausible that the remaining constructs are also implementable as Wittgen code. Now what about basic arithmetic?

How would you make Wittgen do addition of two numbers of arbitrary length? Computers do this very fast while human beings are billions of times slower. How do we human being do addition? We spend years learning some basic single digit addition. So we memorize, for example, that 3+4 is 7, that 8+5=13 etc. Then we painstakingly learn procedures that teach us to write one number on one line and the second number on the second line with the digits lined up in the correct columns. Then, starting with the first column on the right we add the digits using the single digit additions we've memorized. If the answer has two digits we write down the first digit and put the second digit in the next column in a “carry” row. We do column after column this way until there are no more digits in the current working column. It’s very slow and very inefficient but it’s the way we learn to add arbitrarily large numbers.

If you want Wittgen to do addition, the right way to go about it, is to make it do addition the way humans do it. The code can be found on the Wittgen Sourceforge web site: http://sourceforge.net/projects/wittgen/

Can Wittgen do anything any other language can do? Technically, yes. Wittgen can be used to create a Universal Turing Machine - a Turing Machine that can implement a Turing Machine. This proves the standard Computer Science theoretical capability. This says nothing about what kinds of program are easier to write in which language and certainly not whether specific programming languages create mindsets that are appropriate for creative thinking regarding tough problems. A Turing Machine implementation will hopefully be up on the Wittgen Sourceforge Web site soon.

That concludes the all-too-brief usage section for now. It may only have been a taste, but talking about why Wittgen was created in the first place has been delayed for too long. So what follows in the next few posts is some Cognitive Science and philosophy. Once that's done we can get back to expanding on the art of programming Wittgen.

Using Wittgen

This post and the next one will present some simple examples showing how to use Wittgen. The purpose is to make the language definition easier to understand. More advanced examples will come later, but for now the goal is simply to understand what the language is.

Motivations for the creation of Wittgen and its structure will be explained in later posts. However, for now, it is important to emphasize that the motivation is not the creation of a high level language so that standard programming problems can be solved in a small number of highly intuitive code lines. It is also not designed as a low level language so that the code should run as fast as possible. In fact, simple problems, may take more code to program and, performance may be ridiculously worse than in typical environments - particularly for maths.

The first, introductory program has already been presented. (It is traditional to introduce programming languages  using “hello world!”). The next program is designed to produce an infinite series of a’s. It will demonstrate the use of Doing Now as well as how to create a simple loop.

1 prog1:=

2 say:=@say} a}

3 Doing Now:=@prog1}}

4 }

5 say:=a}

6 Doing Now:=@prog1}}

Ignore the line numbers, they should not be included in the source. They are here only to make explaining the code easier. 

How does the code above work? There is an intuitive way to think of this. Imagine a game with one rule and one box. The box starts off containing one letter a. The one rule in the game goes like this: “Whatever is in the box, add another “a” to the box and then do this rule again.” Such a procedure is natural and easy for a human being.

That is what the code above does. The rule and the box are held in memory. The box is the variable called “say” and the rule is the variable called “prog1” which refers to the text from line 2-4 inclusive. 

Now, the full details of how Wittgen actually processes this short piece of code will be presented:

To start Wittgen the entire text is assigned to the variable Doing Now. Wittgen’s memory will now contain one variable called Doing Now whose value will be the text string:

        “prog1:=say:=@say} a}Doing Now:=@prog1}}}say:=a}Doing Now:=@prog1}}”

From then on the program will continue until the next time Doing Now needs to be executed and there is no assign to perform.

The first assign is the text from line 1 to 4 inclusive. The first step that Wittgen will take is to set Doing Now to:

        “say:=a}Doing Now:=@prog1}}”

and now it must create a new variable called prog1 and assign to it a text:

        “say:=@say} a}Doing Now:=@prog1}}}” 

Now here is an interesting point. There are two retrieve instructions in this text string; @say} and @prog1}. Why were they not evaluated before assigning the text string to prog1? The answer is that both of these are inside assign statements. @say} is part of the assign say:=@say} a} and is therefore left as is. Similarly, @prog1} is part of the assign Doing Now:=@prog1}}. We are not up to doing those assigns because we are still doing the outer assign. The result is that the whole string is assigned without replacement to prog1.The first assign is done. Wittgen’s memory can be represented by the following table:

Doing Now	say:=a}Doing Now:=@prog1}}
prog1	say:=@say} a}Doing Now:=@prog1}}

The first assign to Doing Now from line 5. This is a simple assign to a new variable called "say". The memory now looks like:

Doing Now	Doing Now:=@prog1}}
prog1	say:=@say} a}Doing Now:=@prog1}}
say	a

Now Doing Now has only one assign left - the assign of line 6 in the original code. Wittgen needs to evaluate the string “@prog1}” and then assign it to Doing Now. This requires retrieving the value of prog1 which is the second entry in the table here. Careful attention must be paid to the order here. Before the evaluation is done the assign is removed from Doing Now, so Doing Now now refers to an empty text. However, the execution of the assign instruction will put a new value in Doing Now. So that at the end of the assign, Doing Now has a new activity. The memory will now look as follows:

Doing Now	say:=@say} a}Doing Now:=@prog1}}
prog1	say:=@say} a}Doing Now:=@prog1}}
say	a

Doing Now now has two assigns to do. The first is line 2 of the original code and the second is line 3. The first assign evaluates “@say} a” by retrieving the value of say, which happens to be “a”, and therefore replacing @say} with an a to get the new string “a a”. This value is now put in the variable say. The memory now looks like:

Doing Now	Doing Now:=@prog1}}
prog1	say:=@say} a}Doing Now:=@prog1}}
say	a a

Doing Now looks exactly the way it looked two steps ago. The difference is that now the value of say is “a a” and not “a”. Therefore in two steps time say will have the value “a a a”. Wittgen will run forever, making say have more and more a’s.

The explanation was a little lengthy but actually all that the code does is “Whatever is in the box, add another “a” to the box and then do this rule again.”. It was important to explore what is actually going on in memory during execution and how each of the instructions works.

The next post will go through a few more examples. After that it will be time to explain why Wittgen was designed the way it was and how all this relates to cognition and philosophy.

Wittgen now available in Sourceforge

I have uploaded a simple implementation of Wittgen to Sourceforge.

It can be accessed from:

http://www.sourceforge.com/p/wittgen

Of course, Wittgen is so simple that you may be of the opinion that there is no need to provide an implementation. After all, with a map class, you could just build it yourself.

Nevertheless, the code is provided as well as a little IDE that lets you step through assign by assign. The code is written in C++ using Qt. I have tested it in Linux.

There  are also a few trivial examples there that I hope to provide explanation for in subsequent posts.

Any feedback will be appreciated.