Domany E., Van Hemmen J.L., Schulten K. (eds.) Models of Neural Networks I

Springer, 1995, -371 p.The first volume of the Physics of Neural Networks series.This book offers a multifaceted presentation of several main issues in the theory of neural networks that have recently witnessed considerable progress: statistics and dynamics of Hopfield-type nets with symmetric and asymmetric couplings, learning algorithms, temporal association, structured data (software), and structured nets (hardware). We consider these issues in turn.
Each review collection that is to appear in the series Physics of Neural Networks is intended to begin with a longer paper that puts together the theoretical foundations of the articles that follow. Here the introductory chapter, authored by van Hemmen and Kilhn, concentrates on the long neglected Collective Phenomena in Neural Networks. It shows that the physical insights and techniques that have been obtained from both equilibrium and nonequilibrium statistical mechanics and the theory of spin glasses can be extremely fruitful. It also shows that Yang's prophetic remarks (see the Foreword), which were written two decades ago, have been vindicated in a rather surprising way.
The present series of review papers, then, starts with Braitenberg's thoughtprovoking Information from Structure: A Sketch of Neuroanatomy. In our opinion, theory needs feedback from experimental neurobiology.
A learning algorithm should solve the problem of coding the information presented to a network. Forrest and Wallace review in Storage Capacity and Learning in Ising-Spin Neural Networks the beautiful work of the Edinburgh group, in particular that of the late Elizabeth Gardner, and show how the different learning algorithms are implemented. In their paper Dynamics of Learning, Kinzel and Opper emphasize supervised learning (adaline and perceptron). They also estimate the learning time and provide additional information on forgetting.
The data that are offered to a network and have to be learned are usually assumed to be random. This kind of data is convenient to generate, often allows an analytical treatment, and avoids specific assumptions. It is known from real-life situations, though, that information can also be hierarchically structured, being tree-like with branches that become finer as we proceed. Here is an academic example: physics (an overall notion), theoretical physics (a category), the theory of neural networks (a class), the theory of learning (a family of notions) to which, say, the perceptron algorithm belongs. In their paper Hierarchical Organization of Memory, Feigel'man and loffe describe how this type of information can be modeled and how the memory's performance can be optimized.
Both software and hardware determine the performance of a network. In the present context, a hardware setup means either specifying the synapses, which contain the infonnation, or devising a layered structure. At the moment, two types of synaptic organization of a single network (without hidden units) allow for an exact analytic treatment of the dynamical response and the storage capacity. One treats either a fully connected network with finitely many patterns or a net which is so diluted that dynamical correlations between different neurons need not be taken into account. In their paper Asymmetrically Diluted Neural Networks, Kree and Zippelius analyze this second type Of network, which has proven useful in several different contexts.
Compared with stationary data, the storage and retrieval of patterns that change in space and time require a new type of synaptic organization. Various approaches to Temporal Association are analyzed by Kiihn and van Hemmen. Not only do they examine the perfonnance but they also compare theoretical predictions of the timing with simulation results, i.e. experiment. It seems that Hebbian learning, for long a venerable abstract issue but revived recently both in experimental work and in theory through appropriate techniques and mathematical implementation, gives rise to unexpected possibilities.
Only a part of the brain's architecture is genetically specified. Most of the synaptic connections are achieved through self-organization in response to input percepts. In their paper Self-organizing Maps and Adaptive Filters, Ritter, Obermayer, Schulten, and Rubner study competitive learning networks .and show how self-organizing feature maps a la Kohonen are able to generate connections between, for example, the retina and the visual cortex. They provide an analysis of the fonnation of "striped" projections, well known from the work of Hubel and Wiesel, and present simulations of a system containing 216 = 65 536 neurons in the (primary) cortex. Though the system size surpasses most of what has been done so far, the performance of the brain is even more impressive, if one realizes that several hours on the Connection Machine, one of the world's most powerful parallel computers, were required to simulate a system with about as many neurons as are contained in 1 mm3 of the brain.
It has been known for a long time that the cerebral cortex has a specific "hardware" structure in that it consists of six layers. Regrettably, present-day theory falls short of appropriately explaining this extremely complicated system. In spite of that, it is a challenging, and rewarding, effort to try to catch the essence of information processing in layered feed-forward structures. In their essay Layered Neural Networks, Domany and Meir present an in-depth analysis.
Several promising developments in the theory of neural networks, in particular the idea of abstractly analyzing a task in the space of interactions, were initiated by the late Elizabeth Gardner. Her ideas permeate several chapters of this book. She died in 1988 at the age of thirty. It is with great respect and admiration that we dedicate the first volume of Models of Neural Networks to her.Collective Phenomena in Neural Networks
Information from Structure: A Sketch of Neuroanatomy
Storage Capacity and Learning in Ising-Spin Neural Networks
Hierarchical Organization of Memory
Asymmetrically Diluted Neural Networks
Temporal Association
Self-organizing Maps and Adaptive Filters
Layered Neural Networks