Hammer B. Learning with Recurrent Neural Networks

Springer, 2000. — 155 p.A challenging question in machine learning is the following task: Is it possible to combine symbolic and connectionistic systems in some mechanism such that it contains the benefits of both approaches? A satisfying answer to this question does not exist up to now. However, approaches which tackle small parts of the problem exist. This monograph constitutes another piece in the puzzle which eventually may become a running system. It investigates so-called folding networks neural networks dealing with structured, i.e., symbolic inputs.
Classifications of symbolic data in a connectionistic way can be learned with folding neural networks which are the subject of this monograph. This capability is obtained in a very natural way via enlarging standard neural networks with appropriate recurrent connections. Several successful applications in classical symbolic areas exist some of which are presented in the second chapter of this monograph. However, the main aim of this monograph is a precise mathematical foundation of the ability to learn with folding networks. Apart from the in-principle guarantee that folding networks can succeed in concrete tasks, this investigation yields practical consequences: Bounds on the number of neurons which are sufficient to represent the training data are obtained. Furthermore, explicit bounds on the generalization error in a concrete learning scenario can be derived. Finally, the complexity of training is investigated.
Moreover, several results of general interest in the context of neural networks and learning theory are included in this monograph since they form the basis of the results for folding networks: Approximation results for discrete time recurrent neural networks, in particular explicit bounds on the number of neurons and a short proof of the super-Turing universality of sigmoidal recurrent networks, are presented. Several contributions to distribution-dependent learnability, an answer to an open question posed by Vidyasagar, and a generalization of the luckiness framework are included. The complexity of standard feed-forward networks is investigated and several new results on the so-called loading problem are derived in this context.Recurrent and Folding Networks
Approximation Ability
Learnability
Complexity