Computational Intelligence - Learning with Neural Methods on Structured Data

Theory of FNNs and RNNs

Main contributors:

Barbara Hammer, several cooperations with: Bhaskar DasGupta (Univerity of Illinois), Jochen J. Steil (Universität Bielefeld), Peter Tino (Aston University),

Publications:

See publications on Barbara's page.

Main directions:

• Architectural bias of recurrent networks:

  Recurrent networks are commonly initialized with small weights. We have established an equivalence of such RNNs to finite memory machines. Moreover, benefits from the point of learning theory can be proved and further investigaion of the bias estimating fractal dimensions is possible.

• Representation capability of recursive networks:

  Recursive networks, holographic reduced representation, or the recursive autoassociative memory are examples of networks which have been proposed for direct processing of tree structured data or directed ordered acyclic graphs. They share the basic underlying dynamics. A uniform notation for the approaches has been developed and their in-principle capabilities of representing structured data in a connectionistic way as well as their approximation abilities have been investigated.

• Statistical learning theory of recurrent/recursive networks:

  Recurrent and recursive networks, unlike standard feedforward networks, have a VC dimension which depends on the maximum size of inputs. Hence for obtaining guarantees for the generalization ability and concrete bounds advanced concepts of statistical learning theory are to be used. Based on covering numbers and the luckiness framework, respectively, distribution or data dependent guarantees can be developed.

• The loading problem of neural networks:

  The question of the complexity of training neural networks in realistic situations is so far an unsolved problem. We made several further steps into this direction, establishing NP-hardness results for several more realistic scenarios such as approximate loading, related training set size and architecture, and more realistic activation functions.

B.Hammer