Fuzzy neural network
From Scholarpedia
| This article is undergoing 1 initial review (1 completed); It may contain inaccuracies and unapproved changes made by anonymous reviewers. | ||||||||||||||||||||
A fuzzy neural network or neuro-fuzzy system is a learning machine that finds the parameters of a fuzzy system (i.e., fuzzy sets, fuzzy rules) by exploiting approximation techniques from neural networks.
Contents |
Combining fuzzy systems with neural networks
Both neural networks and fuzzy systems have some things in common. They can be used for solving a problem (e.g. pattern recognition, regression or density estimation) if there does not exist any mathematical model of the given problem. They solely do have certain disadvantages and advantages which almost completely disappear by combining both concepts.
Neural networks can only come into play if the problem is expressed by a sufficient amount of observed examples. These observations are used to train the black box. On the one hand no prior knowledge about the problem needs to be given. On the other hand, however, it is not straightforward to extract comprehensible rules from the neural network's structure.
On the contrary, a fuzzy system demands linguistic rules instead of learning examples as prior knowledge. Furthermore the input and output variables have to be described linguistically. If the knowledge is incomplete, wrong or contradictory, then the fuzzy system must be tuned. Since there is not any formal approach for it, the tuning is performed in a heuristic way. This is usually very time consuming and error-prone.
| Neural Networks | Fuzzy Systems |
|---|---|
| no mathematical model necessary | no mathematical model necessary |
| learning from scratch | apriori knowledge essential |
| several learning algorithms | not capable to learn |
| black-box behavior | simple interpretation and implementation |
It is desirable for fuzzy systems to have an automatic adaption procedure which is comparable to neural networks. As it can be seen in Table 1, combining both approaches should unite advantages and exclude disadvantages.
Characteristics
Compared to a common neural network, connection weights and propagation and activation functions of fuzzy neural networks differ a lot. Although there are many different approaches to model a fuzzy neural network (Buckley and Hayashi, 1994, 1995; Nauck and Kruse, 1996), most of them agree on certain characteristics such as the following:
- A neuro-fuzzy system based on an underlying fuzzy system is trained by means of a data-driven learning method derived from neural network theory. This heuristic only takes into account local information to cause local changes in the fundamental fuzzy system.
- It can be represented as a set of fuzzy rules at any time of the learning process, i.e., before, during and after.
- Thus the system might be initialized with or without prior knowledge in terms of fuzzy rules.
- The learning procedure is constrained to ensure the semantic properties of the underlying fuzzy system.
- A neuro-fuzzy system approximates a n-dimensional unknown function which is partly represented by training examples.
- Fuzzy rules can thus be interpreted as vague prototypes of the training data.
- A neuro-fuzzy system is represented as special three-layer feedforward neural network as it is shown in Figure 1.
- The first layer corresponds to the input variables.
- The second layer symbolizes the fuzzy rules.
- The third layer represents the output variables.
- The fuzzy sets are converted as (fuzzy) connection weights.
- Some approaches also use five layers where the fuzzy sets are encoded in the units of the second and fourth layer, respectively. However, these models can be transformed into a three-layer architecture.
Learning fuzzy sets
In order to build a fuzzy controller, membership functions which express the linguistic terms of the inference rules have to be defined. In fuzzy set theory, there does not exist any formal approach to define these functions. Any shape (e.g., triangular, Gaussian) can be considered as membership function with an arbitrary set of parameters. Thus the optimization of these functions in terms of generalizing the data is very important for fuzzy systems. Neural networks can be used to solve this problem.
By fixing a distinct shape of the membership functions, say triangular, the neural network must optimize their parameters by gradient descent (Nomura et al., 1992). Thus, aside information about the shape of the membership functions, training data must be available as well-
Another approach (Hayashi et al., 1992) is to group the training data 
into 
clusters. Every cluster represents a rule 
where 
. Hence these rules are not defined linguistically but rather by crisp data points 
. Thus a neural network with 
input units, hidden layers and output units might be applied to train on the pre-defined clusters. For testing, an arbitrary pattern 
is presented to the trained neural network. Every output unit 
will return a degree to which extend may fit to the antecedent of rule .
To guarantee the characteristics of a fuzzy system, the learning algorithm must enforce the following mandatory constraints:
- Fuzzy sets must stay normal and convex.
- Fuzzy sets must not exchange their relative positions (they must not pass each other).
- Fuzzy sets must always overlap.
Additionally there do exist some optional constraints like the following:
- Fuzzy sets must stay symmetric.
- The membership degrees must sum up to 1.
Learning fuzzy rules
By definition the first and third layer, i.e., the input and output variables, are given. If the rules for the controller (i.e., the second layer of the fuzzy neural network) are not known yet, fuzzy rules can also be induced from the given training data. Such a fuzzy rule base can be created by
- using prior knowledge
- fuzzy cluster analysis (every found cluster corresponds to a rule)
- fuzzy decision trees
- genetic algorithms
- grid-based rule induction
- neural networks
- or other rule induction techniques that are not listed here.
As an example for learning fuzzy rules, an initial fuzzy partition for all input variables might be given. This is shown in Figure 3. Then, algorithms like in (Wang and Mendel, 1992) can be run to induce a rule base. The resulting network and the rules are shown in Figure 4.
Neural Networks Including Fuzzy Systems
One can also directly include a fuzzy controller into a neural network. In (Berenji, 1992) the ARIC (approximate reasoning based intelligent control) is presented as a neural network. Here, a prior defined rule base is tuned by updating the networks's prediction. Thus the advantages of fuzzy systems and neural networks are easily combined as presented in Table 1.
The ARIC is represented by two feed-forward neural networks, the action-state evaluation network (AEN) and the action selection network (ASN). The ASN is a multilayer neural network representation of a fuzzy system. It then again consists of two separate. The first one represents the fuzzy inference and the second one computes a confidence measure based on the current and next system state. Both parts are eventually combined to the ASN's output.
As it is shown in Figure 1, the first layer represents the rule antecedents, whereas the second layer corresponds to the implemented fuzzy rules and the third layer symbolized the system action. The network flow is at follows. In the first layer the system variables are fuzzified. In the next step these membership values are multiplied by the attached weights of the connections between the first and second layer. In the latter layer, every rule's input corresponds to the minimum of its input connections.
A rule's conclusion is installed as membership function. This function maps the inverse rule input value. Its output values is then multiplied by the weights of the connections between second and third layer. The final output value is eventually computed by the weighted average of all rules' conclusions.
The AEN (which is as three-layer feed-forward neural network as well) aims to forecast the system behavior. The hidden layer obtains as input both the system state and an error signal from the underlying system. The output of the networks shall represent the prediction of the next reinforcement which depends on the weights and the system state. The weights are changed by a reinforcement procedure which takes into consideration the outputs of both networks ASN and AEN, respectively. ARIC was successfully applied to the cart-pole balancing problem.
Whereas the ARIC model can be easily interpret as a set of fuzzy-if-then rules, the ASN network to adjust the weights is rather difficult to understand. It is a working neural network architecture that utilizes aspects of fuzzy systems. However, a semantical interpretation of some learning steps is not possible.
Berenji and Khedkar (1992) introduced an improvement of the their former approach named GARIC (generalized ARIC). This idea does not suffer from different interpretations of the linguistic values anymore by refraining from weighted connections in the ASN. Instead the fuzzy sets are represented as nodes in the network. Moreover the learning procedure changes parameters of these nodes and thus the shape of the membership functions. GARIC is also able to use any kind of membership functions in the conclusion since a different defuzzifier and a differentiable soft-minimum function are used.
Note that the ANFIS model (Jang, 1993) also implements a Sugeno-like fuzzy system in a network structure. Here a mixture of plain backpropagation and least mean squares procedure is used to train the system. Both the ANFIS and the GARIC model are not so easy to interpret as, e.g., Mamdani-type fuzzy systems. Therefore models like NEFCON (Nauck, 1994), NEFCLASS (Nauck and Kruse, 1996) and NEFPROX (Nauck and Kruse, 1997) have been developed for neuro-fuzzy control, classification and regression, respectively. They all implement Mamdani-type fuzzy systems and thus use special learning algorithms.
In addition to multilayer feedforward networks there are also combinations of fuzzy techniques with other neural network architectures, e.g., self-organizing feature maps (Bezdek et al., 1992; Vuorimaa, 1994), or fuzzy associative memories (Kosko, 1992). Some approaches refrain from representing a fuzzy system in a network architecture. They only utilize a learning procedure to the parameters of the fuzzy system, or explicitly use a neural network to determine them.
References
- Berenji, H.R. (1992). A Reinforcement Learning Based Architecture for Fuzzy Logic Control. International Journal of Approximate Reasoning 6, 267-292.
- Berenji, H. R. and Khedkar, P. (1992). Learning and Tuning Fuzzy Logic Controllers Through Reinforcements, IEEE Trans. Neural Networks, 3, pp. 724-740.
- Bezdek, J. C., Tsao, E. C.-K. and Pal, N. R. (1992). Fuzzy Kohonen Clustering Networks, in Proc. IEEE Int. Conf. on Fuzzy Systems 1992 (San Diego), pp. 1035-1043.
- Buckley, J. J. and Hayashi, Y. (1994). Fuzzy neural networks: A survey, Fuzzy Sets and Systems 66, pp. 1-13.
- Buckley, J. J. and Hayashi, Y. (1995). Neural networks for fuzzy systems, Fuzzy Sets and Systems 71, pp. 265-276.
- Hayashi, I., Nomura, H., Yamasaki, H. and Wakami, N. (1992). Construction of Fuzzy Inference Rules by NFD and NDFL. International Journal of Approximate Reasoning, 6, pp. 241-266.
- Jang, J.-S. R. (1993). ANFIS: Adaptive-Network-Based Fuzzy Inference Systems, IEEE Transaction on Systems, Man, and Cybernetics 23, pp. 665-685.
- Kosko, B. (1992). Neural Networks and Fuzzy Systems. A Dynamical Systems Approach to Machine Intelligence (Prentice-Hall, Englewood Cliffs).
- Nauck, D. (1994). A Fuzzy Perceptron as a Generic Model for Neuro-Fuzzy Approaches, in Proc. Fuzzy-Systeme 94 (Munich).
- Nauck, D. and Kruse, R. (1996). Neuro-Fuzzy Classification with NEFCLASS, in P. Kleinschmidt, A. Bachem, U. Derigs, D. Fischer, U. Leopold-Wildburger and R. Möhring (eds.), Operations Research Proceedings 1995, (Berlin), pp. 294-299.
- Nauck, D. and Kruse, R. (1997). Function Approximation by NEFPROX, in Proc. Second European Workshop on Fuzzy Decision Analysis and Neural Networks for Management, Planning, and Optimization (EFDAN'97), (Dortmund), pp. 160-169.
- Nomura, H., Hayashi, I. and Wakami, N. (1992). A Learning Method of Fuzzy Inference Rules by Descent Method, in Proc. IEEE Int. Conf. on Fuzzy Systems 1992 (San Diego), pp. 203-210.
- Vuorimaa, P. (1994). Fuzzy Self-Organizing Map, Fuzzy Sets and Systems 66, pp. 223-231.
- Wang, L.-X. and Mendel, J. M. (1992). Generating Fuzzy Rules by Learning from Examples, IEEE Transaction on Systems, Man, and Cybernetics 22, pp. 1414-1427.
Recommended reading
- Klawonn, F. Nauck D. and Kruse, R. (1997). Foundations of Neuro-Fuzzy Systems (Wiley, Chichester, United Kingdom).
- Klawonn, F., Kruse R., Nauck, D. and Borgelt, C. (2003). Neuro-Fuzzy-Systeme (Vieweg, Wiesbaden).
