Neural networks are computer-based systems that learn models from data presented to them. They are built of simple processing units called perceptrons. Each perceptron takes a set of inputs (which, for formulation, correspond to the ingredients and processing conditions), weights these inputs, sums them and then transforms them to produce an output. This calculated output is compared with the known experimental value, and the amount of error in the prediction is calculated. The weights are adjusted in an iterative procedure, until the calculated outputs match the experimental ones.

One of the keys to effective use of neural networks is to make sure that they retain the ability to predict, and don't simply "memorize" the data. This is generally accomplished by reserving some of the known data points from the training set, and using them to validate the model.

Neural networks are especially useful when there are many input variables, and when the relationship between the inputs and outputs is poorly understood. Unlike statistics, they do not require you to make any assumptions about the functional form of the relationship. However, they are 'black box' models and it is not easy to extract a simple model from them. Because their strengths and weaknessed complement those of statistical treatments, they make an excellent complement to statistics (but are easier to use!).

Neural networks can be combined with other intelligent technologies - for example fuzzy logic. This can give more 'accountable' models than those obtained from simple neural networks alone.

Neural networks have been widely used for formulation modelling, and have been shown to develop models similar to those from statistics. For a review of the area, ask for a copy of our article (published in 2003) in The Chemical Educator.

Genetic algorithms are a sophisticated optimization technique. They rely on the fact that a criterion of fitness can be defined - this reflects how well each individual solution meets our goals. A population of possible solutions are generated, and the fitness of each is assessed. The population evolves (using evolutionary concepts like mutation and cross-over) so that ultimately the fittest solutions are found.

Genetic algorithms are particularly useful in very complicated problems, where a more directed optimization (like conjugate gradients or steepest descent) would be likely to find a local optimum rather than the global one.

Genetic programming is an evolutionary computing technique in which equations are derived to fit to a data set. Each member of the population is an expression tree (sometimes called a 'parse tree') is used to represent an equation. For example, the equation (a+b)*(c-d) could be represented by the tree

Operations like mutation and crossover can be performed on the tree. For example, if you have two trees, then for crossover, a node can be selected for each, and all parts of the tree that lie below these nodes will be swapped between the two trees. For mutation, a node is selected and a new sub-tree is grown from that node, replacing the previous one.

The population is allowed to evolve during successive generations, using a criterion of 'fitness' which means that the 'fittest' individuals (trees/equations) are those which best fit the known data.

Genetic programming can grow very complex trees (each of which represents an equation which is a possible solution to the problem, i.e. a possible fit to the 'training' data). It is also somewhat inefficient, since unrealistic trees can be obtained if, for example, an operator like + (which takes two arguments) is replaced by one like exp (which takes only one). However, genetic programming has found some applications in the formulation domain.

Gene Expression Programming (GEP) is an extension of Genetic Programming which avoids the issues associated with mutation and crossover of the tree structures. It does this by using a link (so-called K-notation, or Karva language) which links a linear string of information to the tree structure. For the equation (a+b)*(c-d) whose tree representation is given above, the K-expression is *+-abcd. This K-expression forms part of the gene, which must have at least the same length as the K-expression.

Each member of the population in GEP is therefore a 'chromosome' made up of one or more genes, which represents a tree/equation. Operations like mutation and crossover can be carried out on the chromosomes, and in turn can have a substantial effect on the individual trees (and hence equations) that are proposed as solutions. As in GP and Genetic Algorithms, there is a criterion of 'fitness' and the population evolves over successive generations to obtain the best, or 'fittest' solution.

A full explanation of Gene Expression Programming can be obtained in a book by Candida Ferreira, Gene Expression Programming: Mathematical Modeling by an Artificial Intelligence, 2nd Edition, Springer (2006).

Conventional logic demands that we categorize data: true/false, yes/no, black/white. There is no allowance for ambiguity or imprecision where conventional logic is concerned.

However, much information that we need to deal with day-to-day is vague, ambiguous, incomplete and imprecise. Fuzzy logic has attracted much attention in recent years as a problem-solving technique whose power lies in its ability to draw conclusions and generate responses based on such ambiguous and imprecise information. Rather than the simple true/false of conventional logic, fuzzy logic allows elements to belong to different ‘sets’ with a defined membership function in each.

Fuzzy logic allows specification of data in qualitative, and intuitively obvious, ways. For example, in defining a room temperature, fuzzy sets let you specify terms like hot and cold (which traditional logic would allow) but also to add terms like warm, cool etc. Ask any group of 100 people whether a given room is hot or cold - unless the temperatures are at extremes, you will get a mixed response, which you can define using fuzzy logic.

Fuzzy logic has been used extensively for process control, but has been applied less frequently in the formulation domain.

Neurofuzzy logic is an artificial intelligence technology which generates rules directly from numerical data, and assigns probabilities to those rules. Neurofuzzy logic combines fuzzy logic with the learning and adaptive capabilities of neural networks. This means that a user can initialize the neural network using expert knowledge expressed as fuzzy rules, and then train it using numerical data. After training the net, the rules can be evaluated and edited to overcome any deficiencies in the training data.

More recent research has produced ‘learning’ fuzzy systems that can automatically extract a set of fuzzy rules from a given data set, without the requirement to initialize the system with known rules at the outset. Recent algorithms add a means for validating and editing the trained fuzzy rules, even in the absence of expert knowledge. There are added advantages in that the network produced by neurofuzzy techniques can be simplified automatically as they search for the best model to represent the data set.

Where has neurofuzzy logic been applied?

Increasingly, neurofuzzy logic is being used in data-mining applications. Presented with data, the neural net capabilities extract rules, expressed in a linguistic form, which allow the user to make decisions about what to do next.

A lot of the early applications are concerned with data mining for financial applications, and for marketing. Here, there are generally large amounts of data - although the data may be of variable quality. For example, it might be collected by looking at purchase decisions by monitoring supermarket loyalty cards. However, more recent work has shown that these techniques can be used effectively for formulation projects.

For most formulation problems, we do not have a large amount of data - certainly not the amount that exists for large-scale marketing applications, for example.

However, the data we have are generally high quality experiments, which means that even from relatively few data points, useful rules can be extracted.

The pages on Automatic Rule Generation and Intelligent Models will lead you to some Applications Examples where neurofuzzy logic has been applied to formulation.