This is a FeedForward Neural Network with back-propagation written in C++ with no external libraries. The aim for this project was to create a simple version of TensorFlow's Keras sequential neural network from scratch in C++. As a challenge, I did not refer to any of TF's source code or implementation.
In Dependencies folder:
- Matrix: This is a simple matrix class that allows for matrix operations and several other useful methods.
at(row,col)
: returns value at specified row and column (0-indexed).Transpose()
: returns a transposed copy of current matrix.- static
Matrix::Zero(rows,cols)
andMatrix::One(rows,cols)
: return a matrix of all zeroes or ones respectively provided number of rows and columns. set_values(vector,rows,cols)
: initialize a matrix with a 2d vector of floats (can accept 1d vector of floats).
- Layer: Basic element of the NN class. It provides a handful of activation functions.
- Activation Functions:
- Sigmoid (Default)
- ReLU
- Weight Initialization:
- Normalized Xavier
- Some Methods:
forward(Matrix)
: returns the output of activation(Weights*Matrix + Bias) as a Matrix.backward(Matrix,float)
: calculates delta_W and delta_B for both weights and biases respectively.summary()
: prints a summary of the layer properties (units, input shape, etc.).
- Activation Functions:
- Cost function: MSE
- Methods:
forward_pass(Matrix)
: returns output of one forward pass using the neural network given Matrix input.back_prop(Matrix,float)
: performs one backward pass and updates weights and biases of all layers given expected Matrix and learnRate float.train(Matrix,Matrix,int,float)
: trains neural network given X_train and y_train matrices provided number of epochs and learnRate. (Please refer to data set preparation section for more details)test(Matrix)
: prints neural network output after 1 pass given input. (equivalent to printing forwad_pass).summary()
: prints a summary of all the neural network properties. (shows weight and biases matrices of each layer).
In order to train our model we would require a data set. The method train() expects 2 Matrix instances. Each of the matrices should be as follows.
[Column_1 Column_2 ...]
For the matrix X Each column would represent one input to the neural network (a bit counter intuitive, I know 😄). On the other hand, it would represent the expected output for y.
Note: when it comes to test() method it should be provided with a Column vector instance of class Matrix.
To initialize a neural network:
#include "Neural Network/NeuralNetwork.h"
int main(){
//Create array of layers
// Layer(units, input_shape, activation, weight_initialization)
// all strings are typed in **lower-case**
// You need to only specify input shape in first layer
Layer Layers[] = {
Layer(2,2,"relu","nx"),
Layer(1,"relu")
};
//Initialize neural network with array and size
NN model(Layers, 2);
}
This would create an architecture shown in the picture below.
After that, you would want to prepare a dataset. A way you could do that is by doing the following:
Matrix X(2, 4), Y(1, 4);
X.at(0, 0) = 0;
X.at(1, 0) = 0;
X.at(0, 1) = 1;
X.at(1, 1) = 0;
X.at(0, 2) = 0;
X.at(1, 2) = 1;
X.at(0, 3) = 1;
X.at(1, 3) = 1;
Y.at(0, 0) = 1;
Y.at(0, 1) = 1;
Y.at(0, 2) = 1;
Y.at(0, 3) = 0;
This would create the equivalent matrices shown below.
If you noticed, it acts as the logical table of a NAND gate!.
Now, we want to train our model. We can use the train()
method.
int epochs = 10000;
float learnRate = 0.05;
//Display train data set
cout <<"X_train:\n"<< X <<"Y_train:\n" << Y<<endl;
//Train
model.train(X, Y, epochs, learnRate);
You would see this on your screen. It will also print a progress bar that would gradually print 10 "#" hashes, each representing 10% closer to completion.
When it comes to testing, all you need is have column vectors of your input, then use test()
method.
//Test
cout << "\nTesting:\n";
Matrix test;
test = X.Column(0);
cout << "Y0:";
model.test(test);
test = X.Column(1);
cout << "Y1:";
model.test(test);
test = X.Column(2);
cout << "Y2:";
model.test(test);
test = X.Column(3);
cout << "Y3:";
model.test(test);
cout << "\nExpected to be close to: " <<Y;
Then, you should have something like this.
If you want, you could print a model's summary using the summary()
method as shown below.
//Printing summary of model
model.summary();
- Add ability to save and load a model.
- Implement Dropout layers.
- Implement BatchNormalization layers.
- Add ability to change cost function.
- Add Adam optimizer.
- Optimize matrix multiplication. (maybe use Strassen's Algorithm)