Why it’s great to master Neural Networks?

Mastering Neural Networks offers numerous advantages in the field of machine learning and artificial intelligence:

1. Versatility: Neural Networks provide a versatile framework for modeling and solving various machine learning tasks. By mastering Neural Networks, you gain a powerful tool that can be applied to tasks such as classification, regression, and pattern recognition, among others.
2. Model integration: Understanding Neural Networks allows you to model and integrate other machine learning techniques within the network architecture. This capability enables you to combine the strengths of different algorithms and create more robust and accurate models.
3. Activation functions: Neural Networks rely on activation functions to introduce non-linearity and complex representations. By mastering activation functions, you can design and fine-tune the behavior of Neural Networks, enhancing their learning and prediction capabilities.
4. Weight calculation: Neural Networks involve the calculation of weights that determine the strength of connections between neurons. Mastering different weight calculation techniques, such as gradient descent and its variants, equips you with the knowledge to optimize the performance of Neural Networks and improve their accuracy.
5. Batch sizes and Epochs: Understanding the concepts of batch sizes and epochs is crucial for efficient training of Neural Networks. By mastering these concepts, you can control the amount of data processed in each iteration (batch) and the number of iterations over the entire dataset (epochs), leading to faster convergence and improved training outcomes.

What will be covered in this tutorial?

This tutorial on Neural Networks will cover the following topics:

• Understanding Neural Networks: Exploring the fundamental concepts and principles behind Neural Networks, including their architecture, layers, and connectivity patterns. This foundational knowledge provides a solid understanding of how Neural Networks function and their potential for solving complex problems.
• Modeling other machine learning techniques: Demonstrating how Neural Networks can be used to model and integrate various machine learning algorithms. This includes incorporating decision trees, support vector machines, and other techniques within the Neural Network framework, offering more flexibility and improved performance.
• Activation functions: Exploring different types of activation functions used in Neural Networks, such as sigmoid, ReLU, and tanh. Understanding the characteristics and behaviors of these activation functions allows you to choose the most suitable one for a given problem and optimize the performance of Neural Networks.
• Building a simple OR function: Implementing a Neural Network to learn and perform the logical OR function. This hands-on exercise provides practical experience in designing and training a Neural Network to solve a simple problem.
• Weight calculation techniques: Introducing different methods for calculating weights in Neural Networks, including gradient descent and its variations like stochastic gradient descent. Understanding these techniques empowers you to effectively optimize the weights of Neural Networks and improve their learning capabilities.
• Batch sizes and Epochs: Explaining the concepts of batch sizes and epochs in Neural Network training. Understanding the impact of batch sizes on convergence and the role of epochs in controlling the number of training iterations allows you to fine-tune the training process for optimal performance.

By mastering Neural Networks and completing this tutorial, you will acquire the knowledge and skills to effectively design, train, and optimize Neural Networks for a wide range of machine learning tasks. This expertise will enable you to leverage the versatility and power of Neural Networks to solve complex problems and contribute to advancements in the field of artificial intelligence.

Step 1: What is Artificial Neural Network

Artificial Neural Network are computing systems inspired by the biological neural networks that constitute animal brains.

Often just called Neural Network.

The first Neural Network is the following simple network.

Where w1 and w2 are weights and the nodes on the left represent input nodes and the node on the right is the output node.

It can also be represented with a function: h(x1, x2) = w0 + w1*x1 + w2*x2

This is a simple calculation, and the goal of the network is to find optimal weights. But we are still missing something. We need an activation function. That is, how to interpret the output.

Here are some possible activation functions.

• Step function: 𝑔(𝑥)=1 if 𝑥≥0, else 0
• Rectified linear unit (ReLU): 𝑔(𝑥)=max(0,𝑥)
• Sigmoid activation function: sigmoid(𝑥)=1/(1+exp(−𝑥))(x)

Step 2: How to model the OR function

We see the weights are one each. Then let’s analyse it with the activation function g, given by Step function.

• x1 = 0 and x2=0 then we have g(-1 + x1 + x2) = g(-1 + 0 + 0) = g(-1) = 0
• x1 = 1 and x2=0 then we have g(-1 + x1 + x2) = g(-1 + 1 + 0) = g(0) = 1
• x1 = 0 and x2=1 then we have g(-1 + x1 + x2) = g(-1 + 0 + 1) = g(0) = 1
• x1 = 1 and x2=1 then we have g(-1 + x1 + x2) = g(-1 + 1 + 1) = g(-1) = 1

Exactly like the OR function.

Step 3: Neural Network in the General Case and how to Calculate Weights

In general a Neural Networks can have any number of input and output nodes, where each input node is connected with each output node.

We will later learn about Deep Neural Network – where we can have any number of layers – but for now, let’s only focus Neural Networks with input and output layer.

To calculate weights there are several options.

Gradient Descent

• Calculate the weights (wiki)
• Algorithm for minimizing the loss when training neural networks

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on all data ponits direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Tradoff

• Expensive to calculate for all data points

Stocastic Gradient Descent

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on one data point direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Mini-Batch Gradient Descent

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on one small batch of data ponits direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Step 4: Perceptron

 The perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.

• Only capable of learning linearly separable decision boundary.
• It cannot model the XOR function (we need multi-layer perceptrons (multi-layer neural network))
• It can take multiple inputs and map linearly to one output with an activation function.

Let’s try some example to show it.

import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Sequential
import matplotlib.pyplot as plt

data = np.random.randn(200, 3)
data[:100, :2] += (10, 10)
data[:100, 2] = 0
data[100:, 2] = 1

fig, ax = plt.subplots()
ax.scatter(x=data[:,0], y=data[:,1], c=data[:,2])
plt.show()

This should be simple to validate if we can create a Neural Networks model to separate the two classes.

Step 5: Creating a Neural Network

First let’s create a train and test set.

X = data[:,:2]
y = data[:,2]

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

Then we need to create the model and set batch size and epochs.

• Batch size: a set of N samples.
• Epoch: an arbitrary cutoff, generally defined as “one pass over the entire dataset”.

model = Sequential()
model.add(Dense(1, input_dim=2, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(X_train, y_train, epochs=1000, batch_size=32, verbose=0)

model.evaluate(X_test, y_test)

Which should give 1.000 (100%) accuracy.

This can be visualized by.

y_pred = model.predict(X)
y_pred = np.where(y_pred < .5, 0, 1)

fig, ax = plt.subplots()
ax.scatter(x=X[:,0], y=X[:,1], c=y_pred)
plt.show()

In the video we also show how to visualize the prediction in the different way.

Want to learn more?

In the next lesson you will learn Deep Neural Network.

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).