Category: Machine Learning

What will we cover?

• Understand Deep Neural Network (DNN)
• How algorithms calculate weights in DNN
• Show tools to visually understand what DNN can solve

Step 1: What is Deep Neural Network?

Be sure to read the Artificial Neural Network Guide.

The adjective “deep” in deep learning refers to the use of multiple layers in the network (Wiki).

Usually having two or more hidden layers counts as deep.

Deep learning (also known as deep structured learning) is part of a broader family of machine learning methods based on artificial neural networks with representation learning.

Step 2: How to train and difficulties in training DNN

Training an Artificial Neural Network only relies on finding weights from input to output nodes. In a Deep Neural Network (DNN) become a bit more complex and requires more techniques.

To do that we need backpropagation, which is an algorithm for training Neural Networks with hidden layers (DNN).

• Algorithm
  • Start with a random choice of weights
  • Repeat
    • Calculate error for output layer
    • For each layer – starting with output layer
      • Propagate error back one layer
      • Update weights

A problem you will encounter is overfitting. Which means to fit too close to training data and not generalize well.

That is, you fit the model to the training data, but the model will not predict well on data not coming from your training data.

To deal with that, dropout is a common technique.

• Temporarily remove units – selectat random – from a neural network to prevent over reliance on certain units
• Dropout value of 20%-50%
• Better performance when dropout is used on a larger network
• Dropout at each layer of the network has shown good results.
• Original Paper

Step 3: Play around with it

To learn more about fitting check out the playground at tensorflow.

• https://playground.tensorflow.org/

Ideas to check that

• If you have no hidden layers then you can only fit with straight lines.
• If you add hidden layers you can model the XOR function.

Step 4: A DNN model of XOR

Let’s go crazy and fit an XOR dataset with a DNN model.

import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.models import Sequential
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
data = pd.read_csv('https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/xor.csv')
fig, ax = plt.subplots()
ax.scatter(x=data['x'], y=data['y'], c=data['class id'])
plt.show()

This is the data we want to fit.

Then let’s create it.

Remember to insert the dropout and play around with it.

X_train, X_test, y_train, y_test = train_test_split(data[['x', 'y']], data['class id'], random_state=42)
accuracies = []
for i in range(5):
    tf.random.set_seed(i)
    
    model = Sequential()
    model.add(Dense(6, input_dim=2, activation='relu'))
    # model.add(Dropout(.2))
    model.add(Dense(4, activation='relu'))
    # model.add(Dropout(.2))
    model.add(Dense(1, activation='sigmoid'))
   
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    
    model.fit(X_train, y_train, epochs=100, batch_size=32, verbose=0)
    _, accuracy = model.evaluate(X_test, y_test)
    accuracies.append(accuracy*100)
    
sum(accuracies)/len(accuracies)

Resulting in accuracy of 98%.

Want to learn more?

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

• Understand Neural Networks
• How you can model other machine learning techniques
• Activation functions
• How to make simple OR function
• Different ways to calcualte weights
• What Batch sizes and Epochs are

Step 1: What is Artificial Neural Network

Artificial Neural Network are computing systems inspired by the biological neural networks that constitute animal brains.

Often just called Neural Network.

The first Neural Network is the following simple network.

Where w1 and w2 are weights and the nodes on the left represent input nodes and the node on the right is the output node.

It can also be represented with a function: h(x1, x2) = w0 + w1*x1 + w2*x2

This is a simple calculation, and the goal of the network is to find optimal weights. But we are still missing something. We need an activation function. That is, how to interpret the output.

Here are some possible activation functions.

• Step function: 𝑔(𝑥)=1 if 𝑥≥0, else 0
• Rectified linear unit (ReLU): 𝑔(𝑥)=max(0,𝑥)
• Sigmoid activation function: sigmoid(𝑥)=1/(1+exp(−𝑥))(x)

Step 2: How to model the OR function

We see the weights are one each. Then let’s analyse it with the activation function g, given by Step function.

• x1 = 0 and x2=0 then we have g(-1 + x1 + x2) = g(-1 + 0 + 0) = g(-1) = 0
• x1 = 1 and x2=0 then we have g(-1 + x1 + x2) = g(-1 + 1 + 0) = g(0) = 1
• x1 = 0 and x2=1 then we have g(-1 + x1 + x2) = g(-1 + 0 + 1) = g(0) = 1
• x1 = 1 and x2=1 then we have g(-1 + x1 + x2) = g(-1 + 1 + 1) = g(-1) = 1

Exactly like the OR function.

Step 3: Neural Network in the General Case and how to Calculate Weights

In general a Neural Networks can have any number of input and output nodes, where each input node is connected with each output node.

We will later learn about Deep Neural Network – where we can have any number of layers – but for now, let’s only focus Neural Networks with input and output layer.

To calculate weights there are several options.

Gradient Descent

• Calculate the weights (wiki)
• Algorithm for minimizing the loss when training neural networks

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on all data ponits direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Tradoff

• Expensive to calculate for all data points

Stocastic Gradient Descent

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on one data point direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Mini-Batch Gradient Descent

Pseudo algorithm

• Start with a random choice of weights
• Repeat:
  • Calculate the gradient based on one small batch of data ponits direction that will lead to decreasing loss
  • Update wieghts accorinding to the gradient

Step 4: Perceptron

 The perceptron is an algorithm for supervised learning of binary classifiers. A binary classifier is a function which can decide whether or not an input, represented by a vector of numbers, belongs to some specific class.

• Only capable of learning linearly separable decision boundary.
• It cannot model the XOR function (we need multi-layer perceptrons (multi-layer neural network))
• It can take multiple inputs and map linearly to one output with an activation function.

Let’s try some example to show it.

import numpy as np
import tensorflow as tf
from sklearn.model_selection import train_test_split
from tensorflow.keras.layers import Input, Dense
from tensorflow.keras.models import Sequential
import matplotlib.pyplot as plt
data = np.random.randn(200, 3)
data[:100, :2] += (10, 10)
data[:100, 2] = 0
data[100:, 2] = 1
fig, ax = plt.subplots()
ax.scatter(x=data[:,0], y=data[:,1], c=data[:,2])
plt.show()

This should be simple to validate if we can create a Neural Networks model to separate the two classes.

Step 5: Creating a Neural Network

First let’s create a train and test set.

X = data[:,:2]
y = data[:,2]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)

Then we need to create the model and set batch size and epochs.

• Batch size: a set of N samples.
• Epoch: an arbitrary cutoff, generally defined as “one pass over the entire dataset”.

model = Sequential()
model.add(Dense(1, input_dim=2, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=1000, batch_size=32, verbose=0)
model.evaluate(X_test, y_test)

Which should give 1.000 (100%) accuracy.

This can be visualized by.

y_pred = model.predict(X)
y_pred = np.where(y_pred < .5, 0, 1)
fig, ax = plt.subplots()
ax.scatter(x=X[:,0], y=X[:,1], c=y_pred)
plt.show()

In the video we also show how to visualize the prediction in the different way.

Want to learn more?

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

In this lesson we will learn about Unsupervised learning.

• Understand how Unsupervised Learning is different from Supervised Learning
• How it can organize data without knowledge
• Understand how k-Means Clustering works
• Train a 𝑘-Means Cluster model

Step 1: What is Unsupervised Learning?

Machine Learning is often divided into 3 main categories.

• Supervised: where you tell the algorithm what categories each data item is in. Each data item from the training set is tagged with the right answer.
• Unsupervised: is when the learning algorithm is not told what to do with it and it should make the structure itself.
• Reinforcement: teaches the machine to think for itself based on past action rewards.

Where we see that Unsupervised is one of the main groups.

Unsupervised learning is a type of algorithm that learns patterns from untagged data. The hope is that through mimicry, which is an important mode of learning in people, the machine is forced to build a compact internal representation of its world and then generate imaginative content from it. In contrast to supervised learning where data is tagged by an expert, e.g. as a “ball” or “fish”, unsupervised methods exhibit self-organization that captures patterns as probability densities…

https://en.wikipedia.org/wiki/Unsupervised_learning

Step 2: k-Means Clustering

What is clustering?

Organize a set of objects into groups in such a way that similar objects tend to be in the same group.

What is k-Means Clustering?

Algorithm for clustering data based on repeatedly assigning points to clusters and updating those clusters’ centers.

• First we chose random cluster centroids (hollow point), then assign points to neareast centroid.
• Then we update the centroid to be centered to the points.
• Repeat

This can be repeated a specific number of times or until only small change in centroids positions.

Step 3: Create an Example

Let’s create some random data to demonstrate it.

import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
# Generate some numbers
data = np.random.randn(400,2)
data[:100] += 5, 5
data[100:200] += 10, 10
data[200:300] += 10, 5
data[300:] += 5, 10
fig, ax = plt.subplots()
ax.scatter(x=data[:,0], y=data[:,1])
plt.show()

This shows some random data in 4 clusters.

Then the following code demonstrates how it works. You can change max_iter to be the number iteration – try to do it for 1, 2, 3, etc.

model = KMeans(n_clusters=4, init='random', random_state=42, max_iter=10, n_init=1)
model.fit(data)
y_pred = model.predict(data)
fig, ax = plt.subplots()
ax.scatter(x=data[:,0], y=data[:,1], c=y_pred)
ax.scatter(x=model.cluster_centers_[:,0], y=model.cluster_centers_[:,1], c='r')
plt.show()

Want to learn more?

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

• Understand how Reinforcement Learning works
• Learn about Agent and Environment
• How it iterates and gets rewards based on action
• How to continuously learn new things
• Create own Reinforcement Learning from scratch

Step 1: Reinforcement Learning simply explained

Reinforcement Learning is like training a dog. You and the dog talk different languages. This makes it difficult to explain the dog what you want.

A common way to train a dog is like Reinforcement Learning. When the dog does something good, it get’s a reward. This teaches the dog that you want it to do it.

Said differently, if we relate it to the illustration above. The Agent is the dog. The dog is exposed to an Environment called a state. Based on this Agent (the dog) takes an Action. Based on whether you (the owner) likes the Action, you Reward the Agent.

The goal of the Agent is to get the most Reward. This way it makes it possible for you the owner to get the desired behaviour with adjusting the Reward according to the Actions.

Step 2: Markov Decision Process

The model for decision-making represents States (from the Environment), Actions (from the Agent), and the Rewards.

Written a bit mathematical.

• S is the set of States
• Actions(s) is the set of Actions when in state s
• The transition model is P(s´, s, a)
• The Reward function R(s, a, s’)

Step 3: Q-Learning

Q-learning is a model-free reinforcement learning algorithm to learn the value of an action in a particular state. It does not require a model of the environment (hence “model-free”), and it can handle problems with stochastic transitions and rewards without requiring adaptations. (wiki)

This can be modeled by a learning function Q(s, a), which estimates the value of performing action a when in state s.

It works as follows

• Start with Q(s, a) = 0 for all s, a
• Update Q when we take an action

𝑄(𝑠,𝑎)=𝑄(𝑠,𝑎)+𝛼(Q(s,a)=Q(s,a)+α(reward+𝛾max(𝑠′,𝑎′)−𝑄(𝑠,𝑎))=(1−𝛼)𝑄(𝑠,𝑎)+𝛼(+γmax(s′,a′)−Q(s,a))=(1−α)Q(s,a)+α(reward+𝛾max(𝑠′,𝑎′))+γmax(s′,a′))

The ϵ-Greedy Decision Making

The idea behind it is to either explore or exploit

• With probability ϵ take a random move
• Otherwise, take action 𝑎a with maximum 𝑄(𝑠,𝑎)

Let’s demonstrate it with code.

Step 3: Code Example

Assume we have the following Environment

• You start at a random point.
• You can either move left or right.
• You loose if you hit a red box
• You win if you hit the green box

Quite simple, but how can you program an Agent using Reinforcement Learning? And how can you do it from scratch.

The great way is to use an object representing the field (environment).

To implement it all there are some background resources if needed.

Programming Notes:

• Libraries used
  • numpy – scientific computing with Python (Lecture on NumPy)
  • random – pseudo-random generators
• Functionality and concepts used
  • Object-Oriented Programming (OOP): Lecture on Object Oriented Programming

What if there are more states?

• Reinforcement Learning from Scratch

import numpy as np
import random
class Field:
    def __init__(self):
        self.states = [-1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0]
        self.state = random.randrange(0, len(self.states))
        
    def done(self):
        if self.states[self.state] != 0:
            return True
        else:
            return False
        
    # action: 0 => left
    # action: 1 => right
    def get_possible_actions(self):
        actions = [0, 1]
        if self.state == 0:
            actions.remove(0)
        if self.state == len(self.states) - 1:
            actions.remove(1)
        return actions
    def update_next_state(self, action):
        if action == 0:
            if self.state == 0:
                return self.state, -10
            self.state -= 1
        if action == 1:
            if self.state == len(self.states) - 1:
                return self.state, -10
            self.state += 1
        
        reward = self.states[self.state]
        return self.state, reward
field = Field()
q_table = np.zeros((len(field.states), 2))
alpha = .5
epsilon = .5
gamma = .5
for _ in range(10000):
    field = Field()
    while not field.done():
        actions = field.get_possible_actions()
        if random.uniform(0, 1) < epsilon:
            action = random.choice(actions)
        else:
            action = np.argmax(q_table[field.state])
            
        cur_state = field.state
        next_state, reward = field.update_next_state(action)
        
        q_table[cur_state, action] = (1 - alpha)*q_table[cur_state, action] + alpha*(reward + gamma*np.max(q_table[next_state]))

Step 4: A more complex Example

Check out the video to see a More complex example.

Want to learn more?

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

In this tutorial we will cover the following.

• Learn about the problem of seperation
• The idea to maximize the distance
• Work with examples to demonstrate the issue
• Use the Support Vector Machine (SVM) model on data.
• Explore the result of SVM on classification data.

Step 1: What is Maximum Margin Separator?

Boundary that maximizes the distances between any of the data points (Wiki)

The problem can be illustrated as follows.

Looking at the image to the left we separate all the red dots from the blue dots. This separation is perfect. But we know that this line might not be ideal if more dots are coming. Imagine another blue dot is added (right image).

Could we have chosen the a better line of separation?

As you see above – there is a better line to chose from the start. The one that is the longest from all points.

Step 2: What is Support Vector Machine (SVM)?

The Support Vector Machine solves the separation problem stated above.

In machine learning, support-vector machines (SVMs, also support-vector networks) are supervised learning models with associated learning algorithms that analyze data for classification and regression analysis (source: wiki).

But what do we use SVM for?

• Classify data.
• Face detection
• Classification of images
• Handwriting recognition
• Inverse geosounding problem
• Facial expression
• Text classification

Among things.

But basically, it is all about classifying data. That is, given a collection of data and a set of categories for this data, the model helps classifies data into the correct categories.

Example of facial expression you might have categories of happy, sad, surprised, and angry. Then given an image of a face it can categorize it into one of the categories.

How does it do it?

Well, you need training data with correct labels.

In this tutorial we will make a gentle introduction to classification based on simple data.

Step 3: Gender classification based on height and heir length

Let’s consider the a list of measured height and hair lengths with the given gender.

import pandas as pd
url = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/gender.csv'
data = pd.read_csv(url)
print(data.head())

Resulting in this.

   Height  Hair length Gender
0     151           99      F
1     193            8      M
2     150          123      F
3     176            0      M
4     188           11      M

Step 4: Visualize the data

You can visualize the result as follows.

import pandas as pd
import matplotlib.pyplot as plt
url = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/gender.csv'
data = pd.read_csv(url)
data['Class'] = data['Gender'].apply(lambda x: 'r' if x == 'F' else 'b')
data = data.iloc[:25]
fig, ax = plt.subplots()
ax.scatter(x=data['Height'], y=data['Hair length'], c=data['Class'])
plt.show()

Where we only keep the first 25 points to simplify the plot.

Step 5: Creating a SVC model

We will use Sklearns SVC (Support Vector Classification (docs)) model to fit the data.

import pandas as pd
import numpy as np
from sklearn import svm
url = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/gender.csv'
data = pd.read_csv(url)
data['Class'] = data['Gender'].apply(lambda x: 'r' if x == 'F' else 'b')
X = data[['Height', 'Hair length']]
y = data['Gender']
y = np.array([0 if gender == 'M' else 1 for gender in y])
clf = svm.SVC(kernel='linear')
clf.fit(X, y)

Step 6: Visualize the model

We create a “box” to color the model prediction.

import pandas as pd
import numpy as np
from sklearn import svm
import matplotlib.pyplot as plt
url = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/gender.csv'
data = pd.read_csv(url)
data['Class'] = data['Gender'].apply(lambda x: 'r' if x == 'F' else 'b')
X = data[['Height', 'Hair length']]
y = data['Gender']
y = np.array([0 if gender == 'M' else 1 for gender in y])
clf = svm.SVC(kernel='linear')
clf.fit(X, y)
X_test = np.random.rand(10000, 2)
X_test = X_test*(70, 140) + (140, 0)
y_pred = clf.predict(X_test)
fig, ax = plt.subplots()
ax.scatter(x=X_test[:,0], y=X_test[:,1], c=y_pred, alpha=.25)
y_color = ['r' if value == 0 else 'b' for value in y]
ax.scatter(x=X['Height'], y=X['Hair length'], c=y_color)
plt.show()

Resulting in.

Want to learn more?

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

This tutorial will explain what Machine Learning is by comparing it to classical programming. Then how Machine Learning works and the three main categories of Machine Learning: Supervised, Unsupervised, and Reinforcement Learning.

Finally, we will explore a Supervised Machine Learning model called k-Nearest-Neighbors (KNN) classifier to get an understanding through practical application.

Goal of Lesson

• Understand the difference between Classical Computing and Machine Learning
• Know the 3 main categories of Machine Learning
• Dive into Supervised Learning
• Classification with 𝑘-Nearest-Neighbors Classifier (KNN)
• How to classify data
• What are the challenges with cleaning data
• Create a project on real data with 𝑘-Nearest-Neighbor Classifier

Step 1: What is Machine Learning?

• In the classical computing model every thing is programmed into the algorithms. 
  • This has the limitation that all decision logic need to be understood before usage. 
  • And if things change, we need to modify the program.
• With the modern computing model (Machine Learning) this paradigm is changes. 
  • We feed the algorithms (models) with data.
  • Based on that data, the algorithms (models) make decisions in the program.

Imagine you needed to teach your child how to bike a bicycle.

In the classical computing sense, you will instruct your child how to use a specific muscle in all cases. That is, if you lose balance to the right, then activate the your third muscle in your right leg. You need instructions for all muscles in all situations.

That is a lot of instructions and chances are, you forget specific situations.

Machine Learning feeds the child data, that is it will fall, it will fail – but eventually, it will figure it out itself, without instructions on how to use the specific muscles in the body.

Well, that is actually how most learn how to bike.

Step 2: How Machine Learning Works

On a high level, Machine Learning is divided into two phases.

• Learning phase: Where the algorithm (model) learns in a training environment. Like, when you support your child learning to ride the bike, like catching the child while falling not to hit too hard.
• Prediction phase: Where the algorithm (model) is applied on real data. This is when the child can bike on its own.

The Learning Phase is often divided into a few steps.

• Get Data: Identify relevant data for the problem you want to solve. This data set should represent the type of data that the Machine Learn model will use to predict from in Phase 2 (predction).
• Pre-processing: This step is about cleaning up data. While the Machine Learning is awesome, it cannot figure out what good data looks like. You need to do the cleaning as well as transforming data into a desired format.
• Train model: This is where the magic happens, the learning step (Train model). There are three main paradigms in machine learning.
  • Supervised: where you tell the algorithm what categories each data item is in. Each data item from the training set is tagged with the right answer.
  • Unsupervised: is when the learning algorithm is not told what to do with it and it should make the structure itself.
  • Reinforcement: teaches the machine to think for itself based on past action rewards.
• Test model: Finally, the testing is done to see if the model is good. The training data was divided into a test set and training set. The test set is used to see if the model can predict from it. If not, a new model might be necessary.

The Prediction Phase can be illustrated as follows.

Step 3: Supervised Learning explained with Example

Supervised learning can be be explained as follows.

Given a dataset of input-output pairs, learn a function to map inputs to outputs.

There are different tasks – but we start to focus on Classification. Where supervised classification is the task of learning a function mapping an input point to a discrete category.

Now the best way to understand new things is to relate it to something we already understand.

Consider the following data.

Given the Humidity and Pressure for a given day can we predict if it will rain or not.

How will a Supervised Classification algorithm work?

Learning Phase: Given a set of historical data to train the model – like the data above, given rows of Humidity and Pressure and the label Rain or No Rain. Let the algorithm work with the data and figure it out.

Note: we leave out pre-processing and testing the model here.

Prediction Phase: Let the algorithm get new data – like in the morning you read Humidity and Pressure and let the algorithm predict if will rain or not that given day.

Written mathematically, it is the task to find a function 𝑓 as follows.

Ideally: 𝑓(ℎ𝑢𝑚𝑖𝑑𝑖𝑡𝑦,𝑝𝑟𝑒𝑠𝑠𝑢𝑟𝑒)

Examples:

• 𝑓(93,999.7) = Rain
• 𝑓(49,1015.5) = No Rain
• 𝑓(79,1031.1) = No Rain

Goal: Approximate the function 𝑓 – the approximation function is often denoted ℎ

Step 4: Visualize the data we want to fit

We will use pandas to work with data, which is a fast, powerful, flexible and easy to use open source data analysis and manipulation tool, built on top of the Python programming language.

The data we want to work with can be downloaded from a here and stored locally. Or you can access it directly as follows.

import pandas as pd
file_dest = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/weather.csv'
data = pd.read_csv(file_dest, parse_dates=True, index_col=0)

First lets’s visualize the data we want to work with.

import matplotlib.pyplot as plt
import pandas as pd
file_dest = 'https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/weather.csv'
data = pd.read_csv(file_dest, parse_dates=True, index_col=0)
dataset = data[['Humidity3pm', 'Pressure3pm', 'RainTomorrow']]
fig, ax = plt.subplots()
dataset[dataset['RainTomorrow'] == 'No'].plot.scatter(x='Humidity3pm', y='Pressure3pm', c='b', alpha=.25, ax=ax)
dataset[dataset['RainTomorrow'] == 'Yes'].plot.scatter(x='Humidity3pm', y='Pressure3pm', c='r', alpha=.25, ax=ax)
plt.show()

Resulting in.

The goal is to make a mode which can predict Blue or Red dots.

Step 5: The k-Nearest-Neighbors Classifier

Given an input, choose the class of nearest datapoint.

𝑘-Nearest-Neighbors Classification

• Given an input, choose the most common class out of the 𝑘 nearest data points

Let’s try to implement a model. We will use sklearn for that.

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score
dataset_clean = dataset.dropna()
X = dataset_clean[['Humidity3pm', 'Pressure3pm']]
y = dataset_clean['RainTomorrow']
y = np.array([0 if value == 'No' else 1 for value in y])
neigh = KNeighborsClassifier()
neigh.fit(X_train, y_train)
y_pred = neigh.predict(X_test)
accuracy_score(y_test, y_pred)

This actually covers what you need. Make sure to have the dataset data from the previous step available here.

To visualize the code you can run the following.

fig, ax = plt.subplots()
y_map = neigh.predict(X_map)
ax.scatter(x=X_map[:,0], y=X_map[:,1], c=y_map, alpha=.25)
plt.show()

Want more help?

Check out this video explaining all steps in more depth. Also, it includes a guideline for making your first project with Machine Learning along with a solution for it.

This is part of a FREE 10h Machine Learning course with Python.

• 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
• 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
• 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).

What will we cover?

We will learn what Reinforcement Learning is and how it works. Then by using Object-Oriented Programming technics (more about Object-Oriented Programming), we implement a Reinforcement Model to solve the problem of figuring out where to pick up and drop of item on a field.

Step 1: What is Reinforcement Learning?

Reinforcement Learning is one of the 3 main categories of Machine Learning (get started with Machine Learning here) and is concerned with how intelligent agents ought to take actions in an environment in order to maximize the notion of cumulative reward.

How Reinforcement Learning works

Reinforcement Learning teaches the machine to think for itself based on past action rewards.

• Basically, the Reinforcement Learning algorithm tries to predict actions that gives rewards and avoids punishment.
• It is like training a dog. You and the dog do not talk the same language, but the dogs learns how to act based on rewards (and punishment, which I do not advise or advocate).
• Hence, if a dog is rewarded for a certain action in a given situation, then next time it is exposed to a similar situation it will act the same.
• Translate that to Reinforcement Learning.
  • The agent is the dog that is exposed to the environment.
  • Then the agent encounters a state.
  • The agent performs an action to transition to a new state.
  • Then after the transition the agent receives a reward or penalty (punishment).
  • This forms a policy to create a strategy to choose actions in a given state.

What algorithms are used for Reinforcement Learning?

• The most common algorithm for Reinforcement Learning are.
  • Q-Learning: is a model-free reinforcement learning algorithm to learn a policy telling an agent what action to take under what circumstances.
  • Temporal Difference: refers to a class of model-free reinforcement learning methods which learn by bootstrapping from the current estimate of the value function.
  • Deep Adversarial Network: is a technique employed in the field of machine learning which attempts to fool models through malicious input.
• We will focus on the Q-learning algorithm as it is easy to understand as well as powerful.

How does the Q-learning algorithm work?

• As already noted, I just love this algorithm. It is “easy” to understand and powerful as you will see.

• The Q-Learning algorithm has a Q-table (a Matrix of dimension state x actions – don’t worry if you do not understand what a Matrix is, you will not need the mathematical aspects of it – it is just an indexed “container” with numbers).
• The agent (or Q-Learning algorithm) will be in a state.
• Then in each iteration the agent needs take an action.
• The agent will continuously update the reward in the Q-table.
• The learning can come from either exploiting or exploring.
• This translates into the following pseudo algorithm for the Q-Learning.
• The agent is in a given stateºº and needs to choose an action.

Algorithm

• Initialise the Q-table to all zeros
• Iterate
  • Agent is in state state.
  • With probability epsilon choose to explore, else exploit.
    • If explore, then choose a random action.
    • If exploit, then choose the best action based on the current Q-table.
  • Update the Q-table from the new reward to the previous state.
  • Q[state, action] = (1 – alpha) * Q[state, action] + alpha * (reward + gamma * max(Q[new_state]) — Q[state, action])

Variables

As you can se, we have introduced the following variables.

• epsilon: the probability to take a random action, which is done to explore new territory.
• alpha: is the learning rate that the algorithm should make in each iteration and should be in the interval from 0 to 1.
• gamma: is the discount factor used to balance the immediate and future reward. This value is usually between 0.8 and 0.99
• reward: is the feedback on the action and can be any number. Negative is penalty (or punishment) and positive is a reward.

Step 2: The problem we want to solve

Here we have a description of task we want to solve.

• To keep it simple, we create a field of size 10×10 positions. In that field there is an item that needs to be picked up and moved to a drop-off point.

• At each position there are 6 different actions that can be taken.
  • Action 0: Go South if on field.
  • Action 1: Go North if on field.
  • Action 2: Go East if on field (Please notice, I mixed up East and West (East is Left here)).
  • Action 3: Go West if on field (Please notice, I mixed up East and West (West is right here)).
  • Action 4: Pickup item (it can try even if it is not there)
  • Action 5: Drop-off item (it can try even if it does not have it)
• Based on these actions we will make a reward system.
  • If the agent tries to go off the field, punish with -10 in reward.
  • If the agent makes a (legal) move, punish with -1 in reward, as we do not want to encourage endless walking around.
  • If the agent tries to pick up item, but it is not there or it has it already, punish with -10 in reward.
  • If the agent picks up the item correct place, reward with 20.
  • If agent tries to drop-off item in wrong place or does not have the item, punish with -10 in reward.
  • If the agent drops-off item in correct place, reward with 20.
• That translates into the following code. I prefer to implement this code, as I think the standard libraries that provide similar frameworks hide some important details. As an example, and shown later, how do you map this into a state in the Q-table?

Step 3: Implementing the field

First we need a way to represent the field, representing the environment our model lives in. This is defined in Step 2 and could be implemented as follows.

class Field:
    def __init__(self, size, item_pickup, item_drop_off, start_position):
        self.size = size
        self.item_pickup = item_pickup
        self.item_drop_off = item_drop_off
        self.position = start_position
        self.item_in_car = False
        
    def get_number_of_states(self):
        return self.size*self.size*self.size*self.size*2
    
    def get_state(self):
        state = self.position[0]*self.size*self.size*self.size*2
        state = state + self.position[1]*self.size*self.size*2
        state = state + self.item_pickup[0]*self.size*2
        state = state + self.item_pickup[1]*2
        if self.item_in_car:
            state = state + 1
        return state
        
    def make_action(self, action):
        (x, y) = self.position
        if action == 0:  # Go South
            if y == self.size - 1:
                return -10, False
            else:
                self.position = (x, y + 1)
                return -1, False
        elif action == 1:  # Go North
            if y == 0:
                return -10, False
            else:
                self.position = (x, y - 1)
                return -1, False
        elif action == 2:  # Go East
            if x == 0:
                return -10, False
            else:
                self.position = (x - 1, y)
                return -1, False
        elif action == 3:  # Go West
            if x == self.size - 1:
                return -10, False
            else:
                self.position = (x + 1, y)
                return -1, False
        elif action == 4:  # Pickup item
            if self.item_in_car:
                return -10, False
            elif self.item_pickup != (x, y):
                return -10, False
            else:
                self.item_in_car = True
                return 20, False
        elif action == 5:  # Drop off item
            if not self.item_in_car:
                return -10, False
            elif self.item_drop_off != (x, y):
                self.item_pickup = (x, y)
                self.item_in_car = False
                return -10, False
            else:
                return 20, True

Step 4: A Naive approach to solve it (NON-Machine Learning)

A naive approach would to just take random actions and hope for the best. This is obviously not optimal, but nice to have as a base line to compare with.

def naive_solution():
    size = 10
    item_start = (0, 0)
    item_drop_off = (9, 9)
    start_position = (0, 9)
    
    field = Field(size, item_start, item_drop_off, start_position)
    done = False
    steps = 0
    
    while not done:
        action = random.randint(0, 5)
        reward, done = field.make_action(action)
        steps = steps + 1
    
    return steps

To make an estimate on how many steps it takes you can run this code.

runs = [naive_solution() for _ in range(100)]
print(sum(runs)/len(runs))

Where we use List Comprehension (learn more about list comprehension). This gave 143579.21. Notice, you most likely will get something different, as there is a high level of randomness involved.

Step 5: Implementing our Reinforcement Learning Model

Here we give the algorithm for what we need to implement.

Algorithm

• Initialise the Q-table to all zeros
• Iterate
  • Agent is in state state.
  • With probability epsilon choose to explore, else exploit.
    • If explore, then choose a random action.
    • If exploit, then choose the best action based on the current Q-table.
  • Update the Q-table from the new reward to the previous state.
  • Q[state, action] = (1 – alpha) * Q[state, action] + alpha * (reward + gamma * max(Q[new_state]) — Q[state, action])

Then we end up with the following code to train our Q-table.

size = 10
item_start = (0, 0)
item_drop_off = (9, 9)
start_position = (0, 9)
field = Field(size, item_start, item_drop_off, start_position)
number_of_states = field.get_number_of_states()
number_of_actions = 6
q_table = np.zeros((number_of_states, number_of_actions))
epsilon = 0.1
alpha = 0.1
gamma = 0.6
for _ in range(10000):
    field = Field(size, item_start, item_drop_off, start_position)
    done = False
    
    while not done:
        state = field.get_state()
        if random.uniform(0, 1) < epsilon:
            action = random.randint(0, 5)
        else:
            action = np.argmax(q_table[state])
            
        reward, done = field.make_action(action)
        # Q[state, action] = (1 – alpha) * Q[state, action] + alpha * (reward + gamma * max(Q[new_state]) — Q[state, action])
        
        new_state = field.get_state()
        new_state_max = np.max(q_table[new_state])
        
        q_table[state, action] = (1 - alpha)*q_table[state, action] + alpha*(reward + gamma*new_state_max - q_table[state, action])

Then we can apply our model as follows.

def reinforcement_learning():
    epsilon = 0.1
    alpha = 0.1
    gamma = 0.6
    
    field = Field(size, item_start, item_drop_off, start_position)
    done = False
    steps = 0
    
    while not done:
        state = field.get_state()
        if random.uniform(0, 1) < epsilon:
            action = random.randint(0, 5)
        else:
            action = np.argmax(q_table[state])
            
        reward, done = field.make_action(action)
        # Q[state, action] = (1 – alpha) * Q[state, action] + alpha * (reward + gamma * max(Q[new_state]) — Q[state, action])
        
        new_state = field.get_state()
        new_state_max = np.max(q_table[new_state])
        
        q_table[state, action] = (1 - alpha)*q_table[state, action] + alpha*(reward + gamma*new_state_max - q_table[state, action])
        
        steps = steps + 1
    
    return steps

And evaluate it as follows.

runs_rl = [reinforcement_learning() for _ in range(100)]
print(sum(runs_rl)/len(runs_rl))

This resulted in 47.45. Again, you should get something different.

But a comparison to taking random moves (Step 4) it is a factor 3000 better.

Want more?

Want to learn more Python, then this is part of a 8 hours FREE video course with full explanations, projects on each levels, and guided solutions.

The course is structured with the following resources to improve your learning experience.

• 17 video lessons teaching you everything you need to know to get started with Python.
• 34 Jupyter Notebooks with lesson code and projects.
• A FREE 70+ pages eBook with all the learnings from the lessons.

See the full FREE course page here.

If you instead want to learn more about Machine Learning. Do not worry.

Then check out my Machine Learning with Python course.

• 15 video lessons teaching you all aspects of Machine Learning
• 30 JuPyter Notebooks with lesson code and projects
• 10 hours FREE video content to support your learning journey.

Go to the course page for details.