What will we cover?
The goal is to learn about Supervised Learning and explore how to use it for classification.
This includes learning
- What is Supervised Learning
- Understand the classification problem
- What is the Perceptron classifier
- How to use the Perceptron classifier as a linear classifier
Step 1: What is Supervised Learning?
Said differently, if you have some items you need to classify, it could be books you want to put in categories, say fiction, non-fiction, etc.
Then if you were given a pile of books with the right categories given to them, how can you make a function (the machine learning model), which on other books without labels can guess the right category.
Supervised learning simply means, that in the learning phase, the algorithm (the one creating the model) is given examples with correct labels.
Notice, that supervised learning does not only restrict to classification problems, but it could predict anything.
If you are new to Machine Learning, I advise you start with this tutorial.
Step 2: What is the classification problem?
The classification problem is a supervised learning task of getting a function mapping an input point to a discrete category.
There is binary classification and multiclass classification, where the binary maps into two classes, and the multi classmaps into 3 or more classes.
I find it easiest to understand with examples.
Assume we want to predict if will rain or not rain tomorrow. This is a binary classification problem, because we map into two classes: rain or no rain.
To train the model we need already labelled historic data.
Hence, the task is given rows of historic data with correct labels, train a machine learning model (a Linear Classifier in this case) with this data. Then after that, see how good it can predict future data (without the right class label).
Step 3: Linear Classification explained mathematically and visually
Some like the math behind an algorithm. If you are not one of them, focus on the visual part – it will give you the understanding you need.
The task of Supervised Learning mathematically can be explained simply with the example data above to find a function f(humidity, pressure) to predict rain or no rain.
- f(93, 000.7) = rain
- f(49, 1015.5) = no rain
- f(79, 1031.1) = no rain
The goal of Supervised Learning is to approximate the function f – the approximation function is often denoted h.
Why not identify f precisely? Well, because it is not ideal, as this would be an overfitted function, that would predict the historic data 100% accurate, but would fail to predict future values very well.
As we work with Linear Classifiers, we want the function to be linear.
That is, we want the approximation function h, to be on the form.
- x_1: Humidity
- x_2: Pressure
- h(x_1, x_2) = w_0 + w_1*x_1 + w_2*x_2
Hence, the goal is to optimize values w_0, w_1, w_2, to find the best classifier.
What does all this math mean?
Well, that it is a linear classifier that makes decisions based on the value of a linear combination of the characteristics.
The above diagram shows how it would classify with a line whether it will predict rain or not. On the left side, this is the data classified from historic data, and the line shows an optimized line done by the machine learning algorithm.
On the right side, we have a new input data (without label), then with this line, it would classify it as rain (assuming blue means rain).
Step 4: What is the Perceptron Classifier?
The Perceptron Classifier is a linear algorithm that can be applied to binary classification.
It learns iteratively by adding new knowledge to an already existing line.
The learning rate is given by alpha, and the learning rule is as follows (don’t worry if you don’t understand it – it is not important).
- Given data point x and y update each weight according to this.
- w_i = w_i + alpha*(y – h_w(x)) X x_i
The rule can also be stated as follows.
- w_i = w_i + alpha(actual value – estimated value) X x_i
Said in words, it adjusted the values according to the actual values. Every time a new values comes, it adjusts the weights to fit better accordingly.
Given the line after it has been adjusted to all the training data – then it is ready to predict.
Let’s try this on real data.
Step 5: Get the Weather data we will use to train a Perceptron model with
You can get all the code in a Jupyter Notebook with the csv file here.
First let’s just import all the libraries used.
import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.linear_model import Perceptron import matplotlib.pyplot as plt
Notice that in the Notebook we have an added line %matplotlib inline, which you should add if you run in a Notebook. The code here will be aligned with PyCharm or a similar IDE.
Then let’s read the data.
data = pd.read_csv('files/weather.csv', parse_dates=True, index_col=0) print(data.head())
If you want to read the data directly from GitHub and not download the weather.csv file, you can do that as follows.
data = pd.read_csv('https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/weather.csv', parse_dates=True, index_col=0) print(data.head())
This will result in an output similar to this.
MinTemp MaxTemp Rainfall ... RainToday RISK_MM RainTomorrow Date ... 2008-02-01 19.5 22.4 15.6 ... Yes 6.0 Yes 2008-02-02 19.5 25.6 6.0 ... Yes 6.6 Yes 2008-02-03 21.6 24.5 6.6 ... Yes 18.8 Yes 2008-02-04 20.2 22.8 18.8 ... Yes 77.4 Yes 2008-02-05 19.7 25.7 77.4 ... Yes 1.6 Yes
Step 6: Select features and Clean the Weather data
We want to investigate the data and figure out how much missing data there.
A great way to do that is to use isnull().
This results in the following output.
MinTemp 3 MaxTemp 2 Rainfall 6 Evaporation 51 Sunshine 16 WindGustDir 1036 WindGustSpeed 1036 WindDir9am 56 WindDir3pm 33 WindSpeed9am 26 WindSpeed3pm 25 Humidity9am 14 Humidity3pm 13 Pressure9am 20 Pressure3pm 19 Cloud9am 566 Cloud3pm 561 Temp9am 4 Temp3pm 4 RainToday 6 RISK_MM 0 RainTomorrow 0 dtype: int64
This shows how many rows in each column has null value (missing values). We want to work only with a two features (columns), to keep our classification simple. Obviously, we need to keep RainTomorrow, as that is carrying the label of the class.
We select the features we want and drop the rows with null-values as follows.
dataset = data[['Humidity3pm', 'Pressure3pm', 'RainTomorrow']].dropna()
Step 7: Split into trading and test data
The next step we need to do is to split the dataset into a features and labels.
But we also want to rename the labels from No and Yes to be numeric.
X = dataset[['Humidity3pm', 'Pressure3pm']] y = dataset['RainTomorrow'] y = np.array([0 if value == 'No' else 1 for value in y])
Then we do the splitting as follows, where we but a random_state in order to be able to reproduce. This is often a great idea, if you randomness and encounter a problem, then you can reproduce it.
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
This has divided the features into a train and test set (X_train, X_test), and the labels into a train and test (y_train, y_test) dataset.
Step 8: Train the Perceptron model and measure accuracy
Finally we want to create the model, fit it (train it), predict on the training data, and print the accuracy score.
clf = Perceptron(random_state=0) clf.fit(X_train, y_train) y_pred = clf.predict(X_test) print(accuracy_score(y_test, y_pred))
This gives an accuracy of 0.773 or 77,3% accuracy.
Is that good?
Well what if it rains 22.7% of the time? And the model always predicts No rain?
Well, then it is correct 77.3% of the time.
Let’s just check for that.
Well, it is not raining in 74.1% of the time.
print(sum(y == 0)/len(y))
Is that a good model? Well, I find the binary classifiers a bit tricky because of this problem. The best way to get an idea is to visualize it.
Step 9: Visualize the model predictions
To visualize the data we can do the following.
fig, ax = plt.subplots() X_data = X.to_numpy() y_all = clf.predict(X_data) ax.scatter(x=X_data[:,0], y=X_data[:,1], c=y_all, alpha=.25) plt.show()
This results in the following output.
Finally, let’s visualize the actual data to compare.
ax.scatter(x=X_data[:,0], y=X_data[:,1], c=y, alpha=.25) plt.show()
Here is the full code.
import pandas as pd import numpy as np from sklearn.model_selection import train_test_split from sklearn.metrics import accuracy_score from sklearn.linear_model import Perceptron import matplotlib.pyplot as plt data = pd.read_csv('https://raw.githubusercontent.com/LearnPythonWithRune/MachineLearningWithPython/main/files/weather.csv', parse_dates=True, index_col=0) print(data.head()) print(data.isnull().sum()) dataset = data[['Humidity3pm', 'Pressure3pm', 'RainTomorrow']].dropna() X = dataset[['Humidity3pm', 'Pressure3pm']] y = dataset['RainTomorrow'] y = np.array([0 if value == 'No' else 1 for value in y]) X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0) clf = Perceptron(random_state=0) clf.fit(X_train, y_train) y_pred = clf.predict(X_test) print(accuracy_score(y_test, y_pred)) print(sum(y == 0)/len(y)) fig, ax = plt.subplots() X_data = X.to_numpy() y_all = clf.predict(X_data) ax.scatter(x=X_data[:,0], y=X_data[:,1], c=y_all, alpha=.25) plt.show() fig, ax = plt.subplots() ax.scatter(x=X_data[:,0], y=X_data[:,1], c=y, alpha=.25) plt.show()
Want to learn more?
This is part of a FREE 10h Machine Learning course with Python.
- 15 video lessons – which explain Machine Learning concepts, demonstrate models on real data, introduce projects and show a solution (YouTube playlist).
- 30 JuPyter Notebooks – with the full code and explanation from the lectures and projects (GitHub).
- 15 projects – with step guides to help you structure your solutions and solution explained in the end of video lessons (GitHub).