Simple Machine Learning Trading Bot in Python – Evaluating how it Performs

What will we cover in this tutorial

  • To create a machine learning trading bot in Python
  • How to build a simple Reinforcement Learning Trading bot.
  • The idea behind the Reinforcement Learning trading bot
  • Evaluate how the trading bot performs

Machine Learning and Trading?

First thing first. Machine Learning trading bot? Machine Learning can be used for various things in regards to trading.

Well, good to set our expectations. This tutorial is also experimental and does not claim to make a bullet-proof Machine Learning Trading bot that will make you rich. I strongly advice you not to use it for automated trading.

This tutorial is only intended to test and learn about how a Reinforcement Learning strategy can be used to build a Machine Learning Trading Bot.

Step 1: The idea behind the Reinforcement Learning strategy

I wanted to test how a Reinforcement Learning algorithm would do in the market.

First let us understand what Reinforcement Learning is. Reinforcement learning teaches the machine to think for itself based on past action rewards.

Reinforcement Learning (in Machine Learning) teaches the machine to think based on past action rewards.
Reinforcement Learning (in Machine Learning) teaches the machine to think based on past action rewards.

It is like training a dog. You and the dog do not talk the same language, but the dogs learns how to act based on rewards (and punishment, which I do not advise or advocate). 

Hence, if a dog is rewarded for a certain action in a given situation, then next time it is exposed to a similar situation it will act the same. 

Translate that to Reinforcement Learning. 

  • The agent is the dog that is exposed to the environment
  • Then the agent encounters a state
  • The agent performs an action to transition from that state to a new state
  • Then after the transition the agent receives a reward or penalty(punishment).
  • This forms a policy to create a strategy to choose actions in a given state

That turns out to fit well with trading, or potentially? That is what I want to investigate.

Step 2: The idea behind how to use Reinforcement Learning in Trading

The environment in trading could be translated to rewards and penalties (punishment). You win or loose on the stock market, right?

But we also want to simplify the environment for the bot, not to make it too complex. Hence, in this experiment, the bot is only knows 1 stock and has to decide to buy, keep or sell.

Said differently.

  • The trading bot (agent) is exposed to the stock history (environment).
  • Then the trading bot (agent) encounters the new stock price (state).
  • The trading bot (agent) then performs a choice to keep, sell or buy (action), which brings it to a new state.
  • Then the trading bot (agent) will receives a reward based on the value difference from day to day.

The reward will often first be encountered after some time, hence, the feedback from steps after should be set high. Or at least, that is my expectation.

Step 3: Understand Q-learning as the Reinforcement Learning model

The Q-learning model is easy to understand and has potential to be very powerful. Of course, it is not better than the design of it. But before we can design it, we need to understand the mechanism behind it.

Q-Learning algorithm (Reinforcement / Machine Learning) - exploit or explore - Update Q-table
Q-Learning algorithm (Reinforcement / Machine Learning) – exploit or explore – Update Q-table

The Q-Learning algorithm has a Q-table (a Matrix of dimension state x actions – don’t worry if you do not understand what a Matrix is, you will not need the mathematical aspects of it – it is just an indexed “container” with numbers).

  • The agent (or Q-Learning algorithm) will be in a state.
  • Then in each iteration the agent needs take an action.
  • The agent will continuously update the reward in the Q-table.
  • The learning can come from either exploiting or exploring.

This translates into the following pseudo algorithm for the Q-Learning. 

The agent is in a given state and needs to choose an action.

  • Initialise the Q-table to all zeros
  • Iterate:
    • Agent is in state state.
    • With probability epsilon choose to explore, else exploit.
      • If explore, then choose a random action.
      • If exploit, then choose the best action based on the current Q-table.
    • Update the Q-table from the new reward to the previous state.
      • Q[stateaction] = (1 – alpha) * Q[stateaction] + alpha * (rewardgamma * max(Q[new_state]) — Q[state, action])

As you can se, we have introduced the following variables.

  • epsilon: the probability to take a random action, which is done to explore new territory.
  • alpha: is the learning rate that the algorithm should make in each iteration and should be in the interval from 0 to 1.
  • gamma: is the discount factor used to balance the immediate and future reward. This value is usually between 0.8 and 0.99
  • reward: is the feedback on the action and can be any number. Negative is penalty (or punishment) and positive is a reward.

Step 4: The choices we need to take

Based on that, we need to see how the algorithm should map the stock information to a state. We want the model to be fairly simple and not have too many states, as it will take long time to populate it with data.

There are many parameters to choose from here. As we do not want to tell the algorithm what to do, we still need to feed it what what we find as relevant data.

In this case it was the following.

  • Volatility of the share.
  • The percentage change of the daily short mean (average over last 20 days).
  • Then the percentage of the daily long mean (average over the last 100 days).
  • The daily long mean, which is the average over the last 100 days.
  • The volume of the sales that day.

These values need to be calculated for the share we use. That can be done by the following code.

import pandas_datareader as pdr
import numpy as np


VALUE = 'Adj Close'
ID = 'id'
NAME = 'name'
DATA = 'data'


def get_data(name, years_ago):
    start = dt.datetime.now() - relativedelta(years=years_ago)
    end = dt.datetime.now()
    df = pdr.get_data_yahoo(name, start, end)
    return df


def process():
    stock = {ID: stock, NAME: 'AAPL'}

    stock[DATA] = get_data(stock[ID], 20)

# Updatea it will all values
    stock[DATA]['Short Mean'] = stock[DATA][VALUE].rolling(window=short_window).mean()
    stock[DATA]['Long Mean'] = stock[DATA][VALUE].rolling(window=long_window).mean()

    stock[DATA]['Daily Change'] = stock[DATA][VALUE].pct_change()
    stock[DATA]['Daily Short Change'] = stock[DATA]['Short Mean'].pct_change()
    stock[DATA]['Daily Long Change'] = stock[DATA]['Long Mean'].pct_change()
    stock[DATA]['Volatility'] = stock[DATA]['Daily Change'].rolling(75).std()*np.sqrt(75)

As you probably notice, this will create a challenge. You need to put them into bins, that is a fixed number of “boxes” to fit in.

def process():
    #...
    # Let's put data in bins
    stock[DATA]['Vla bin'] = pd.cut(stock[DATA]['Volatility'], bins=STATES_DIM, labels=False)
    stock[DATA]['Srt ch bin'] = pd.cut(stock[DATA]['Daily Short Change'], bins=STATES_DIM, labels=False)
    stock[DATA]['Lng ch bin'] = pd.cut(stock[DATA]['Daily Long Change'], bins=STATES_DIM, labels=False)
    # stock[DATA]['Srt mn bin'] = pd.cut(stock[DATA]['Short Mean'], bins=DIM, labels=False)
    stock[DATA]['Lng mn bin'] = pd.cut(stock[DATA]['Long Mean'], bins=STATES_DIM, labels=False)
    stock[DATA]['Vol bin'] = pd.cut(stock[DATA]['Volume'], bins=STATES_DIM, labels=False)

This will quantify the 5 dimensions into STATES_DIM, which you can define to what you think is appropriate.

Step 5: How to model it

This can be done by creating an environment, that will play the role as your trading account.

class Account:
    def __init__(self, cash=1000000, brokerage=0.001):
        self.cash = cash
        self.brokerage = brokerage
        self.stocks = 0
        self.stock_id = None
        self.has_stocks = False

    def get_value(self, row):
        if self.has_stocks:
            return self.cash + row[VALUE] * self.stocks
        else:
            return self.cash

    def buy_stock(self, stock_id, row):
        if self.has_stocks:
            return
        self.stock_id = stock_id
        self.stocks = int(self.cash // (row[VALUE]*(1.0 + self.brokerage)))
        self.cash -= self.stocks*row[VALUE]*1.001
        self.has_stocks = True
        self.print_status(row, "Buy")

    def sell_stock(self, row):
        if not self.has_stocks:
            return
        self.print_status(row, "Sell")
        self.cash += self.stocks * (row[VALUE]*(1.0 - self.brokerage))
        self.stock_id = None
        self.stocks = 0
        self.has_stocks = False

    def print_status(self, row, title="Status"):
        if self.has_stocks:
            print(title, self.stock_id, "TOTAL:", self.cash + self.stocks*float(row[VALUE]))
            print(" - ", row.name, "price", row[VALUE])
            print(" - ", "Short", row['Daily Short Change'])
            print(" - ", "Long", row['Daily Long Change'])
        else:
            print(title, "TOTAL", self.cash)

Then it should be iterated over a time where the trading bot can decide what to do.

def process():
    # Now let's prepare our model
    q_learning = QModel()
    account = Account()

    state = None
    reward = 0.0
    action = 0
    last_value = 0.0
    for index, row in stock[DATA].iterrows():
        if state is not None:
            # The reward is the immediate return
            reward = account.get_value(row) - last_value
            # You update the day after the action, when you know the results of your actions
            q_learning.update_reward(row, account.has_stocks, action, state, reward)
        action, state = q_learning.get_action(row, account.has_stocks)

        if action == 0:
            pass
        elif action == 1:
            if account.has_stocks:
                account.sell_stock(row)
            else:
                account.buy_stock(stock[ID], row)
        last_value = account.get_value(row)
    account.print_status(row)
    q_learning.save_pickle()
    return last_value

This code will do what ever the trading bot tells you to do.

Step 6: The Q-learning model

Now to the core of the thing. The actual trading bot, that knows nothing about trading. But can we train it to earn money on trading and how much? We will see that later.

class QModel:
    def __init__(self, alpha=0.5, gamma=0.7, epsilon=0.1):
        self.alpha = alpha
        self.gamma = gamma
        self.epsilon = epsilon

        self.states_per_dim = STATES_DIM
        self.dim = 5
        self.states = (self.states_per_dim ** self.dim) * 2
        self.actions = 2
        self.pickle = "q_model7.pickle"
        self.q_table = np.zeros((self.states, self.actions))
        if os.path.isfile(self.pickle):
            print("Loading pickle")
            with open(self.pickle, "rb") as f:
                self.q_table = pickle.load(f)

    def save_pickle(self):
        with open(self.pickle, "wb") as f:
            pickle.dump(self.q_table, f)

    def get_state(self, row, has_stock):
        dim = []
        dim.append(int(row['Vla bin']))
        dim.append(int(row['Srt ch bin']))
        dim.append(int(row['Lng ch bin']))
        dim.append(int(row['Lng mn bin']))
        dim.append(int(row['Vol bin']))
        for i in range(len(dim)):
            if dim[i] is None:
                dim[i] = 0
        dimension = 0
        if has_stock:
            dimension = 1 * (self.states_per_dim ** self.dim)
        dimension += dim[4] * (self.states_per_dim ** 4)
        dimension += dim[3] * (self.states_per_dim ** 3)
        dimension += dim[2] * (self.states_per_dim ** 2)
        dimension += dim[1] * (self.states_per_dim ** 1)
        dimension += dim[0]
        return dimension

    def get_action(self, row, has_stock):
        state = self.get_state(row, has_stock)

        if random.uniform(0, 1) < self.epsilon:
            action = random.randrange(0, self.actions)
        else:
            action = np.argmax(self.q_table[state])
        return action, state

    def update_reward(self, row, has_stock, last_action, last_state, reward):
        next_state = self.get_state(row, has_stock)

        old_value = self.q_table[last_state, last_action]
        next_max = np.max(self.q_table[next_state])

        new_value = (1 - self.alpha) * old_value + self.alpha * (reward + self.gamma * next_max)
        self.q_table[last_state, last_action] = new_value

Now we have the full code to try it out (the full code is at the end of the tutorial).

Step 7: Training the model

Now we need to train the model.

For that purpose, I have made a list of 134 stocks that I used and placed them in a CSV file.

Then the training is simply to read 1 of the 134 stocks in with 10 years of historical data. Find an 1 year window and run the algorithm on it.

The repeat.

f __name__ == "__main__":
    # source: http://www.nasdaqomxnordic.com/shares/listed-companies/copenhagen
    csv_stock_file = 'DK-Stocks.csv'

    while True:
        iterations = 1000
        for i in range(iterations):
            # Go at most 9 years back, as we only have 10 years available and need 1 year of data
            days_back = random.randrange(0, 9*365)
            process(csv_stock_file)

Then let it run and run and run and run again.

Step 8: Testing the algorithm

Of course, the testing should be done on unknown data. That is a stock it does not know. But you cannot also re-run on the same stock, as it will learn from it (unless you do not save the state from it).

Hence, I chose a good performing stock to see how it would do, to see if it could beat the buy-first-day-and-sell-last-day strategy.

The results of the trading bot on Apple stocks.

The return of 1,000,000$ investment with the Trading Bot was approximately 1,344,500$. This is a return on 34% for one year. Comparing that with the stock price itself.

Stock price was 201.55$ on July 1st 2019 and 362.09$ on June 30th, 2020. This would give the following return (0,10% in brokerage should be included in calculations as the Trading bot pays that on each sell and buy).

  • 1,792,847$

That does not look that good. That means that a simple strategy to buy on day one and sell on the last day would return more than the bot.

Of course, you can’t conclude it is not possible to do better on other stocks, but for this case it was not impressive.

Variations and next step

There are many variable to adjust, I especially think I set the gamma too low. There are other parameters to use to make the state. Can remove some, that might be making noice, and add ones that are more relevant. Also, the number of bins can be adjusted. That the bins are made independent of each other, might also be a problem.

Also read the tutorial on reinforcement learning.

How to Visualize Time Series Financial Data with Python in 3 Easy Steps

What will we cover

  • The easy way visualize financial data with Python
  • How to fetch data from stocks in Python
  • The easy way to visualize it on a plot
  • How to enrich the data with valuable analysis

Step 1: Fetch the data on your favorite stock

As with most things in Python, somebody made an easy to use library to do all the hard work for you. This is also true if you want to visualize financial data with Python.

The Pandas datareader library lets you fetch financial data from various sources and return them in a Pandas Dataframe.

If you do not know what a Dataframe from Pandas is, do not worry. We will cover the necessary here.

The full list data sources available can be found here, but include Tiingo, IEX, Alpha Vantage, Enigma, Quandl, St.Louis FED (FRED), Kenneth French’s data library, World Bank, and many more.

import pandas_datareader as pdr
import datetime 


aapl = pdr.get_data_yahoo('AAPL', 
                          start=datetime.datetime(2010, 1, 1), 
                          end=datetime.datetime(2020, 1, 1))

print(aapl.head())

Which will result in 10 years of data from Apple. See below.

                 High        Low  ...       Volume  Adj Close
Date                              ...                        
2010-01-04  30.642857  30.340000  ...  123432400.0  26.466835
2010-01-05  30.798571  30.464285  ...  150476200.0  26.512596
2010-01-06  30.747143  30.107143  ...  138040000.0  26.090879
2010-01-07  30.285715  29.864286  ...  119282800.0  26.042646
2010-01-08  30.285715  29.865715  ...  111902700.0  26.215786

For each bank day you will be presented with the following data.

High         2.936800e+02
Low          2.895200e+02
Open         2.899300e+02
Close        2.936500e+02
Volume       2.520140e+07
Adj Close    2.921638e+02
Name: 2019-12-31 00:00:00, dtype: float64
  • High: The highest price traded during that day.
  • Low: The lowest price traded during that day.
  • Open: The opening price that day.
  • Close: The closing price that day, that is the price of the last trade that day.
  • Volume: The number of shares that exchange hands for the stock that day.
  • Adj Close: It accurately reflect the stock’s value after accounting for any corporate actions. It is considered to be the true price of that stock and is often used when examining historical returns.

As you can see, to make further infestations on the data, you should use the Adj Close.

Step 2: Visualize the data

This is where Dataframes come in handy. They integrate easy with matplotlib, which is a comprehensive library for creating static, animated, and interactive visualizations in Python.

import pandas_datareader as pdr
import datetime 
import matplotlib.pyplot as plt


aapl = pdr.get_data_yahoo('AAPL', 
                          start=datetime.datetime(2010, 1, 1), 
                          end=datetime.datetime(2020, 1, 1))

aapl['Adj Close'].plot()
plt.show()

Will result in a graph like this.

Apple stock price.

That was easy.

You can see further ways to visualize the data in the documentation here.

Step 3: Enrich the data

That is another great advantage of Dataframes, it is easy to enrich it with valuable data.

A good example is to enrich the data with running mean values of the stock price.

import pandas_datareader as pdr
import datetime 
import matplotlib.pyplot as plt


aapl = pdr.get_data_yahoo('AAPL', 
                          start=datetime.datetime(2015, 1, 1), 
                          end=datetime.datetime(2020, 1, 1))

aapl['Mean Short'] = aapl['Adj Close'].rolling(window=20).mean()
aapl['Mean Long'] = aapl['Adj Close'].rolling(window=100).mean()
aapl[['Adj Close', 'Mean Short', 'Mean Long']].plot()
plt.show()

Which will result in the following graph.

Apple stock with rolling mean values with window of 20 and 100 days.

Now that was simple. The code simple rolls the window of 20 (or 100) days and sets the average (mean) value of that. This kind of analysis is used to see the average trend of the stock. Also, to see when the short term trend (20 days) is crossing the long term trend (100 days).

A simple trading strategy is decides on to buy and sell based on when these averages crosses. This strategy is called dual moving average crossover. You can read more about it here.

A volatility analysis is used to see how “stable” the share is. The higher the volatility value is, the riskier it is.

import pandas_datareader as pdr
import datetime 
import matplotlib.pyplot as plt
import numpy as np


aapl = pdr.get_data_yahoo('AAPL', 
                          start=datetime.datetime(2015, 1, 1), 
                          end=datetime.datetime(2020, 1, 1))


daily_close = aapl['Close']
daily_pc = daily_close.pct_change()

# See the volatibility
vol = daily_pc.rolling(75).std()*np.sqrt(75)
vol.plot()
plt.show()

Which results in the following graph.

Volatility of Apple stock

This is a good indication on how risky the stock is.

Conclusion

This is just a simple introduction to how to retrieve financial stock data in Python and visualize it. Also, how easy it is to enrich it with more valuable analysis.

There are so much more to explore and learn about it.

Build a Financial Trading Algorithm in Python in 5 Easy Steps

What will we cover in this tutorial?

Step 1: Get time series data on your favorite stock

To build a financial trading algorithm in Python, it needs to be fed with data. Hence, the first step you need to master is how to collect time series data on your favorite stock. Sounds like it is difficult, right?

Luckily, someone made an awesome library pandas-datareader, which does all the hard word you for you. Let’s take a look on how easy it is.

import datetime as dt
import pandas_datareader as pdr
import matplotlib.pyplot as plt
from dateutil.relativedelta import relativedelta


# A Danish jewellery manufacturer and retailer
stock = 'PNDORA.CO'
end = dt.datetime.now()
start = end - relativedelta(years=10)
pndora = pdr.get_data_yahoo(stock, start, end)

pndora['Close'].plot()

plt.show()

Which results in a time series of the closing price as shown here.

Pandora A/S time series stock price

The stock is probably quite unknown, considering it is a Danish company. But to prove a point that you can get data, including for Pandora.

In the code you see that you send a start and end date to the call fetching the data. Here we have taken the last 10 years. The object returned integrates well with the matplotlib library to make the plot.

To understand the data better, we need to explore further.

Step 2: Understand the data available

Let us explore the data object returned by the call (pndora).

To get an overview you should run the following code using the iloc-call to a the Dataframe object (pndora returned by pandas_datareader).

pndora.iloc[-1]

This will show what the last item of the object looks like.

High            365.000000
Low             355.600006
Open            360.000000
Close           356.500000
Volume       446004.000000
Adj Close       356.500000
Name: 2020-06-26 00:00:00, dtype: float64

Where you have the following items.

  • High: The highest price traded during that day.
  • Low: The lowest price traded during that day.
  • Open: The opening price that day.
  • Close: The closing price that day, that is the price of the last trade that day.
  • Volume: The number of shares that exchange hands for the stock that day.
  • Adj Close: It accurately reflect the stock’s value after accounting for any corporate actions. It is considered to be the true price of that stock and is often used when examining historical returns.

That means, it would be natural to use Adj Close in our calculations. Hence, for each day we have the above information available.

Step 3: Learning how to enrich the data (Pandas)

Pandas? Yes, you read correct. But not a Panda like this the picture.

Pandas the Python library.
Pandas the Python library.

There is an awesome library Pandas in Python to make data analysis easy.

Let us explore some useful things we can do.

import datetime as dt
import pandas_datareader as pdr
import matplotlib.pyplot as plt
from dateutil.relativedelta import relativedelta


# A Danish jewellery manufacturer and retailer
stock = 'PNDORA.CO'
end = dt.datetime.now()
start = end - relativedelta(years=10)
pndora = pdr.get_data_yahoo(stock, start, end)

pndora['Short'] = pndora['Adj Close'].rolling(window=20).mean()
pndora['Long'] = pndora['Adj Close'].rolling(window=100).mean()

pndora[['Adj Close', 'Short', 'Long']].plot()
plt.show()

Which will result in the following graph.

Pandora A/S stock prices with Short mean average and Long mean average.
Pandora A/S stock prices with Short mean average and Long mean average.

If you inspect the code above, you see, that you easily added to two new columns (Short and Long) and computed them to be the mean value of the previous 20 and 100 days, respectively.

Further, you can add the daily percentage change for the various entries.

import datetime as dt
import pandas_datareader as pdr
import matplotlib.pyplot as plt
from dateutil.relativedelta import relativedelta


# A Danish jewellery manufacturer and retailer
stock = 'PNDORA.CO'
end = dt.datetime.now()
start = end - relativedelta(years=10)
pndora = pdr.get_data_yahoo(stock, start, end)

pndora['Short'] = pndora['Adj Close'].rolling(window=20).mean()
pndora['Long'] = pndora['Adj Close'].rolling(window=100).mean()

pndora['Pct Change'] = pndora['Adj Close'].pct_change()
pndora['Pct Short'] = pndora['Short'].pct_change()
pndora['Pct Long'] = pndora['Long'].pct_change()

pndora[['Pct Change', 'Pct Short', 'Pct Long']].loc['2020'].plot()
plt.show()

Which will result in this graph.

Pandora A/S stock's daily percentage change for Adj Close, short and long mean.
Pandora A/S stock’s daily percentage change for Adj Close, short and long mean.

Again you can see how easy it is to add new columns in the Dataframe object provided by Pandas library.

Step 4: Building your strategy to buy and sell stocks

For the example we will keep it simple and only focus on one stock. The strategy we will use is called the dual moving average crossover.

Simply explained, you want to buy stocks when the short mean average is higher than the long mean average value.

In the figure above, it is translates to.

  • Buy when the yellow crosses above the green line.
  • Sell when the yellow crosses below the green line.

To implement the simplest version of that it would be as follows.

import datetime as dt
import pandas_datareader as pdr
import matplotlib.pyplot as plt
import numpy as np
from dateutil.relativedelta import relativedelta


# A Danish jewellery manufacturer and retailer
stock = 'PNDORA.CO'
end = dt.datetime.now()
start = end - relativedelta(years=10)
pndora = pdr.get_data_yahoo(stock, start, end)

short_window = 20
long_window = 100

pndora['Short'] = pndora['Adj Close'].rolling(window=short_window).mean()
pndora['Long'] = pndora['Adj Close'].rolling(window=long_window).mean()

# Let us create some signals
pndora['signal'] = 0.0
pndora['signal'][short_window:] = np.where(pndora['Short'][short_window:] > pndora['Long'][short_window:], 1.0, 0.0)

pndora['positions'] = pndora['signal'].diff()

To visually see where to buy and sell you can use the following code afterwards on pndora.

fig = plt.figure()

ax1 = fig.add_subplot(111, ylabel='Price')

pndora[['Adj Close', 'Short', 'Long']].plot(ax=ax1)

ax1.plot(pndora.loc[pndora.positions == 1.0].index,
         pndora.Short[pndora.positions == 1.0],
         '^', markersize=10, color='m')

ax1.plot(pndora.loc[pndora.positions == -1.0].index,
         pndora.Short[pndora.positions == -1.0],
         'v', markersize=10, color='k')

plt.show()

Which would result in the following graph.

Finally, you need to see how your algorithm performs.

Step 5: Testing you trading algorithm

There are many ways to test an algorithm. Here we go all in each cycle. We buy as much as we can and sell them all when we sell.

import datetime as dt
import pandas_datareader as pdr
import matplotlib.pyplot as plt
import numpy as np
from dateutil.relativedelta import relativedelta


# A Danish jewellery manufacturer and retailer
stock = 'PNDORA.CO'
end = dt.datetime.now()
start = end - relativedelta(years=10)
pndora = pdr.get_data_yahoo(stock, start, end)

short_window = 20
long_window = 100

pndora['Short'] = pndora['Adj Close'].rolling(window=short_window).mean()
pndora['Long'] = pndora['Adj Close'].rolling(window=long_window).mean()

# Let us create some signals
pndora['signal'] = 0.0
pndora['signal'][short_window:] = np.where(pndora['Short'][short_window:] > pndora['Long'][short_window:], 1.0, 0.0)

pndora['positions'] = pndora['signal'].diff()

cash = 1000000
stocks = 0
for index, row in pndora.iterrows():
    if row['positions'] == 1.0:
        stocks = int(cash//row['Adj Close'])
        cash -= stocks*row['Adj Close']
    elif row['positions'] == -1.0:
        cash += stocks*row['Adj Close']
        stocks = 0

print("Total:", cash + stocks*pndora.iloc[-1]['Adj Close'])

Which results in.

Total: 2034433.8826065063

That is a double in 10 years, which is less than 8% per year. Not so good.

As this is one specific stock, it is not fair to judge the algorithm being poor, it can be the stock which was not performing good, or the variables can be further adjusted.

If compared with the scenario where you bought the stocks at day one and sold them on the last day, your earnings would be 1,876,232. Hence, the algorithm beats that.

Conclusion and further work

This is a simple financial trading algorithm in Python and there are variables that can be adjusted. The algorithm was performing better than the naive strategy to buy on day one and sell on the last day.

It could be interesting to add more data into the decision in the algorithm, which might be some future work to do. Also, can it be combined with some Machine Learning?

Python Twitter Bot to Unfollow Friends that do not Follow Back – 3 Easy Steps

What will we cover in this tutorial?

  • How to unfollow friends in Twitter Python that do not follow back.
  • The process to get the access tokens to use the Twitter API
  • How to connect to the twitter API
  • The actual implementation of the code.

Step 1: Setup up environment

In order to get a connection to twitter you need to have access tokens and consumer keys. If you don’t already have that, or you do not know what it is, then I recommend you follow this tutorial.

You also need the tweepy library. You can install it by typing the following command in the command line or see here for more details.

pip install tweepy

Then you are ready to connect to the Twitter API.’

Step 2: Connecting to Twitter API

The first thing your code should do is to connect to the Twitter API and return the tweepy api to your program.

import tweepy
 
def get_twitter_api():
    # personal details
    consumer_key = "__USE YOUR KEY HERE__"
    consumer_secret = "__USE YOUR KEY HERE__"
    access_token = "__USE YOUR KEY HERE__"
    access_token_secret = "__USE YOUR KEY HERE__"
 
    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
 
    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True)
    return api

This code will authenticate and return the tweepy api.

Step 3: List friends and followers to unfollow non-following friends

Confused by the headline? Me, too. But here is where the magic happens. 

The code simply explained.

  • Retrieves all the users that follow you (followers)
  • Retrieves those you follow (friends)
  • Loops through friends and check if they follow you
    • If not, unfollow them with a call to destroy_friendship
def process():
    api = get_twitter_api()
 
    followers = api.followers_ids(api.me().id)
    print("Followers", len(followers))
    friends = api.friends_ids(api.me().id)
    print("You follow:", len(friends))
 
    for friend in friends:
        if friend not in followers:
            api.destroy_friendship(friend)

Full code example here

You can see the full code here to unfollow friends that do not follow back in Twitter using Python

import tweepy
 
 
def get_twitter_api():
    # personal details
    consumer_key = "__USE YOUR KEY HERE__"
    consumer_secret = "__USE YOUR KEY HERE__"
    access_token = "__USE YOUR KEY HERE__"
    access_token_secret = "__USE YOUR KEY HERE__"
 
    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
 
    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True)
    return api
 
 
def process():
    api = get_twitter_api()
 
    followers = api.followers_ids(api.me().id)
    print("Followers", len(followers))
    friends = api.friends_ids(api.me().id)
    print("You follow:", len(friends))
 
    for friend in friends:
        if friend not in followers:
            api.destroy_friendship(friend) 
 
if __name__ == "__main__":
    process()

Next step

  • Deploy it to an cron job so it runs every hour.
  • You can use PythonAnywhere (not sponsored by them)

Python Twitter Bot to Follow Followers – 3 Easy Steps

What will we cover in this tutorial?

  • To build a Bot to Follow Followers in Twitter using Python
  • Link to how you can get your access tokens and consumer keys to get access to the Twitter API (needed)
  • How to access the Twitter API
  • Finally, full code example of a Python Twitter Bot to follow the followers your account does not follow already.

Step 1: Setup up environment

In order to get a connection to twitter you need to have access tokens and consumer keys. If you don’t already have that, or you do not know what it is, then I recommend you follow this tutorial.

You also need the tweepy library. You can install it by typing the following command in the command line or see here for more details.

pip install tweepy

Then you are ready to connect to the Twitter API.

Step 2: Connecting to Twitter API

The first thing your code should do is to connect to the Twitter API and return the tweepy api to your program.

import tweepy

def get_twitter_api():
    # personal details
    consumer_key = "__USE YOUR KEY HERE__"
    consumer_secret = "__USE YOUR KEY HERE__"
    access_token = "__USE YOUR KEY HERE__"
    access_token_secret = "__USE YOUR KEY HERE__"

    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True)
    return api

This code will authenticate and return the tweepy api.

Step 3: List followers and friends to follow back

Confused by the headline? Me, too. But here is where the magic happens.

The code simply explained.

  • Retrieves all the users that follow you (followers)
  • Retrieves those you follow (friends)
  • Loops through followers and check if you follow them.
    • If not, follow them back
def process():
    api = get_twitter_api()

    followers = api.followers_ids(api.me().id)
    print("Followers", len(followers))
    friends = api.friends_ids(api.me().id)
    print("You follow:", len(friends))

    for follower in followers:
        if follower not in friends:
            api.create_friendship(follower)

Full code example here

You can see the full code here.

import tweepy


def get_twitter_api():
    # personal details
    consumer_key = "__USE YOUR KEY HERE__"
    consumer_secret = "__USE YOUR KEY HERE__"
    access_token = "__USE YOUR KEY HERE__"
    access_token_secret = "__USE YOUR KEY HERE__"

    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True)
    return api


def process():
    api = get_twitter_api()

    followers = api.followers_ids(api.me().id)
    print("Followers", len(followers))
    friends = api.friends_ids(api.me().id)
    print("You follow:", len(friends))

    for follower in followers:
        if follower not in friends:
            api.create_friendship(follower)


if __name__ == "__main__":
    process()

Next steps

  • Deploy it to an cron job so it runs every hour.
  • You can use PythonAnywhere (not sponsored by them)

Plot Tweets Locations on a Leaflet Map using Python in 3 Easy Steps

What will we cover?

  • How to plot locations of tweets on a leaflet map using Python
  • Setup your access to the Twitter API
  • How to collect location data from Twitter and tweets.
  • Finally, how to plot it on an interactive leaflet map.

Step 1: Getting ready to collect data from Twitter

Twitter is an amazing place to explore data as the API is easy to get access to and the data is public available to everyone. This is also the case if you want to plot Tweet Locations on a Leaflet Map using Python.

Using Python to interact with Twitter is easy and does require a lot to get started. I prefer to use the tweepy library, which is, as they say, “an easy-to-use Python library to accessing the Twitter API”.

Python.org

To install the tweepy library, simply type the following in a command shell.

pip install tweepy
Tweepy.org

The next step is to gather your key values to access the API.

You can get them from https://developer.twitter.com/.

If you need help to get them, I can suggest you follow this tutorial first, which will help you set everything up correctly.

Step 2: Collect the locations from the Tweets

Exploring the data available on a tweet, it has a coordinates and place field.

If you read the first word then you realize.

  • Coordinates: Nullable. Represent the geographic location of this Tweet as reported by the user or client application.
  • Place: Nullable. When present, indicates that the tweet is associated (but not necessarily originating from) a Place.

Nullable, which mean that it can be null, i.e., have no value.

But let us see how often they are set.

import tweepy

def get_twitter_api():
    # personal details
    consumer_key = "___INSERT_YOUR_VALUE_HERE___"
    consumer_secret = "___INSERT_YOUR_VALUE_HERE___"
    access_token = "___INSERT_YOUR_VALUE_HERE___"
    access_token_secret = "___INSERT_YOUR_VALUE_HERE___"

    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)
    return api

def get_twitter_location(search):
    api = get_twitter_api()

    count = 0
    for tweet in tweepy.Cursor(api.search, q=search).items(500):
        if hasattr(tweet, 'coordinates') and tweet.coordinates is not None:
            count += 1
            print("Coordinates", tweet.coordinates)
        if hasattr(tweet, 'location') and tweet.location is not None:
            count += 1
            print("Coordinates", tweet.location)
    print(count)

get_twitter_location("#100DaysOfCode")

Which resulted in 0. I would not expect this to be the case, but you never know.

Hence, the second best thing you can use, is then the location of the user. Most users have a location given in the user object you see the following.

User Object from developer.twitter.com.
User Object from developer.twitter.com.

This results in the following way to collect it. We need to check for the object being None.

def get_tweets(search):
    api = get_twitter_api()

    location_data = []
    for tweet in tweepy.Cursor(api.search, q=search).items(500):
        if hasattr(tweet, 'user') and hasattr(tweet.user, 'screen_name') and hasattr(tweet.user, 'location'):
            if tweet.user.location:
                location_data.append((tweet.user.screen_name, tweet.user.location))
    return location_data

Here we collect all the locations of the users of the tweets and return a list of them.

Step 3: Plot the data on an interactive map

The folium library is amazing to plot data on an interactive leaflet map.

To install the folium library simply type the following command in a terminal.

pip install folium

Or read more here, on how to install it.

We also need to find the coordinates from each location. This can be done by using the library geopy. It can be installed by typing the following command in a terminal.

pip install geopy

Or read more here.

Given that the plotting is done by the following lines of code. Please notice, I put a try-except around the geocode call, as it tends to get an timeout.

import folium
from geopy.exc import GeocoderTimedOut
from geopy.geocoders import Nominatim


def put_markers(map, data):
    geo_locator = Nominatim(user_agent="LearnPython")

    for (name, location) in data:
        if location:
            try:
                location = geo_locator.geocode(location)
            except GeocoderTimedOut:
                continue
            if location:
                folium.Marker([location.latitude, location.longitude], popup=name).add_to(map)


if __name__ == "__main__":
    map = folium.Map(location=[0, 0], zoom_start=2)
    location_data = get_tweets("#100DaysOfCode")
    put_markers(map, location_data)
    map.save("index.html")

This results in the following beautiful map.

Interactive map.
Interactive map.

Want to learn more Python? Also, check out my online course on Python.

How to Create a Sentiment Analysis model to Predict the Mood of Tweets with Python – 4 Steps to Compare the Mood of Python vs Java

What will we cover in this tutorial?

  • We will learn how the supervised Machine Learning algorithm Sentiment Analysis can be used on twitter data (also, called tweets).
  • The model we use will be Naive Bayes Classifier.
  • The tutorial will help install the necessary Python libraries to get started and how to download training data.
  • Then it will give you a full script to train the model.
  • Finally, we will use the trained model to compare the “mood” of Python with Java.

Step 1: Install the Natural Language Toolkit Library and Download Collections

We will use the Natural Language Toolkit (nltk) library in this tutorial.

NLTK is a leading platform for building Python programs to work with human language data.

http://www.nltk.org

To install the library you should run the following command in a terminal or see here for other alternatives.

pip install nltk

To have the data available that you need to run the following program or see installing NLTK Data.

import nltk
nltk.download()

This will prompt you with a screen similar to this. And select all packages you want to install (I took them all).

Download all packages to NLTK (Natural Language Toolkit)
Download all packages to NLTK (Natural Language Toolkit)

After download you can use the twitter_samples as you need in the example.

Step 2: Reminder of the Sentiment Analysis learning process (Machine Learning)

On a high level you can divide Machine Learning into two phases.

  • Phase 1: Learning
  • Phase 2: Prediction

The Sentiment Analysis model is supervised learning process. The process is defined in the picture below.

The Sentiment Analysis model (Machine Learning) Learning phase
The Sentiment Analysis model (Supervised Machine Learning) Learning phase

On a high level the the learning process of Sentiment Analysis model has the following steps.

  • Training & test data
    • The Sentiment Analysis model is a supervised learning and needs data representing the data that the model should predict. We will use tweets.
    • The data should be categorized into the groups it should be able to distinguish. In our example it will be in positive tweets and negative tweets.
  • Pre-processing
    • First you need to remove “noise”. In our case we remove URL links and Twitter user names.
    • Then you Lemmatize the data to have the words in the same form.
    • Further, you remove stop words as they have no impact of the mood in the tweet.
    • The data then needs to be formatted for the algorithm.
    • Finally, you need to divide it into a training data and testing data.
  • Learning
    • This is where the algorithm builds the model using the training data.
  • Testing
    • Then we test the accuracy of the model with the categorized test data.

Step 3: Train the Sample Data

The twitter_sample contains 5000 positive and 5000 negative tweets, all ready and classified to use in for your training model.

import random
import pickle

from nltk.corpus import twitter_samples
from nltk.stem import WordNetLemmatizer
from nltk.tag import pos_tag
from nltk.corpus import stopwords
from nltk import NaiveBayesClassifier
from nltk import classify


def clean_data(token):
    return [item for item in token if not item.startswith("http") and not item.startswith("@")]


def lemmatization(token):
    lemmatizer = WordNetLemmatizer()

    result = []
    for token, tag in pos_tag(token):
        tag = tag[0].lower()
        token = token.lower()
        if tag in "nva":
            result.append(lemmatizer.lemmatize(token, pos=tag))
        else:
            result.append(lemmatizer.lemmatize(token))
    return result


def remove_stop_words(token, stop_words):
    return [item for item in token if item not in stop_words]


def transform(token):
    result = {}
    for item in token:
        result[item] = True
    return result


def main():
    # Step 1: Gather data
    positive_tweets_tokens = twitter_samples.tokenized('positive_tweets.json')
    negative_tweets_tokens = twitter_samples.tokenized('negative_tweets.json')

    # Step 2: Clean, Lemmatize, and remove Stop Words
    stop_words = stopwords.words('english')
    positive_tweets_tokens_cleaned = [remove_stop_words(lemmatization(clean_data(token)), stop_words) for token in positive_tweets_tokens]
    negative_tweets_tokens_cleaned = [remove_stop_words(lemmatization(clean_data(token)), stop_words) for token in negative_tweets_tokens]

    # Step 3: Transform data
    positive_tweets_tokens_transformed = [(transform(token), "Positive") for token in positive_tweets_tokens_cleaned]
    negative_tweets_tokens_transformed = [(transform(token), "Negative") for token in negative_tweets_tokens_cleaned]


    # Step 4: Create data set
    dataset = positive_tweets_tokens_transformed + negative_tweets_tokens_transformed
    random.shuffle(dataset)

    train_data = dataset[:7000]
    test_data = dataset[7000:]

    # Step 5: Train data
    classifier = NaiveBayesClassifier.train(train_data)

    # Step 6: Test accuracy
    print("Accuracy is:", classify.accuracy(classifier, test_data))
    print(classifier.show_most_informative_features(10))

    # Step 7: Save the pickle
    f = open('my_classifier.pickle', 'wb')
    pickle.dump(classifier, f)
    f.close()


if __name__ == "__main__":
    main()

The code is structured in steps. If you are not comfortable how a the flow of a general machine learning flow is, I can recommend to read this tutorial here or this one.

  • Step 1: Collect and categorize It reads the 5000 positive and 5000 negative twitter samples we downloaded with the nltk.download() call.
  • Step 2: The data needs to be cleaned, Lemmatized and removed for stop words.
    • The clean_data call removes links and twitter users.
    • The call to lemmatization puts words in their base form.
    • The call to remove_stop_words removes all the stop words that have no affect on the mood of the sentence.
  • Step 3: Format data This step transforms the data to the desired format for the NaiveBayesClassifier module.
  • Step 4: Divide data Creates the full data set. Makes a shuffle to take them in different order. Then takes 70% as training data and 30% test data.
    • This data is mixed different from run to run. Hence, it might happen that you will not get the same accuracy like I will in my run.
    • The training data is used to make the model to predict from.
    • The test data is used to compute the accuracy of the model to predict.
  • Step 5: Training model This is the training of the NaiveBayesClassifier model.
    • This is where all the magic happens.
  • Step 6: Accuracy This is testing the accuracy of the model.
  • Step 7: Persist To save the model for use.

I got the following output from the above program.

Accuracy is: 0.9973333333333333
Most Informative Features
                      :) = True           Positi : Negati =   1010.7 : 1.0
                     sad = True           Negati : Positi =     25.4 : 1.0
                     bam = True           Positi : Negati =     20.2 : 1.0
                  arrive = True           Positi : Negati =     18.3 : 1.0
                     x15 = True           Negati : Positi =     17.2 : 1.0
               community = True           Positi : Negati =     14.7 : 1.0
                    glad = True           Positi : Negati =     12.6 : 1.0
                   enjoy = True           Positi : Negati =     12.0 : 1.0
                    kill = True           Negati : Positi =     12.0 : 1.0
                     ugh = True           Negati : Positi =     11.3 : 1.0
None

Step 4: Use the Sentiment Analysis prediction model

Now we can determine the mood of a tweet. To have some fun let us try to figure out the mood of tweets with Python and compare it with Java.

To do that, you need to have setup your twitter developer account. If you do not have that already, then see the this tutorial on how to do that.

In the code below you need to fill out your consumer_key, consumer_secret, access_token, and access_token_secret.

import pickle
import tweepy


def get_twitter_api():
    # personal details
    consumer_key = "___INSERT YOUR DATA HERE___"
    consumer_secret = "___INSERT YOUR DATA HERE___"
    access_token = "___INSERT YOUR DATA HERE___"
    access_token_secret = "___INSERT YOUR DATA HERE___"

    # authentication of consumer key and secret
    auth = tweepy.OAuthHandler(consumer_key, consumer_secret)

    # authentication of access token and secret
    auth.set_access_token(access_token, access_token_secret)
    api = tweepy.API(auth)
    return api


# This function uses the functions from the learner code above
def tokenize(tweet):
    return remove_noise(word_tokenize(tweet))


def get_classifier(pickle_name):
    f = open(pickle_name, 'rb')
    classifier = pickle.load(f)
    f.close()
    return classifier


def find_mood(search):
    classifier = get_classifier('my_classifier.pickle')

    api = get_twitter_api()

    stat = {
        "Positive": 0,
        "Negative": 0
    }
    for tweet in tweepy.Cursor(api.search, q=search).items(1000):
        custom_tokens = tokenize(tweet.text)

        category = classifier.classify(dict([token, True] for token in custom_tokens))
        stat[category] += 1

    print("The mood of", search)
    print(" - Positive", stat["Positive"], round(stat["Positive"]*100/(stat["Positive"] + stat["Negative"]), 1))
    print(" - Negative", stat["Negative"], round(stat["Negative"]*100/(stat["Positive"] + stat["Negative"]), 1))


if __name__ == "__main__":
    find_mood("#java")
    find_mood("#python")

That is it. Obviously the mood of Python is better. It is easier than Java.

The mood of #java
 - Positive 524 70.4
 - Negative 220 29.6
The mood of #python
 - Positive 753 75.3
 - Negative 247 24.7

If you want to learn more about Python I can encourage you to take my course here.

3 Steps to Plot Shooting Incident in NY on a Map Using Python

What will you learn in this tutorial?

  • Where to find interesting data contained in CSV files.
  • How to extract a map to plot the data on.
  • Use Python to easily plot the data from the CSV file no the map.

Step 1: Collect the data in CSV format

You can find various interesting data in CSV format on data.world that you can play around with in Python.

In this tutorial we will focus on Shooting Incidents in NYPD from the last year. You can find the data on data.world.

data.world with NYPD Shooting Incident Data (Year To Date)
data.world with NYPD Shooting Incident Data (Year To Date)

You can download the CSV file containing all the data by pressing on the download link.

To download CSV file press the download.
To download CSV file press the download.

Looking at the data you see that each incident has latitude and longitude coordinates.

{'INCIDENT_KEY': '184659172', 'OCCUR_DATE': '06/30/2018 12:00:00 AM', 'OCCUR_TIME': '23:41:00', 'BORO': 'BROOKLYN', 'PRECINCT': '75', 'JURISDICTION_CODE': '0', 'LOCATION_DESC': 'PVT HOUSE                     ', 'STATISTICAL_MURDER_FLAG': 'false', 'PERP_AGE_GROUP': '', 'PERP_SEX': '', 'PERP_RACE': '', 'VIC_AGE_GROUP': '25-44', 'VIC_SEX': 'M', 'VIC_RACE': 'BLACK', 'X_COORD_CD': '1020263', 'Y_COORD_CD': '184219', 'Latitude': '40.672250312', 'Longitude': '-73.870176252'}

That means we can plot on a map. Let us try to do that.

Step 2: Export a map to plot the data

We want to plot all the shooting incidents on a map. You can use OpenStreetMap to get an image of a map.

We want a map of New York, which you can find by locating it on OpenStreetMap or pressing the link.

OpenStreetMap (sorry for the Danish language)

You should press the blue Download in the low right corner of the picture.

Also, remember to get the coordinates of the image in the left side bar, we will need them for the plot.

map_box = [-74.4461, -73.5123, 40.4166, 41.0359]

Step 3: Writing the Python code that adds data to the map

Importing data from a CVS file is easy and can be done through the standard library csv. Making plot on a graph can be done in matplotlib. If you do not have it installed already, you can do that by typing the following in a command line (or see here).

pip install matplotlib

First you need to transform the CVS data of the longitude and latitude to floats.

import csv


# The name of the input file might need to be adjusted, or the location needs to be added if it is not located in the same folder as this file.
csv_file = open('nypd-shooting-incident-data-year-to-date-1.csv')
csv_reader = csv.DictReader(csv_file)
longitude = []
latitude = []
for row in csv_reader:
    longitude.append(float(row['Longitude']))
    latitude.append(float(row['Latitude']))

Now you have two lists (longitude and latitude), which contains the coordinates to plot.

Then for the actual plotting into the image.

import matplotlib.pyplot as plt


# The boundaries of the image map
map_box = [-74.4461, -73.5123, 40.4166, 41.0359]

# The name of the image of the New York map might be different.
map_img = plt.imread('map.png')

fig, ax = plt.subplots()
ax.scatter(longitude, latitude)
ax.set_ylim(map_box[2], map_box[3])
ax.set_xlim(map_box[0], map_box[1])
ax.imshow(map_img, extent=map_box, alpha=0.9)


plt.savefig("mad_mod.png")
plt.show()

This will result in the following beautiful map of New York, which highlights where the shooting in the last year has occurred.

Shootings in New York in the last year. Plot by Python using matplotlib.
Shootings in New York in the last year. Plot by Python using matplotlib.

Now that is awesome. If you want to learn more, this and more is covered in my online course. Check it out.

You can also read about how to plot the mood of tweets on a leaflet map.

5 Steps to Master the Reinforcement Learning with a Q-Learning Python Example

What will we learn in this article?

The Q-Learning algorithm is a nice and easy to understand algorithm used with Reinforcement Learning paradigm in Machine Learning. It can be implemented from scratch and we will do that in this article.

After you go through this article you will know what Reinforcement Learning is, the main types of algorithm used, fully understand the Q-learning algorithm and implement an awesome example from scratch in Python.

The steps towards that are.

  • Learn and understand what reinforcement learning in machine learning?
  • What are the main algorithm in reinforcement learning?
  • Deep dive to understand the Q-learning algorithm
  • Implement a task that we want the Q-learning algorithm to learn – first we let a random choices try (1540 steps on average).
  • Then we implement the Q-learning algorithm from scratch and let it solve learn how to solve it (22 steps).

Step 1: What is Reinforcement Learning?

Reinforcement learning teaches the machine to think for itself based on past action rewards.

Reinforcement Learning (in Machine Learning) teaches the machine to think based on past action rewards.
Reinforcement Learning (in Machine Learning) teaches the machine to think based on past action rewards.

Basically, the Reinforcement Learning algorithm tries to predict actions that gives rewards and avoids punishment.

It is like training a dog. You and the dog do not talk the same language, but the dogs learns how to act based on rewards (and punishment, which I do not advise or advocate).

Hence, if a dog is rewarded for a certain action in a given situation, then next time it is exposed to a similar situation it will act the same.

Translate that to Reinforcement Learning.

  • The agent is the dog that is exposed to the environment.
  • Then the agent encounters a state.
  • The agent performs an action to transition from that state to a new state.
  • Then after the transition the agent receives a reward or penalty (punishment).
  • This forms a policy to create a strategy to choose actions in a given state.

Step 2: What are the algorithm used for Reinforcement Learning?

The most common algorithm for Reinforcement Learning are.

We will focus on the Q-learning algorithm as it is easy to understand as well as powerful.

Step 3: Understand the Q-Learning algorithm

As already noted, I just love this algorithm. It is “easy” to understand and seems very powerful.

Q-Learning algorithm (Reinforcement / Machine Learning) - exploit or explore - Update Q-table
Q-Learning algorithm (Reinforcement / Machine Learning) – exploit or explore – Update Q-table

The Q-Learning algorithm has a Q-table (a Matrix of dimension state x actions – don’t worry if you do not understand what a Matrix is, you will not need the mathematical aspects of it – it is just an indexed “container” with numbers).

  • The agent (or Q-Learning algorithm) will be in a state.
  • Then in each iteration the agent needs take an action.
  • The agent will continuously update the reward in the Q-table.
  • The learning can come from either exploiting or exploring.

This translates into the following pseudo algorithm for the Q-Learning.

The agent is in a given state and needs to choose an action.

  • Initialise the Q-table to all zeros
  • Iterate:
    • Agent is in state state.
    • With probability epsilon choose to explore, else exploit.
      • If explore, then choose a random action.
      • If exploit, then choose the best action based on the current Q-table.
    • Update the Q-table from the new reward to the previous state.
      • Q[state, action] = (1 – alpha) * Q[state, action] + alpha * (reward + gamma * max(Q[new_state]) — Q[state, action])

As you can se, we have introduced the following variables.

  • epsilon: the probability to take a random action, which is done to explore new territory.
  • alpha: is the learning rate that the algorithm should make in each iteration and should be in the interval from 0 to 1.
  • gamma: is the discount factor used to balance the immediate and future reward. This value is usually between 0.8 and 0.99
  • reward: is the feedback on the action and can be any number. Negative is penalty (or punishment) and positive is a reward.

Step 4: A task we want the Q-learning algorithm to master

We need to test and understand our the above algorithm. So far, it is quite abstract. To do that we will create a simple task to show how the Q-learning algorithm will solve it efficient by learning by rewards.

To keep it simple, we create a field of size 10×10 positions. In that field there is an item that needs to be picket up and moved to a drop-off point.

At each position there are 6 different actions that can be taken.

  • Action 0: Go south if on field.
  • Action 1: Go north if on field.
  • Action 2: Go east if on field.
  • Action 3: Go west if on field.
  • Action 4: Pickup item (it can try even if it is not there)
  • Action 5: Drop-off item (it can try even if it does not have it)

Based on these action we will make a reward system.

  • If the agent tries to go off the field, punish with -10 in reward.
  • If the agent makes a (legal) move, punish with -1 in reward, as we do not want to encourage endless walking around.
  • If the agent tries to pick up item, but it is not there or it has it already, punish with 10.
  • If the agent picks up the item correct place, reward with 20.
  • If agent tries to drop-off item in wrong place or does not have the item, punish with 10.
  • If the agent drops-off item in correct place, reward with 20.

That translates into the following code. I prefer to implement this code, as I think the standard libraries that provide similar frameworks hide some important details. As an example, and shown later, how do you map this into a state in the Q-table?

class Field:
    def __init__(self, size, item_pickup, item_drop_off, start_position):
        self.size_x = size
        self.size_y = size
        self.item_in_car = False
        self.item_position = item_pickup
        self.item_drop_off = item_drop_off
        self.position = start_position

    def move_driver(self, action):
        (x, y) = self.item_position
        if action == 0: # south
            if y == 0:
                return -10, False
            else:
                self.item_position = (x, y-1)
                return -1, False
        elif action == 1: # north
            if y == self.size_y - 1:
                return -10, False
            else:
                self.item_position = (x, y+1)
                return -1, False
        elif action == 2: # east
            if x == self.size_x - 1:
                return -10, False
            else:
                self.item_position = (x+1, y)
                return -1, False
        elif action == 3: # west
            if x == 0:
                return -10, False
            else:
                self.item_position = (x-1, y)
                return -1, False
        elif action == 4: # pickup
            if self.item_in_car:
                return -10, False
            elif self.item_position != (x, y):
                return -10, False
            else:
                self.item_in_car = True
                return 20, False
        elif action == 5: # drop-off
            if not self.item_in_car:
                return -10, False
            elif self.item_drop_off != (x, y):
                self.item_position = (x, y)
                return -20, False
            else:
                return 20, True

If you let the agent just do random actions, how long will it take for it to succeed (to be done)? Let us try that out.

import random


size = 10
item_start = (0, 0)
item_drop_off = (9, 9)
start_position = (9, 0)

field = Field(size, item_start, item_drop_off, start_position)
done = False
steps = 0
while not done:
    action = random.randrange(0, 6)
    reward, done = field.move_driver(action)
    steps += 1
print(steps)

A single run of that resulted in 2756 steps. That seems to be inefficient. I ran it 1000 times to find an average, which resulted to 1540 steps on average.

Step 5: How the Q-learning algorithm can improve that

There is a learning phase where the Q-table is updated iteratively. But before that, we need to add two helper functions to our Field.

  • We need to be able to map the current it to a state to an index in the Q-table.
  • Further, we need to a get the number of states needed in the Q-table, which we need to know when we initialise the Q-table.
import numpy as np
import random


class Field:
    def __init__(self, size, item_pickup, item_drop_off, start_position):
        self.size_x = size
        self.size_y = size
        self.item_in_car = False
        self.item_position = item_pickup
        self.item_drop_off = item_drop_off
        self.position = start_position

    def get_number_of_states(self):
        return self.size_x*self.size_y*self.size_x*self.size_y*2

    def get_state(self):
        state = self.item_position[0]*(self.size_y*self.size_x*self.size_y*2)
        state += self.item_position[1]*(self.size_x*self.size_y*2)
        state += self.position[0] * (self.size_y * 2)
        state += self.position[1] * (2)
        if self.item_in_car:
            state += 1
        return state

    def move_driver(self, action):
        (x, y) = self.item_position
        if action == 0: # south
            if y == 0:
                return -10, False
            else:
                self.item_position = (x, y-1)
                return -1, False
        elif action == 1: # north
            if y == self.size_y - 1:
                return -10, False
            else:
                self.item_position = (x, y+1)
                return -1, False
        elif action == 2: # east
            if x == self.size_x - 1:
                return -10, False
            else:
                self.item_position = (x+1, y)
                return -1, False
        elif action == 3: # west
            if x == 0:
                return -10, False
            else:
                self.item_position = (x-1, y)
                return -1, False
        elif action == 4: # pickup
            if self.item_in_car:
                return -10, False
            elif self.item_position != (x, y):
                return -10, False
            else:
                self.item_in_car = True
                return 20, False
        elif action == 5: # drop-off
            if not self.item_in_car:
                return -10, False
            elif self.item_drop_off != (x, y):
                self.item_position = (x, y)
                return -20, False
            else:
                return 20, True

Then we can generate our Q-table by iterating over the task 1000 times (it is just an arbitrary number I chose). As you see, it simply just runs over the task again and again, but updates the Q-table with the “learnings” based on the reward.

states = field.get_number_of_states()
actions = 6

q_table = np.zeros((states, actions))

alpha = 0.1
gamma = 0.6
epsilon = 0.1

for i in range(1000):
    field = Field(size, item_start, item_drop_off, start_position)
    done = False
    steps = 0
    while not done:
        state = field.get_state()
        if random.uniform(0, 1) < epsilon:
            action = random.randrange(0, 6)
        else:
            action = np.argmax(q_table[state])

        reward, done = field.move_driver(action)
        next_state = field.get_state()

        old_value = q_table[state, action]
        next_max = np.max(q_table[next_state])

        new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)
        q_table[state, action] = new_value

        steps += 1

After that we can use it, our Q-table is updated. To test it, we will run the same code again, just with the updated Q-table.

alpha = 0.1
gamma = 0.6
epsilon = 0.1

field = Field(size, item_start, item_drop_off, start_position)
done = False
steps = 0
while not done:
    state = field.get_state()
    if random.uniform(0, 1) < epsilon:
        action = random.randrange(0, 6)
    else:
        action = np.argmax(q_table[state])

    reward, done = field.move_driver(action)
    next_state = field.get_state()

    old_value = q_table[state, action]
    next_max = np.max(q_table[next_state])

    new_value = (1 - alpha) * old_value + alpha * (reward + gamma * next_max)
    q_table[state, action] = new_value

    steps += 1

print(steps)

This resulted in 22 steps. That is awesome.

4 Easy Steps to Understand Unsupervised Machine Learning with an Example in Python

Step 1: Learn what is unsupervised machine learning?

An unsupervised machine learning model takes unlabelled (or categorised) data and lets the algorithm determined the answer for us.

Unsupervised Machine Learning model - takes unstructured data and finds patterns itself
Unsupervised Machine Learning model – takes unstructured data and finds patterns itself

The unsupervised machine learning model data without apparent structures and tries to identify some patterns itself to create categories.

Step 2: Understand the main types of unsupervised machine learning

There are two main types of unsupervised machine learning types.

  • Clustering: Is used for grouping data into categories without knowing any labels before hand.
  • Association: Is a rule-based for discovering interesting relations between variables in large databases.

In clustering the main algorithms used are K-means, hierarchy clustering, and hidden Markov model.

And in the association the main algorithm used are Apriori and FP-growth.

Step 3: How does K-means work

The K-means works in iterative steps

The k-means algorithm starts is an NP-hard problem, which mean there is no efficient way to solve in the general case. For this problem there are heuristics algorithms that converge fast to local optimum, which means you can find some optimum fast, but it might not be the best one, but often they can do just fine.

Enough, theory.

How does the algorithm work.

  • Step 1: Start by a set of k means. These can be chosen by taking k random point from the dataset (called the Random Partition initialisation method).
  • Step 2: Group each data point into the cluster of the nearest mean. Hence, each data point will be assigned to exactly one cluster.
  • Step 3: Recalculate the the means (also called centroids) to converge towards local optimum.

Steps 2 and 3 are repeated until the grouping in Step 2 does not change any more.

Step 4: A simple Python example with the k-means algorithm

In this example we are going to start assuming you have the basic knowledge how to install the needed libraries. If not, then see the following article.

First of, you need to import the needed libraries.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
from sklearn.cluster import KMeans

In the first basic example we are only going to plot some points on a graph.

style.use('ggplot')

x = [1, 2, 0.3, 9.2, 2.4,  9, 12]
y = [2, 4, 2.5, 8.5, 0.3, 11, 10]
plt.scatter(x, y)
plt.show()

The first line sets a style of the graph. Then we have the coordinates in the arrays x and y. This format is used to feed the scatter.

Output of the plot from scatter plotter in Python.
Output of the plot from scatter plotter in Python.

An advantage of plotting the points before you figure out how many clusters you want to use. Here it looks like there are two “groups” of plots, which translates into using to clusters.

To continue, we want to use the k means algorithm with two clusters.

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import style
from sklearn.cluster import KMeans

style.use('ggplot')

x = [1, 2, 0.3, 9.2, 2.4,  9, 12]
y = [2, 4, 2.5, 8.5, 0.3, 11, 10]

# We need to transform the input coordinates to plot use the k means algorithm
X = []
for i in range(len(x)):
    X.append([x[i], y[i]])
X = np.array(X)

# The number of clusters
kmeans = KMeans(n_clusters=2)
kmeans.fit(X)
labels = kmeans.labels_

# Then we want to have different colors for each type.
colors = ['g.', 'r.']
for i in range(len(X)):
    # And plot them one at the time
    plt.plot(X[i][0], X[i][1], colors[labels[i]], markersize=10)

# Plot the centres (or means)
plt.scatter(centroids[:, 0], centroids[:, 1], marker= "x", s=150, linewidths=5, zorder=10)
plt.show()

This results in the following result.

Example of k means algorithm used on simple dataset
Example of k means algorithm used on simple dataset

Considerations when using K-Means algorithm

We could have changed to use 3 clusters. That would have resulted in the following output.

Using 3 clusters instead of two in the k-mean algorithm
Using 3 clusters instead of two in the k-mean algorithm

This is not optimal for this dataset, but could be hard to predict without this visual representation of the dataset.

Uses of K-Means algorithm

Here are some interesting uses of the K-means algorithms:

  • Personalised marketing to users
  • Identifying fake news
  • Spam filter in your inbox