Master the NumPy Basics

What will we cover in this Tutorial

If you are starting from scratch with NumPy and do not know what ndarray is, then you should read this tutorial first.

  • How to make arithmetics with ndarray.
  • Sliding and indexing of ndarray with 1-dimension.
  • Sliding and indexing of ndarray with 2-dimensions.

Arithmetics with NumPy

An amazing feature with ndarrays is that you do not need to make forloops for simple operations.

import numpy as np

a1 = np.array([[1., 2., 3.], [3., 2., 1.]])
a2 = np.array([[4., 5., 6.], [6., 5., 4.]])

print(a1)
print(a2)

print(a2 - a1)
print(a1*a2)
print(1/a1)
print(a2**0.5)

This looks too good to be true. Right?

The output is as you would expect.

[[1. 2. 3.]
 [3. 2. 1.]]
[[4. 5. 6.]
 [6. 5. 4.]]
[[3. 3. 3.]
 [3. 3. 3.]]
[[ 4. 10. 18.]
 [18. 10.  4.]]
[[1.         0.5        0.33333333]
 [0.33333333 0.5        1.        ]]
[[2.         2.23606798 2.44948974]
 [2.44948974 2.23606798 2.        ]]

Then you understand why all are so madly in love with NumPy.

This type of “batch” operation is called vectorization.

You can also make comparisons.

import numpy as np

a1 = np.array([[1., 2., 3.], [6., 5., 4.]])
a2 = np.array([[4., 5., 6.], [3., 4., 5.]])

print(a1 < a2)

Which gives what you expect.

[[ True  True  True]
 [False False  True]]

At least I hope you would expect the above.

Slicing and basic indexing

If you are familiar with Python lists, then this should not surprise you.

import numpy as np

a = np.arange(10)
print(a)
print(a[5])
print(a[2:5])

You guessed it.

[0 1 2 3 4 5 6 7 8 9]
5
[2 3 4]

But this might surprise you a bit.

import numpy as np

a = np.arange(10)
print(a)
a[4:7] = 10
print(a)

Resulting in.

[0 1 2 3 4 5 6 7 8 9]
[ 0  1  2  3 10 10 10  7  8  9]

That is quite a surprise.

You can take a “view” from it (also called a slice) like the following example shows.

import numpy as np

a = np.arange(10)
print(a)
a_slice = a[4:7]
print(a_slice)
a_slice[0:1] = 30
print(a_slice)
print(a)

Resulting in.

[0 1 2 3 4 5 6 7 8 9]
[4 5 6]
[30  5  6]
[ 0  1  2  3 30  5  6  7  8  9]

Slicing and indexing of 2-dimensions

First of, this seems similar to Python lists.

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print(a[1])
print(a[2][2])
print(a[2, 2])

Maybe the last statement is surprising, but it does the same as the above. That is, the effect of a[2][2] is the same as of a[2, 2].

[4 5 6]
9
9

Slicing the above ndarray will be done by rows.

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print(a[:2])

Which results in the following.

[[1 2 3]
 [4 5 6]]

A bit more advanced to slice it as.

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print(a[:2, 1:])

Resulting in.

[[2 3]
 [5 6]]

It might not be clear the the second slice does fully vertical slices, which is illustrated by the following example.

import numpy as np

a = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])

print(a[:, :1])

This will most likely surprise you. Right?

[[1]
 [4]
 [7]]

Makes sense, right?

Quick NumPy Tutorial

What is NumPy?

NumPy is a scientific library that provides multidimensional array object with fast routines on it. NumPy is short for Numerical Python.

When we talk about NumPy often we refer to the powerful ndarray, which is the multidimensional array (N-dimensional array).

A few comparisons between Python lists and ndarray.

ndarrayPython list
Have fixed size at creation.Is dynamic. You can add and remove elements.
All elements have the same type.Elements have type independent of each other.
Can execute fast mathematical operations with simple syntax.Need loops to make operations on each element.
Comparison between Numpy and Python list

Examples showing the difference: Fixed after creation

The Numpy is imported by default imported by import numpy as np. To create a ndarray, you can use the array call as defined below.

import numpy as np

data = np.array([[1, 2, 3], [1, 2, 3]])
print(data)

Which will create an 2 dimensional array object with 2 times 3 elements.

[[1 2 3]
 [1 2 3]]

That will be a fixed sized ndarray. You cannot add new dimensions or elements to the the single arrays.

A Python list is a more flexible.

my_list = []
my_list.append(2)
my_list.append(4)
my_list.remove(2)
print(my_list)

Which demonstrates the flexibility and power of Python lists. It is simple to add and remove elements. The above code will result in the following output.

[4]

Examples showing the difference: One type

The type of a ndarray is stored in dtype. Interesting thing is that each element must have the same type.

import numpy as np

data = np.random.randn(2, 3)

print(data)
print(data.dtype)

It will result in a random ndarray of type float64.

[[-0.85925182 -0.89247774 -2.40920842]
 [ 0.84647869  0.27631307 -0.80772023]]
float64

An interesting way to demonstrate that only one type can be present in an ndarray, is by trying to create it with a mixture of ints and floats.

import numpy as np

data = np.array([[1.0, 2, 3], [1, 2, 3]])
print(data)
print(data.dtype)

As the first element is of type float they are all cast to float64.

[[1. 2. 3.]
 [1. 2. 3.]]
float64

While the following list is valid.

my_list = [1.0, 2, 3]
print(my_list)

Where the first element will be float the second and third element are ints.

[1.0, 2, 3]

Examples showing the difference: No loops needed

Many operations can be made directly on the ndarray.

import numpy as np

data = np.random.randn(2, 3)

print(data)
print(data*10)
print(data + data)

Which will result in the following output.

[[ 1.18303358 -2.20017954  0.46294824]
 [-0.56508587  0.0990272  -1.8431866 ]]
[[ 11.83033584 -22.00179538   4.62948243]
 [ -5.65085867   0.990272   -18.43186601]]
[[ 2.36606717 -4.40035908  0.92589649]
 [-1.13017173  0.1980544  -3.6863732 ]]

Expected right? But easy to multiply and add out of the box.

Similar of the Python list would be.

my_list = [1, 2, 3]
for i in range(len(my_list)):
    my_list[i] *= 10

for i in range(len(my_list)):
    my_list[i] += my_list[i]

And it is not even the same, as you write it directly to the old elements.

Another way to compare differences

It might at first glance seem like ndarrays are inflexible with all the restrictions comparing the Python lists. Yes, that is true, but the benefit is the speed.

import time
import numpy as np

my_arr = np.arange(1000000)
my_list = list(range(1000000))

start = time.time()
for _ in range(10): my_arr2 = my_arr * 2
end = time.time()
print(end - start)

start = time.time()
for _ in range(10): my_list2 = [x * 2 for x in my_list]
end = time.time()
print(end - start)

Which resulted in.

0.03456306457519531
0.9373760223388672

The advantage is that ndarrays are 10-100 times faster than Python lists, which makes a considerable impact on scientific calculations.

Understand Lambda Function with a Simple Example in Python

What is a lambda function?

A lambda function is often called an anonymous function, which you will understand in a moment. The difference between a simple function and a lambda function is small.

Let’s try with an example to explain it.

def multiply_two(x):
    return 2*x

print(multiply_two(10))

my_lambda = lambda x: 2*x

print(my_lambda(10))

Which both will return 20. The first one, multiply_two, is a simple function that has a return statement. The lambda function does the same, just defined differently.

Looking at the lambda function.

my_lambda = lambda x: 2*x

You see the structure that you use the key-word lambda and what it takes of input x and what it returns 2*x. In this case we assigned the lambda function to a variable my_lambda. This is not necessary as the use-case below will show.

Why use lambda functions?

It can be convenient to use lambda functions in cases where you want to make simple operations.

Again, a simple example will demonstrate it.

def apply_to_list(the_list, f):
    return [f(x) for x in the_list]


the_list = [6, 3, 9, 1, 4, 2, 8]
print(apply_to_list(the_list, lambda x: x + 5))
print(apply_to_list(the_list, lambda x: x*2))

Which will return.

[11, 8, 14, 6, 9, 7, 13]
[12, 6, 18, 2, 8, 4, 16]

That is nice. You can easily define small functions to make simple functionality you might only need once.

When you understand the syntax of lambda functions it is easy.

Alternatively, you could use the following code the get the same result without using the lambda function.

print([x + 5 for x in the_list])
print([x*2 for x in the_list])

The simplicity of these examples are only meant to teach that lambda functions gives you some flexibility in your programs.

Binary Search Explained and Implemented in Python

Understand binary serach

The whole idea behind binary search is that you can take advantage of having the list you search in ordered.

Binary search explained

Say, we need to search for 7 and we look at the element in the middle of the list.

Binary search explained

Then we can conclude, in this example, that 7 is not part of the right side of the list, as all numbers must be greater than 10. That is because the list is ordered.

Next we ask the in the middle of the left side of the list which is left unknown is 7 is there.

Binary search explained

As -9 is less than 7, we know that 7 cannot be before in the list and are left with the remaining element between -9 and 10.

Binary search explained

As -3 is less than 7, we know that if 7 is part of the list, then it must be to the right of -3. Also, we know it must be before 10 (our first comparison).

Binary search explained

Hence, it can only be in the last spot left. But as it is 8, we now know that 7 is not part of the list.

Why is that impressive?

Consider if the list was unsorted. Then you would have to look through the entire list to make sure 7 was not part of it.

Binary search explained

In terms of complexity that means if the list contains N element, it must make N comparisons to search of an element. That is O(N) time complexity.

The binary search on the other hand is way more efficient. For each comparison the algorithm can skip one half of the list. That is O(log(N)) time complexity.

The source code

def recursive_binary_search(my_list, element):
    return recursive_binary_search_internal(my_list, element, 0, len(my_list) - 1)


def recursive_binary_search_internal(my_list, element, low, high):
    if low > high:
        return False
    else:
        mid = (low + high)//2
        if my_list[mid] == element:
            return True
        else:
            if my_list[mid] > element:
                return recursive_binary_search_internal(my_list, element, low, mid - 1)
            else:
                return recursive_binary_search_internal(my_list, element, mid + 1, high)


def binary_search(my_list, element):
    low = 0
    high = len(my_list) - 1
    while low <= high:
        mid = (low + high)//2
        if my_list[mid] == element:
            return True
        else:
            if my_list[mid] > element:
                high = mid - 1
            else:
                low = mid + 1
    return False


def main():
    my_list = [-29, -16, -15, -9, -6, -3, 8, 10, 17, 19, 27, 47, 54, 56, 60]
    print(my_list)
    element = 56
    print("Binary Search:", binary_search(my_list, element), element)
    print("Recursive Binary Search:", recursive_binary_search(my_list, element), element)


if __name__ == "__main__":
    main()

There are two implementations of the binary search in the above example.

Want to learn how to sort a list?

For the insertion sort read this tutorial. For the merge sort read this tutorial.

A Simple Implementation of Merge Sort in Python

Understand Merge Sort in 6 Minutes

Understand Merge Sort in 6 Minutes

Merge sort is one of the algorithms you need to master. Why?

Because it is in the class of efficient algorithms and is easy to understand.

But what does efficient mean?

Let’s get back to that. First how does Merge Sort work?

Merge Sort explained

It takes the list and breaks it down into two sub lists. Then it takes these sublists and break them down into two. This process continues until there is only 1 element in each sublist.

And a list containing only 1 element, is a sorted list.

Merge Sort explained

It then takes two sublists and merge them together. Notice, that each of these sublists (in the first place, the sublists only contain 1 element each) are sorted.

Then it is effective to merge them together sorted. The algorithm looks at the first element of each sorted sublist and takes the smaller element first.

Merge Sort explained

This process continues all the way down.

Then the next row is taken of sublists. Again, the sample algorithm is used to merge them together. Take the smaller element of the two and add them to the new list. This continues.

Merge Sort explained

This process continues until we end up with one list.

Merge Sort explained

Which by magic (or the logic behind the algorithm) is sorted.

Time complexity

Well, we talked about it is one of the efficient sorting algorithm. That means it runs in O(N log(N)) time.

That means, if you have a list of N unsorted elements, it will take N log(N) operations.

Is that true for Merge Sort?

How many layers do you have in the algorithm?

Well, for each layer you half the size of each sublist. You can do that log(N) times.

For each layer, you do N comparisons. That results in N log(N) operations, hence, the O(N log(N)) time complexity.

The implementation of Merge Sort in Python

def merge_sort(my_list):
    if len(my_list) <= 1:
        return my_list

    mid = len(my_list)//2
    left_list = my_list[:mid]
    right_list = my_list[mid:]

    merge_sort(left_list)
    merge_sort(right_list)

    index_left = 0
    index_right = 0
    index_main = 0

    while index_left < len(left_list) and index_right < len(right_list):
        if right_list[index_right] < left_list[index_left]:
            my_list[index_main] = right_list[index_right]
            index_right += 1
            index_main += 1
        else:
            my_list[index_main] = left_list[index_left]
            index_left += 1
            index_main += 1

    while index_left < len(left_list):
        my_list[index_main] = left_list[index_left]
        index_left += 1
        index_main += 1

    while index_right < len(right_list):
        my_list[index_main] = right_list[index_right]
        index_right += 1
        index_main += 1


def main():
    my_list = [19, 56, 8, -6, -3, 27, -9, -29]
    print(my_list)
    merge_sort(my_list)
    print(my_list)


if __name__ == "__main__":
    main()

That is awesome.

Want to learn more about sorting. Check out the Insertion Sort, which is also one of the sorting algorithms you need to master. It is not as efficient, but it has one advantage you need to understand.

Master Insertion Sort and Implement it in Python

Understand the algorithm

Understand Insertion Sort in 6 Minutes

Insertion sort is an awesome algorithm that has some quite interesting use-cases due to it nature.

First understand the basics of it.

Consider the list of integers.

Insertion sort explained

The above list has 10 elements. Now if you only consider the list of the first element (number 3), then that part is actually sorted. If number 3 was the only number in the list, then it would be sorted.

Now consider the list of the first two elements.

Insertion sort explained

Then we have the list of two elements, 3 and 5. Here we are lucky, as this is still sorted.

Now for the next element, 1, we are a bit out of luck, because if we just added that element to part of the list (the green part), then it would not be sorted.

Insertion sort explained

If one was added, we would need to swap it down until it is on place. First swapping 1 with 5.

Insertion sort explained

Then swapping 1 with 3. Then we are having a ordered list again, that is a list of the 3 first elements (the green part of the list below).

Insertion sort explained

Hence, the process is to think of the first element as a list and realize it is ordered. Then insert one element at the time. For each element inserted, swap it down until it is on place.

Why is Insertion Sort great to know?

Well, first of all it is simple to understand. Second of all, it is easy to implement (see the code below). But that is not the exiting stuff.

Well, what is it?

It has some interesting use case. See.

  • If you were to keep an ordered list in memory as part of your program.
  • But it also had a twist.
  • Someone would keep coming with new numbers all the time, and still ask you to keep the list ordered.
  • Then the efficiency of keeping this list ordered is good.

Why?

Because if the list is ordered and you need to add another element, you just follow the above procedure.

The worst case run-time for a list of N element is O(N).

What is the catch?

Well, the performance of the algorithm is not good. That is, there are more efficient algorithms out there to sort N elements.

Let’s analyze.

For the first element you use 1 operation. The second element 2 operations. The third element 3 operations. Wait a minute, isn’t that this formula?

1 + 2 + 3 + 4 + … + N = N(N+1)/2

Which unfortunately gives a performance of O(N^2).

So why bother about this algorithm?

Well, because of the use-case explained above. That you can keep it ordered and with low cost insert an element into it.

The code

import random


def generate_random_list(n):
    # n is the length of the list
    my_list = []
    for i in range(n):
        my_list.append(random.randint(0, 4*n))
    return my_list


def insertion_sort(my_list):
    for i in range(1, len(my_list)):
        for j in range(i, 0, -1):
            if my_list[j] < my_list[j-1]:
                my_list[j], my_list[j-1] = my_list[j-1], my_list[j]
            else:
                break


def main():
    my_list = generate_random_list(20)
    print(my_list)
    insertion_sort(my_list)
    print(my_list)


if __name__ == "__main__":
    main()

The above code implements the algorithm (see only 7 lines of code) and tests it.

I hope you enjoyed.

How to Setup an Automated Bitly URL-shortener in Python in 3 Easy Steps

What will we cover in this tutorial?

  • To get you started from scratch to use Bitly in Python.
  • How to register at Bitly and get your Authentication token to access the Bitly API v4.
  • To get the Python library to connect to the API.
  • A walk-through of the API functionality available.

Step 1: Register at Bitly and get your Access Token

The first thing you need is to register at Bitly to use their functionality.

You can do that be follow this link.

Press the Sign up at Bitly

You should press the Sign up at the right upper corner.

Scroll down and press the Get Started for Free.

Press Get Started for Free

Then fill out the fields with your preferred credentials.

Fill out with your credentials

You should receive an email in the inbox of the email address you provided. Click the link in that email.

You should be forwarded to Bitly and then you should find your profile settings.

When clicking the Profile Settings you should find the Generic Access Token.

You will be asked to fill out your password, then press Generate Token.

Then you should get an access token like this one (this one is not valid, so you cannot use it):

d9742171bc53c4847c76ec2d02f7b83466aedd8d

Step 2: The URL-shortener library in Python

Luckily someone else did all the complicated work to use the Bitly API. All you need to do in Python is to get the library. The library is called pyshortener, which is a library to wrap and consume the most used shorteners APIs. Hence, you can use it with many URL shorteners. Please notice, that the access token you have generated in this tutorial is only valid for Bitly.

To install that library simply type the following command in your shell.

pip install pyshortener

You get a full list of the supported URL-shorteners in their documentation.

Step 3: Use the URL-shortener library

You get access to your URL-shortener functionality at Bitly by the following setup.

import pyshorteners
s = pyshorteners.Shortener(api_key='YOUR_KEY')

You should obviously use replace ‘YOUR_KEY’ with the key you obtained in Step 1.

Then you can create a Bitly shortened URL with the following call.

import pyshorteners


ACCESS_TOKEN= "YOUR_KEY"

s = pyshorteners.Shortener(api_key=ACCESS_TOKEN)
url = s.bitly.short("http://www.learnpythonwithrune.org/how-to-setup-an-automated-bitly-url-shortener-in-python-in-3-easy-steps/")
print(url)

Which in my example gave me the following link.

https://bit.ly/2NLc1Zq

This should also populate your web API at Bitly.

You can also retrieve how many clicks have been made on your link.

import pyshorteners


ACCESS_TOKEN= "YOUR_KEY"

s = pyshorteners.Shortener(api_key=ACCESS_TOKEN)
clicks = s.bitly.total_clicks("https://bit.ly/2NLc1Zq")
print(clicks)

Which in this case (surprisingly) returned 0 clicks.

Finally, you can also expand on a URL you already created a Bitly link on.

import pyshorteners


ACCESS_TOKEN= "YOUR_KEY"

s = pyshorteners.Shortener(api_key=ACCESS_TOKEN)
expand = s.bitly.expand("https://bit.ly/2NLc1Zq")
print(expand)

Which will result in the URL you created it with.

http://www.learnpythonwithrune.org/how-to-setup-an-automated-bitly-url-shortener-in-python-in-3-easy-steps/

That is it. See the full documentation of it here of how to use Bitly in Python.

New to Python? Also check out my Free interactive online tutorial for beginners.