What will we cover in this tutorial?
How to find all the locations of your followers on Twitter and create a choropleth map (maps where the color of each shape is based on the value of an associated variable) with all countries. This will all be done by using Python.
This is done in connection with my interest of where the followers are from on my Twitter account. Today my result looks like this.
Step 1: How to get the followers from your Twitter account
If you are new to Twitter API you will need to create a developer account to get your secret key. You can follow this tutorial to create you developer account and get the needed tokens.
When that is done, you can use the tweepy library to connect to the Twitter API. The library function api.followers_ids(api.me().id) will give you a list of all your followers by user-id.
import tweepy
# Used to connect to the Twitter API
def get_twitter_api():
# You need your own keys/secret/tokens here
consumer_key = "--- INSERT YOUR KEY HERE ---"
consumer_secret = "--- INSERT YOUR SECRET HERE ---"
access_token = "--- INSERT YOUR TOKEN HERE ---"
access_token_secret = "--- INSERT YOUR TOKEN SECRET HERE ---"
# authentication of consumer key and secret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# authentication of access token and secret
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
return api
# This function is used to process it all
def process():
# Connecting to the twitter api
api = get_twitter_api()
# Get the list of all your followers - it only gives user-id's
# - we need to gather all user data after
# - This only returns first 5,000 followers, then you need to cursor to get more.
followers = api.get_follower_ids()
print("Followers", len(followers))
if __name__ == "__main__":
process()
Which will print out the number of followers you have on your account.
Step 2: Get the location of your followers
How do we transform the twitter user-ids to a location?
We need to look them all up. Luckily, not one-by-one. We can do it in chunks of 100 users per call.
The function api.lookup_users(…) can lookup 100 users per call with users-ids or user-names.
import tweepy
# Used to connect to the Twitter API
def get_twitter_api():
# You need your own keys/secret/tokens here
consumer_key = "--- INSERT YOUR KEY HERE ---"
consumer_secret = "--- INSERT YOUR SECRET HERE ---"
access_token = "--- INSERT YOUR TOKEN HERE ---"
access_token_secret = "--- INSERT YOUR TOKEN SECRET HERE ---"
# authentication of consumer key and secret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# authentication of access token and secret
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
return api
# This function is used to process it all
def process():
# Connecting to the twitter api
api = get_twitter_api()
# Get the list of all your followers - it only gives user-id's
# - we need to gather all user data after
# - This only returns first 5,000 followers, then you need to cursor to get more.
followers = api.get_follower_ids()
print("Followers", len(followers))
# We need to chunk it up in sizes of 100 (max for api.lookup_users)
followers_chunks = [followers[i:i + 100] for i in range(0, len(followers), 100)]
# Process each chunk - we can call for 100 users per call
for follower_chunk in followers_chunks:
# Get a list of users (with location data)
users = api.lookup_users(user_ids=follower_chunk)
# Process each user to get location
for user in users:
# Print user location
print(user.location)
if __name__ == "__main__":
process()
Before you execute this code, you should now it will print all the locations that all your followers have set.
Step 3: Map all user locations to the same format
When users write their locations, it is done in various ways. As this example shows.
India
Kenya
Temecula, CA
Atlanta, GA
Florida, United States
Hyderabad, India
Atlanta, GA
Agadir / Khouribga, Morocco
Miami, FL
Republic of the Philippines
Tampa, FL
Sammamish, WA
Coffee-machine
And as the last example shows, it might not be a real location. Hence, we need to see if we can find the location by asking a service. For this purpose, we will use the GeoPy library, which is a client for several popular geocoding web services.
Hence, for each of the user specified locations (as the examples above) we will call GeoPy and use the result from it as the location. This will bring everything in the same format or clarify if the location exists.
import tweepy
from geopy.exc import GeocoderTimedOut
from geopy.geocoders import Nominatim
# Used to connect to the Twitter API
def get_twitter_api():
# You need your own keys/secret/tokens here
consumer_key = "--- INSERT YOUR KEY HERE ---"
consumer_secret = "--- INSERT YOUR SECRET HERE ---"
access_token = "--- INSERT YOUR TOKEN HERE ---"
access_token_secret = "--- INSERT YOUR TOKEN SECRET HERE ---"
# authentication of consumer key and secret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# authentication of access token and secret
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
return api
# Used to map the twitter user location description to a standard format
def lookup_location(location):
geo_locator = Nominatim(user_agent="LearnPython")
try:
location = geo_locator.geocode(location, language='en')
except GeocoderTimedOut:
return None
return location
# This function is used to process it all
def process():
# Connecting to the twitter api
api = get_twitter_api()
# Get the list of all your followers - it only gives user-id's
# - we need to gather all user data after
# - This only returns first 5,000 followers, then you need to cursor to get more.
followers = api.get_follower_ids()
print("Followers", len(followers))
# Used to store all the locations from users
locations = {}
# We need to chunk it up in sizes of 100 (max for api.lookup_users)
followers_chunks = [followers[i:i + 100] for i in range(0, len(followers), 100)]
# Process each chunk - we can call for 100 users per call
for follower_chunk in followers_chunks:
# Get a list of users (with location data)
users = api.lookup_users(user_ids=follower_chunk)
# Process each user to get location
for user in users:
# Call used to transform users description of location to same format
location = lookup_location(user.location)
# Add it to our counter
if location:
location = location.address
location = location.split(',')[-1].strip()
if location in locations:
locations[location] += 1
else:
locations[location] = 1
if __name__ == "__main__":
process()
As you see, it will count the occurrences of each location found. The split and strip is used to get the country and leave out the rest of the address if any.
Step 4: Reformat the locations into a Pandas DataFrame
We want to reformat the locations into a DataFrame to be able to join (merge) it with GeoPandas, which contains the choropleth map we want to use.
To convert the locations into a DataFrame we need to restructure it. This will also helps us to remove duplicates. As an example, United States and United States of America both appear. To handle that we will map all country names to a 3 letter code. We will use the pycountry library for that.
import tweepy
import pycountry
import pandas as pd
from geopy.exc import GeocoderTimedOut
from geopy.geocoders import Nominatim
# Used to connect to the Twitter API
def get_twitter_api():
# You need your own keys/secret/tokens here
consumer_key = "--- INSERT YOUR KEY HERE ---"
consumer_secret = "--- INSERT YOUR SECRET HERE ---"
access_token = "--- INSERT YOUR TOKEN HERE ---"
access_token_secret = "--- INSERT YOUR TOKEN SECRET HERE ---"
# authentication of consumer key and secret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# authentication of access token and secret
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
return api
# Helper function to map country names to alpha_3 representation
# Some are not supported - and are hard-coded in
# Function used to map country names from GeoPandas and the country names from geo_locator
def lookup_country_code(country):
try:
alpha_3 = pycountry.countries.lookup(country).alpha_3
return alpha_3
except LookupError:
if country == 'The Netherlands':
country = 'NLD'
elif country == 'Democratic Republic of the Congo':
country = 'COG'
return country
# Used to map the twitter user location description to a standard format
def lookup_location(location):
geo_locator = Nominatim(user_agent="LearnPython")
try:
location = geo_locator.geocode(location, language='en')
except GeocoderTimedOut:
return None
return location
# This function is used to process it all
def process():
# Connecting to the twitter api
api = get_twitter_api()
# Get the list of all your followers - it only gives user-id's
# - we need to gather all user data after
# - This only returns first 5,000 followers, then you need to cursor to get more.
followers = api.get_follower_ids()
print("Followers", len(followers))
# Used to store all the locations from users
locations = {}
# We need to chunk it up in sizes of 100 (max for api.lookup_users)
followers_chunks = [followers[i:i + 100] for i in range(0, len(followers), 100)]
# Process each chunk - we can call for 100 users per call
for follower_chunk in followers_chunks:
# Get a list of users (with location data)
users = api.lookup_users(user_ids=follower_chunk)
# Process each user to get location
for user in users:
# Call used to transform users description of location to same format
location = lookup_location(user.location)
# Add it to our counter
if location:
location = location.address
location = location.split(',')[-1].strip()
if location in locations:
locations[location] += 1
else:
locations[location] = 1
# We reformat the output fo locations
# Done for two reasons
# - 1) Some locations have two entries (e.g., United States and United States of America)
# - 2) To map them into a simple format to join it with GeoPandas
reformat = {'alpha_3': [], 'followers': []}
for location in locations:
print(location, locations[location])
loc = lookup_country_code(location)
if loc in reformat['alpha_3']:
index = reformat['alpha_3'].index(loc)
reformat['followers'][index] += locations[location]
else:
reformat['alpha_3'].append(loc)
reformat['followers'].append(locations[location])
# Convert the reformat into a dictionary to join (merge) with GeoPandas
followers = pd.DataFrame.from_dict(reformat)
pd.set_option('display.max_columns', 50)
pd.set_option('display.width', 1000)
pd.set_option('display.max_rows', 300)
print(followers.sort_values(by=['followers'], ascending=False))
if __name__ == "__main__":
process()
That makes it ready to join (merge) with GeoPandas.
Step 5: Merge it with GeoPandas and show the choropleth map
Now for the fun part. We only need to load the geo data from GeoPandas and merge our newly created DataFrame with it. Finally, plot and show it using matplotlib.pyplot.
import tweepy
import pycountry
import pandas as pd
import geopandas
import matplotlib.pyplot as plt
from geopy.exc import GeocoderTimedOut
from geopy.geocoders import Nominatim
# Used to connect to the Twitter API
def get_twitter_api():
# You need your own keys/secret/tokens here
consumer_key = "--- INSERT YOUR KEY HERE ---"
consumer_secret = "--- INSERT YOUR SECRET HERE ---"
access_token = "--- INSERT YOUR TOKEN HERE ---"
access_token_secret = "--- INSERT YOUR TOKEN SECRET HERE ---"
# authentication of consumer key and secret
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
# authentication of access token and secret
auth.set_access_token(access_token, access_token_secret)
api = tweepy.API(auth, wait_on_rate_limit=True)
return api
# Helper function to map country names to alpha_3 representation
# Some are not supported - and are hard-coded in
# Function used to map country names from GeoPandas and the country names from geo_locator
def lookup_country_code(country):
try:
alpha_3 = pycountry.countries.lookup(country).alpha_3
return alpha_3
except LookupError:
if country == 'The Netherlands':
country = 'NLD'
elif country == 'Democratic Republic of the Congo':
country = 'COG'
return country
# Used to map the twitter user location description to a standard format
def lookup_location(location):
geo_locator = Nominatim(user_agent="LearnPython")
try:
location = geo_locator.geocode(location, language='en')
except GeocoderTimedOut:
return None
return location
# This function is used to process it all
def process():
# Connecting to the twitter api
api = get_twitter_api()
# Get the list of all your followers - it only gives user-id's
# - we need to gather all user data after
# - This only returns first 5,000 followers, then you need to cursor to get more.
followers = api.get_follower_ids()
print("Followers", len(followers))
# Used to store all the locations from users
locations = {}
# We need to chunk it up in sizes of 100 (max for api.lookup_users)
followers_chunks = [followers[i:i + 100] for i in range(0, len(followers), 100)]
# Process each chunk - we can call for 100 users per call
for follower_chunk in followers_chunks:
# Get a list of users (with location data)
users = api.lookup_users(user_ids=follower_chunk)
# Process each user to get location
for user in users:
# Call used to transform users description of location to same format
location = lookup_location(user.location)
# Add it to our counter
if location:
location = location.address
location = location.split(',')[-1].strip()
if location in locations:
locations[location] += 1
else:
locations[location] = 1
# We reformat the output fo locations
# Done for two reasons
# - 1) Some locations have two entries (e.g., United States and United States of America)
# - 2) To map them into a simple format to join it with GeoPandas
reformat = {'alpha_3': [], 'followers': []}
for location in locations:
print(location, locations[location])
loc = lookup_country_code(location)
if loc in reformat['alpha_3']:
index = reformat['alpha_3'].index(loc)
reformat['followers'][index] += locations[location]
else:
reformat['alpha_3'].append(loc)
reformat['followers'].append(locations[location])
# Convert the reformat into a dictionary to join (merge) with GeoPandas
followers = pd.DataFrame.from_dict(reformat)
pd.set_option('display.max_columns', 50)
pd.set_option('display.width', 1000)
pd.set_option('display.max_rows', 300)
print(followers.sort_values(by=['followers'], ascending=False))
# Read the GeoPandas
world = geopandas.read_file(geopandas.datasets.get_path('naturalearth_lowres'))
# Remove the columns not needed
world = world.drop(['pop_est', 'continent', 'iso_a3', 'gdp_md_est'], axis=1)
# Map the same naming convention as followers (the above DataFrame)
# - this step is needed, because the iso_a3 column was missing a few countries
world['iso_a3'] = world.apply(lambda row: lookup_country_code(row['name']), axis=1)
# Merge the tables (DataFrames)
table = world.merge(followers, how="left", left_on=['iso_a3'], right_on=['alpha_3'])
# Plot the data in a graph
table.plot(column='followers', figsize=(8, 6))
plt.show()
if __name__ == "__main__":
process()
Resulting in the following output (for PythonWithRune twitter account (not yours)).
