- Welcome to Lesson 12 Data Mining Twitter.
This lesson is based on chapter 12 of our professional book
Python for Programmers
and chapter 13 of our textbook
Intro to Python for Computer Science and Data Science.
In this lesson, we're going to focus on data mining
one of the biggest big data sources there is
which is Twitter.
Everyday, hundreds of millions of tweets are sent
and you actually can tap into that
and acquire as many as 1% of those tweets,
something on the order of 7.5 to 8 million tweets a day
are available to you if you would like.
So we're going to be working with a library called Tweepy
which makes it super easy to authenticate with Twitter.
You will need a Twitter developer account for this purpose
and you'll have to go ahead and set up what they call an app
which will give you the credentials you need
for interacting with Twitter
and you'll simply feed those into the Tweepy library.
It will do the authentication for you
making it really easy to connect.
And then from that point,
you can start interacting with the other Twitter APIs.
So some of the things that we're going to do in this lesson
we're going to search tweets from the past seven days
and you only have access to the past seven days
using the free APIs,
but there are services out there that you can pay
in order to gain access to the full tweet database
if you would like.
We're also going to show you
how to sample the livestream of tweets
and Twitter does that randomly
so you're never going to be guaranteed
to get the exact same tweets that somebody else gets
if they too are tapping into the livestream.
You will just get a random sample of the tweets
that match the criteria you happen to be searching for.
Also in this lesson,
we're going to take a look at the so called
tweet object meta data.
Every tweet that happens
also has a ton of additional information
that is packaged together with that tweet
into a JavaScript object notation object
which you get access to when you receive that tweet
from the Twitter APIs.
So we're going to take a look at some of that meta data
which includes things like who sent the tweet,
whether it was retweeted,
when it happened, where it happened,
and a lot more as well.
We'll also use some natural language processing
in this lesson.
We'll do that for both pre-processing tweets
to get them ready for analysis
and we'll also do some sentiment analysis
using the same TextBlob capabilities
that we introduced back in lesson 12.
In addition, we're going to show you
how to look at the Twitter trends.
It maintains trending topics worldwide
for about 450 different locations
and one of those is for the world at large
and then the others are for individual cities
and landmarks as well.
And we're going to finish off this lesson
by showing you how to locate the location information
in the context of the tweet meta data
and then we're going to actually plot tweets
on an interactive map
where you can click a map marker
to see the tweet that came from that location.