- [Instructor] For our next example we're going to use
a live tweet stream to perform sentiment analysis.
Now, before I talk about this in detail
let me just go ahead to the command line
where I'm going to execute a script
called sentimentlistener.py.
For this script you can provide a topic
that you would like to track,
and you can specify the total number
of tweets that you want to process,
and what it will then do is start up
a live tweet stream tracking that topic,
and once those 10, in this case, tweets come in
it will summarize how many positive tweets there were,
how many negative tweets there were,
and how many neutral tweets there were
based on the values returned by TextBlob.
So, I'm gonna start that executing
so we can see tweets start to come in.
Again, I'm using marvel as my topic at the moment,
and it's going to precede each tweet by a plus sign
if it's positive, a minus sign if it's negative,
and nothing but a space character if it's
considered to be a neutral tweet.
Now, while those tweets come in
let me just jump back over to my slide here.
So, one of the things that you might use something
like for is dynamic sentiment analysis.
So, I could potentially be receiving these tweets,
figuring out if they're positive, negative, or neutral,
and maybe graphing that information
in a dynamic visualization that shows
what's going on right up to the minute.
So, for instance political researchers
might use that during elections
to see how people are feeling about specific politicians
and topics that might help they determine
how those folks are likely to vote as well.
Companies might use something like this
to see what people are saying about their products
and other people's products and use that
to their competitive advantage.
And as you saw, the script that I'm executing
is going to enable you to check the sentiment
on whatever specified topic you
provide as a command line argument
and the number of tweets that you specify as well.
So, let me jump back out, and as you can see,
the script terminated with four positive, four neutral,
and two negative tweets in this particular case,
and one thing to keep in mind as you start
looking through these tweets here
is that even though they're marked
as positive, negative, or neutral,
these are tweets and tweets don't come
in full sentences a lot of the time,
so it may be difficult for a tool like TextBlob
with its default sentiment analyzer
to truly get a sense of the sentiment in certain tweets
based on what their content actually is.
So, you may want to try this out, for example,
with the Naive Bayes sentiment analyzer
that we demonstrated back in the NLP chapter.
In our case in this example we're working
with the default sentiment analyzer in TextBlob,
and you may find that there are other
sentiment analysis tools out there that are better geared
towards detecting sentiment in tweets,
and similarly, you know, when you're
dealing with other types of text
there may be better tools out there
for certain kinds of text that
you want to analyze and manipulate.
So, now that we saw the script in action
let me jump into a text editor here and show you the script.
Now, just briefly overviewing the script first
before we look at it in detail.
Of course we have the imports for anything
that we're going to use in this script file.
We have our class definition that inherits
from StreamListener so that we can be
notified as tweets arrive in our app.
We have the methods of class SentimentListener,
the initialization method, the on_status method
to receive each tweet, and as I scroll down here
you see after the class we have
a main function that we defined,
which is where we specify, in this case,
the logic of the application itself,
and at the very bottom of the file we have an if statement
that I introduced to you in an earlier lesson,
and just as a matter of review,
recall that when you execute any .py file as a script
there is a global variable called __name__,
and its value is going to be set to the string __main__.
So, this if statement's body will execute
only if I run sentimentlistener.py as a script,
which is what I just demonstrated
to you at the command line.
In that case, we will call the main function.
The main function will set up our authorization,
authentication, excuse me, and then
perform the logic of the application,
and as part of that it will be using
an object of our SentimentListener class.
So, taking it from the top now in more detail,
we're importing our keys.py file
because we need all the information
for authenticating with the Twitter APIs.
We're importing the preprocessor module,
which we demonstrated a few videos back for cleaning tweets,
and we'll use that in this example.
We're using the sys module to access
the command line arguments to the script,
which are going to be the text that we want to track
and the number of tweets to receive
before terminating the livestream,
and we're also importing TextBlob
because we're going to take advantage
of its builtin capabilities for sentiment analysis.
And then finally, of course, we're importing tweepy
because we need to use that to access the Twitter APIs.
So, just as we did in the preceding example,
we're creating a new subclass
of StreamListener from the tweepy module.
We're calling this one SentimentListener,
and again, it's going to handle an incoming tweet stream,
this time checking the tweets for sentiment.
Now, you'll notice that we have a number of arguments
that we are receiving into our init method.
These arguments are going to represent
the tweepy API object.
This is going to be a dictionary
that keeps track of positive sentiment,
neutral sentiment, and negative sentiment.
We're also going to receive a string
that represents the topic that we're tracking,
and we're going to receive a limit argument
specifying the total number of tweets to process.
If we don't specify a value for that, it would be 10.
As you'll see, we're going to use
a command line argument in this case,
so whatever that argument is is what we will supply
as the last argument to the init method.
Now we're going to initialize a number of instance variables
in our SentimentListener object that we create.
We'll store the sentiment dictionary
so that while those tweets are coming in
we can keep updating that dictionary.
We will keep track of how many tweets we've processed
so we know when to terminate the stream,
and we'll use the TWEET_LIMIT to help us with that as well.
And we will store the topic that we are looking for also.
The reason for that is we're going
to look at every tweet that comes in,
and if the actual topic is not in the tweet's text
we're going to ignore that tweet in this example.
So, we're only going to show you tweets
that actually physically contain
the topic in the actual tweet_text,
not necessarily in all of the other
metadata in this example.
Now, we also chose to set the preprocessor
module's options to say that we're going
to drop out URLs and any Twitter RESERVED words
that appear in tweets as we clean them,
and then finally, as we did in the last example as well,
we take the tweepy API object and hand that off
to the superclass StreamListener's init method
because the StreamListener class
has all sorts of builtin capability
that is dependent on using that API object.
Now, separately we have our on_status method,
and again, this is the method that's going
to get called every time we receive a status object
from Twitter representing a given tweet,
and as we did in the preceding example
we are once again going to try to get the full_text
of an extended_tweet, which is one
that has 141 to 280 characters.
If that fails we'll get an exception,
in which case we will just fall back
to the default text attribute of the status object,
and that's going to represent a tweet up to 140,
or up through 140 characters.
Now, if the particular tweet that we receive is a retweet
we are simply going to ignore it.
We want 10 unique tweets for this example,
or whatever number of tweets you specify.
We want them to be unique tweets
that have positive, negative, or neutral sentiment.
If it's a bunch of retweets, and it could be
many of them if you're dealing with a viral topic,
you may get all the same value very quickly,
so we're ignoring any tweet that contains RT
at the beginning of the tweet,
which indicates that it's a retweet.
Separately we're going to clean the text.
We're going to get rid of any URLs
and other Twitter keywords that, like "fav,"
F-A-V, that might be inside of that tweet.
And then we're going to check to see if the topic
that we're searching for is contained in the tweet_text.
So, if the topic in its lowercase form
is not in the tweet_text in its lowercase form,
then we're going to ignore that tweet as well.
So, if I search for marvel, I want the word marvel
to appear in every single tweet in this particular example.
Now, once we have gotten to this point
we have a tweet that we are going to process,
so we create a TextBlob out of it,
and you may recall from the preceding lesson
that every TextBlob has a sentiment attribute,
and inside that sentiment attribute is a polarity value,
which is going to be greater
than zero for positive sentiment,
less, equal to zero for neutral sentiment,
and less than zero for negative sentiment.
So, we're going to check that polarity,
and if it's a positive value we're going
to set the sentiment variable to a plus sign,
which we'll use as part of what we display with the tweet.
We're going to go into the dictionary that we stored
and set its positive keys value
to whatever the value was previously plus one,
so we're gonna modify the value
associated with the positive key.
If the sentiment is neutral we'll do the same thing,
but we'll use a space character to indicate
neutral sentiment, and we'll add one
to the neutral counter, if you will,
and then finally, if it's not positive
or neutral it has to be negative,
so for the else part we'll increment
the negative sentiment key value pair
and we'll use a minus sign to indicate negative sentiment
in the context of the output of this application.
Now, at that point we're going to display the tweet.
We're going to precede it by its sentiment string,
then we're going to show the screen_name
of the user followed by a colon,
and we're going to show their actual cleaned up tweet_text
as the text that gets dumped out
at the command line in this example.
We will also add one to our tweet_count,
because eventually we're going to want to reach
that TWEET_LIMIT and we want to make sure that
the stream terminates properly in our example.
In the main method we're going to set up our authentication,
so this is the same stuff that
we've done a couple of times now.
First we must create the OAuthHandler object,
which requires both the consumer API key
and the consumer API secret key.
We then have to tell the OAuth object
to set its access token,
which is the additional two pieces of information
that we have stored in our keys.py file.
Once we've configured that OAuth object
we can get our tweepy API object,
and again, that requires the OAuthHandler as an argument,
and once again, we are going to specify that
if we reach those rate limits that Twitter imposes
we want the Twitter API object to automatically issue
a wait so that we don't go past the rate limits.
Next up we're going to get our command line arguments.
So, the search_key is going to be
the argument at index number one.
Remember that when you're working
with command line arguments
the sys module provides this argv list.
Element number zero is the name of your script.
Element number one is the first argument after the script.
Element number two is the second argument after the script.
By the way, you can use a multiword search_key
if when you type in your command line arguments
you enclose the search_key in double quote characters.
So, whatever the search_key is
will be stored here as a string,
and the last command line argument,
the one in index number two,
is going to represent the total number of tweets to process,
not including all of the ones that we ignore along the way.
We only add one to the counter for tweets
if we get what we are considering to be a valid tweet
for the purpose of this example.
Now, we are assuming that you properly
supply the two command line arguments,
so if you do not, this program will fail
with an exception and terminate, just so you know.
Of course, we could check to make sure
that there really are two command line arguments first,
and then only process them if in fact
there are the appropriate arguments.
Next up we set up the sentiment dictionary,
and as you can see, we've preinitialized it with three keys,
positive with the value zero, neutral with the value zero,
and negative with the value zero,
and this is the dictionary that the SentimentListener object
is going to continuously update as each tweet arrives.
Next up we create our SentimentListener,
and we're giving it the four arguments
that we talked about up above in the init method.
The first is the tweepy API object
that will be used to interact with Twitter.
The second is this sentiment dictionary that we just created
that the SentimentListener will store and repeatedly update.
The third is the topic that we're actually searching for,
the search_key in the main function,
and finally, limit, which we just created back here
at line 73, will represent the number of tweets to process.
Next up we're going to set up our tweepy.Stream,
and again, we have to tell it the OAuthHandler object
that it's going to use to authenticate with Twitter,
and then we also need to specify the listener object
that will be notified as each tweet arrives,
and that's the listener that we just created.
And finally, to get the stream going
we are going to invoke the filter method
once again on the stream object.
So, as you can see here, we are tracking
a list of terms that contains one term,
the search_key that we specified as a command line argument,
and again, this can be a comma-separated list
of a whole bunch of terms,
so for instance if I was a political researcher
and I was tracking two presidential candidates
I might want to have both of their names in the list here
so that I get all the tweets containing
each name along the way, or not all of them.
Remember, it's only one percent of the livestream,
so a randomly selected one percent of the tweets
that have those names is what I should've said.
Now, in this case we happen to be looking
only for tweets in the English language,
but we did demonstrate to you
in the previous streaming example
that we can receive tweets in multiple languages,
and that we can use things like TextBlob
to automatically translate those tweets as well.
So, the languages argument, again, is going to be a list,
so you can actually track tweets
in several different languages
by putting a comma-separated list of values here.
And finally, in this case we set is_async to False
because we only want to display the final results
after all the tweets that we specified have been read in,
so in this example we did 10 tweets
and I only want to show you the contents
of the sentiment dictionary after we have all 10 of them.
Once we, this call returns, which could take some,
a short amount of time or a long amount of time,
or somewhere in between depending on what you are tracking,
at that point we will display our final summary
of the results where we show you the tweet sentiment
for whatever the search_key was,
and for the positive we'll access
the positive key in the dictionary,
and similarly the neutral and negative keys
in the dictionary to display the final results.