- [Instructor] In this video, we're going
to talk about cleaning and pre-processing tweets
to get them ready for text-based
analysis using NLP techniques.
Now, as we were writing our book,
on which these videos were based,
one of the articles that we came across
pointed out that about 80 percent of the work
that data scientists do is on cleaning
and preparing data for analysis
in the first place.
So, as you know and as you've seen
in earlier videos in this lesson,
tweets come back with all sorts of,
I'll call it gunk, in them.
And, if you want to analyze the actual text
within the tweet, you may need to clean that up
and get the text ready for analysis.
So, some of the common natural language processing tasks
that you'll use to clean and normalize the text
within tweets for the purpose of analysis
is to do things like convert the text all to one case.
That way if you're, for example, parsing the words in tweets
and you want to keep track of how many occurrences
of a given word there are you can treat
both uppercase and lowercase as the same.
Removing things like the hash symbol from hashtags,
at mentions that mention other user accounts,
removing duplicate tweets because of the fact
that there's lots of retweets.
You might want to remove duplicates.
Removing the hashtags themselves from tweets as well.
Also, getting rid of excess white space and punctuation,
removing stop words like we demonstrated
with text blob in the previous lesson,
removing URLs, not that you would do this for everything,
it depends, of course, on what analysis you wish to perform.
You also often see them remove keywords in Twitter
such as RT for retweet and FAV for favorite
which is similar to a like on Facebook
and some other platforms.
You'll perform tasks like stemming and lemmatization
that we introduced in the NLP lesson
and also tokenizing the text inside of tweets
to get to the individual words
for analysis purposes as well.
Now, as you might expect, there are libraries
that can help you with tasks like this.
So, one such library is called tweet pre-processor.
This is the command you'll use to install it,
so you will want to take a moment to go ahead
and execute that command from your command line.
And, again, if you're a windows user,
I recommend that you run the anaconda prompt
as administrator for the purpose of installing libraries
to ensure that they get installed properly.
Now, the tweet pre-processor library has the ability
to automatically remove from a tweet any combination
of the items that you see here,
URLs, at mentions like at NASA which represents
the NASA twitter account, excuse me,
hashtags like pound Mars or hash Mars,
twitter reserve words like the RT and FAV
that I just mentioned a moment ago,
emoji characters, so you can tell it to,
actually, remove all emojis or just the smileys,
like smiley face, sad face, et cetera.
You also have the ability to remove numbers as well
and, in fact, there are a bunch of options that you can pass
when you're configuring a tweet pre-processor object
to specify which options you would like
to take advantage of, so I'll show you
that table in just a moment.
Now, it turns out that there's also
some handy utility functions in the text blob modules
sub-module called utils, so textblob.utils has a function,
for example, called strip_punc
which strips out punctuation either anywhere in a string,
or only at the end of a string.
So, depending on what the contents are of the text
that you're processing, you can decide where
to strip the punctuation from, and that could be important
for properly tokenizing text as well.
So, before we actually do a demo using tweet pre-processor,
let's just take a look at the option constants
that are available to you as part
of the tweet pre-processor's module.
So, those constants are listed down the right hand column,
here, and their corresponding meanings are shown to you
over in the left hand column.
So, for our little demonstration
that we're about to show you,
which we'll do in a new iPython session,
we're going to say that we want to remove URLs
and we also want to remove reserved words,
but again, any combination of these constants is possible.
So, with that said, I'm going to jump back out
to the iPython command prompt and, as you can see here,
I have started a new iPython session for this purpose.
And, as we go forward, we'll use new sessions
for several of the subsequent examples as well.
So, first thing we need to do is import
the tweet pre-processor's module
which happens to be called preprocessor.
And, their documentation recommends
that you import the module as the letter p,
so that's what I've done here.
Now, you can set the options
with the module's function named set options,
you simply provide the arguments comma delimited.
You do need to specify the module name
in order to access each constant.
So, p.OPT.URL gives me the URL option
and p.OPT.RESERVED gives me the reserved words option
for things like RT for retweet and FAV for a favorite.
So, we'll go ahead and configure that.
And, for demo purposes, rather than actually going
and grabbing a tweet here, we just created a string
called tweet text that has some sample text in it
including RT to represent a retweet and a URL as well.
So, let's go ahead and create that and cleaning a tweet,
once you have the tweet pre-processor set-up,
is as simple as calling the clean function
from that module and handing it the text
that you wish to clean.
What you will get back is a new string
with the options having been applied to it.
So, any of the text that needs
to be cleaned out, will be gone.
In our case, we're going to lose the RT for retweet
and we're going to lose the URL as well.
And, as you can see, it removed the extra space
before the letter a and it also removed the extra space
after the word URL as it gave us back that cleaned string.