- Welcome to Python Fundamentals LiveLessons,
part four of five.
My is Paul Deitel,
and I'll be your instructor for these LiveLessons videos.
This part is based on chapters 12 through 14
of our college textbook,
"Intro to Python for Computer Science and Data Science:
"Learning to Program With AI, Big Data, and the Cloud,"
and chapters 11 through 13 of our professional book,
"Python For Programmers."
The prerequisites for this part are
Python Fundamentals LiveLessons parts one, two, and three,
or equivalent Python programming experience.
Many people find that as they work their way
through the videos, it's helpful to have
a copy of the book at hand.
However, this is not required.
There is additional information in the books
that I don't cover as part of these videos.
If you're interested in getting a copy of one of the books,
you can find them in various print and electronic formats
at the sites that are listed for you.
If you're a college instructor, you'll probably want to
take a look at "Intro to Python for Computer Science
"and Data Science," which is our
college textbook version of the book.
This one is in full color.
It's 880 pages, 240 pages more
than "Python For Programmers," and in particular,
it contains a ton of exercises.
557 self-check exercises, as well as 471 additional
end-of-chapter exercises and projects.
Now, many professionals find that they like to
work with the textbook version specifically
because of the exercises.
If you're interested in learning more about
"Intro Python for Computer Science and Data Science,"
please take a look at the links that I've provided below.
The first one is for an architectural diagram
in which we show you the four-part modular structure
of this book, and that really applies both
to the college textbook and our professional book,
"Python For Programmers," as well.
The "For Programmers" book has one fewer chapter,
but the overall architecture of the two books is the same,
and also in the "For Programmers" book
we have less of the lower-end pedagogical material
that's really geared to people
who are novices in programming.
The second link is for the full table of contents,
where you can see everything that we're going to cover
throughout the book, and the third one is
the book's preface, where you can learn more about
our approach to teaching Python, data science,
artificial intelligence, and various big data topics.
I'd also like to recommend you take a look at
these two additional links.
The first one is for the full book cover,
and in particular, if you look at the back cover copy,
you'll see a nice summary of everything
that we're going to do throughout these LiveLessons videos.
And I'd also recommend that you take a moment
to read through the technical and academic
reviewer testimonials to really get a sense
of all the things that they liked about
how we presented Python and the various data science,
AI, and big data topics.
In this part, we're going to be focusing on
lessons 11 through 13, on natural language processing,
data mining Twitter, and IBM Watson and cognitive computing.
In Lesson 11, we're going to take a look at
natural language processing, or NLP for short.
Now, every day, we have various
natural language communications with other people.
We send and receive text messages,
we send and receive emails,
we do Facebook posts and other people read them,
or we send tweets and other people read them,
or we read other people's tweets.
These are all forms of natural language communication,
and what natural language processing tries to do
is enable computers to understand that language.
And as you might expect in Python,
there are libraries that can do this for you.
So we're going to take advantage of
several such open source libraries.
We'll look at TextBlob, NLTK, Textatistic,
and spaCy throughout this lesson.
And we'll focus on English-language processing,
but spaCy in particular does support
several different spoken languages,
and there are other libraries out there
that you can take advantage of as well,
in fact, many other natural language processing libraries
are out there for other languages.
Some of the things that you're going to see how to do
in this lesson are basic tasks like tokenizing text
into sentences and individual words,
recognizing the parts of speech within text.
So in order for a computer to understand text,
it has to be able to recognize various aspects of the text
and then assemble those pieces together
into understanding of that text,
and parts of speech are a key aspect of that.
One of the most common things you'll do
with natural language processing is sentiment analysis,
where you look at text to figure out is it positive,
is it negative, is it neutral?
And this is commonly used, for example,
when people are analyzing tweets from Twitter,
they'll look at the text to see if it's
saying something positive or negative,
and in fact, that's one of the case studies
that we'll do in Lesson 13.
Here, we'll show you some basic sentiment analysis
by just providing a couple of sentences
to what's known as a TextBlob,
and then using its built-in sentiment analyzer
for that purpose.
We'll also talk about various language detection
and translation capabilities.
We'll do this via the TextBlob library,
which interacts with Google Translate for those purposes.
We'll show you some cool visualizations
in this lesson as well.
So one of the ways to analyze text
would be to do calculations like word frequencies,
so you can see what words come up most frequently
in a body of text or a corpus, as we refer to it.
But those are just numbers, and sometimes it's
much more interesting to show visualizations.
So we'll go ahead and calculate word frequencies
and we'll do a couple of different visualizations.
One will be a simple bar chart, which is okay,
and better than just numbers.
But we'll also show you something really cool,
which is the ability to create a word cloud
in which larger words represent words
that appear more frequently in a body of text.
And as you're about to see later on in this lesson,
you can do that with just a couple of lines of code.
In Lesson 12, we're going to take a look at
one of the most fun and interesting lessons,
data mining Twitter.
Twitter is a massive big data data source
that just about anybody can tap into,
and one of the most common things you'll do with it
is actually sentiment analysis from back in Lesson 12.
Now, as part of this lesson,
we're going to be working with a Python library
called Tweepy, and one of the things
that's great about Tweepy is that it makes it
super easy to connect to the Twitter web services
and authenticate with them.
You will need a Twitter developer account for that purpose.
And once you are authenticated,
it makes it easy to invoke the many different
Twitter web services that are available
and to ensure that you don't accidentally
violate Twitter's terms of service
with regard to the rate limits,
meaning how many times you're allowed to invoke
a particular web service over a 15-minute time interval.
So I'll discuss all of those issues throughout this lesson.
Now, once you're authenticated with Twitter
via the Tweepy library, you're going to be able
to take advantage of a Tweepy API object
to do things like search through past tweets
over the last seven days.
You can't go beyond the past seven days with the free APIs.
You're going to be able to tap into 1%
of the live tweets going on at any given time,
and if you think that's a small number,
there's something like 800 million tweets a day
at this point, so 1% still gives you access
to seven or eight million of those tweets on a daily basis.
So that's still big data, even using the free APIs.
We'll show you how to work with tweet objects
and their metadata.
So as you probably know, a tweet is typically
140 characters or less, and back in 2017,
they made it capable of being up to 280 characters.
But in addition to that, there's all sorts of metadata
that Twitter maintains on a tweet by tweet basis.
So we'll talk about some of that metadata,
show you how to use some of that metadata as well.
We'll do some natural language processing on tweets,
looking for sentiment analysis in particular,
whether a tweet is positive or negative.
We'll also show you how to work with the Twitter Trends API
to look at trending topics worldwide
and in specific locations around the world.
And we're going to do our first dynamic
interactive maps as well,
where we will take a bunch of tweets and actually
plot the locations that they came from
on an interactive map where you can
click on a map marker, see the tweet,
and the location on the map as well.
To finish off Part Four, we're going to take a look
at Lesson 13, IBM Watson and Cognitive Computing.
Now, chances are most of you have probably seen
an IBM Watson commercial on TV by now
showing off some of its capabilities.
What we're going to do in this lesson is
introduce you to Watson and show you
how to sign up for an account for working
with their free Lite tier services,
and one of the reasons we specifically
chose IBM Watson for use in our books and videos
versus some of the other cloud vendors like Amazon
and Google and Microsoft is that IBM Watson
does not require a credit card for you to sign up
and experiment with some of these free tier Lite services,
as they refer to them.
So we'll show you how to get set up.
We'll also go ahead and have you download and install
the Watson developer cloud Python SDK.
They actually provide SDKs for
many different programming languages.
And we'll then be able to use objects from that Python SDK
to interact with many different Watson web services,
and we'll take advantage in this lesson
of three of them in particular.
We'll use the text to speech service
to take text and turn it into audio.
We'll use the speech to text service to transcribe audio
back into text, and we'll also take advantage
of their language translation service as well,
and we're going to mash up all three of those services
into a little traveler's companion app
where somebody can speak a question in English.
We will then transcribe that into English text,
translate it into Spanish, and then speak it in Spanish,
and we'll do the reverse as well as a Spanish speaker
responds to the question in English.
So it's a really cool way of demonstrating
some of the power of these online cloud-based services
that are available to you from many cloud vendors.