So, as you see, it's quite a simple one.
And intentionally, I tried to find
a simple web page for you.
So here we go.
I didn't want to distract you with lots of content
for now, later you will be able to grab information
from a big website with lots of data.
So, for now let's try to grab,
let's say, we want to extract the
names of the cities from this page.
So if you want to follow me, please
type in this address on your address bar,
so with .html at the end.
And, so we've got only three cities here that we
will be extracting, but the code that we will
write, will work with any number of rows,
here I'll be using the iPython Notebook, or
the Jupyter Notebook as it is called now.
So it was renamed to Jupyter Notebook.
So right, Shift, right click and
open your command line, jupyter notebook.
[No Audio]
And I'll create a Python 3 notebook.
Great, so the first thing you want to do is
you want to load, this source code in Python.
And the way to do that, is by using the requests library.
So if you don't have that installed, you can
just go ahead and install it with, pip
install requests, just like that.
I have it already, so already satisfied,
but the process is very easy.
So you already know how to install packages with pip.
And you'll also need the BeautifulSoup library.
So to install that, you need to say pip install again.
And not BeautifulSoup, but bs4,
so which stands for BeautifulSoup 4.
So that's the latest version of BeautifulSoup.
And then, so you want to import requests, and,
so, the first thing you want to do is load the source code.
And then we start looking for html
tags and extracting elements from those tags.
But let me import BeautifulSoup as well.
So from bs4 import
[Author Typing]
BeautifulSoup.
So that's the syntax.
You're importing the BeautifulSoup class from bs4.
If you are on Python 2,
and this should be slightly different.
So you want to import BeautifulSoup directly,
directly like this.
[No Audio]
Okay, Alt+Enter and go to the next line.
So to load a web page, it's good
to create a variable, so you can load
the web page source code to this variable.
So r=requests.get.
So the get method.
So you point to the library, and then to the get
method, and all you need to pass here is the URL
of the webpage that you want to load.
So in this case, http://pythonhow.com/example.html.
So don't forget the html.
This is just a static web page,
so you should pass html there.
Now this should create,
[Author Typing]
a request object.
So we're still not there.
And what you want to do is grab
the content from this request data type, and
maybe store it in another variable.
So the content stored in a c variable
like that, and if you want to check
now, what this c, you'll see that this is a bytes
data type, and you can print it if you want.
[No Audio]
Even though this doesn't look very nice, this is
actually the source code, that you see in here.
So we have the head tags and
the html tags, and everything else there.
And now, here is where the
BeautifulSoup comes into play.
So all the request does is, it loads the
source code of the webpage, but in a
very scrambled form as you see here.
Now if you want to make this beautiful, and extract
the elements and the text and everything out of this
source code, you want to use BeautifulSoup.
So all BeautifulSoup does, is parsing this
source code, and giving you what you want.
So giving you the elements of
the html text, you're interested about.
So you have already loaded this content and
now what you want to do is maybe
create a variable and call it soup.
[Author Typing]
And that would be equal to a BeautifulSoup.
And guess what you want to pass here?
Well, that would be the content, and maybe another argument.
So you want to specify, the parser you want to use
for parsing this data. That is normally the html.parser.
So this is what you want to use.
Almost always, if you don't specify this, you'll
get a warning, but still things will work.
So I normally pass it there and once
you've done that, so execute that cell.
If you now print soup.prettify with empty
brackets there, you'll see the source code of
the webpage in an organized form.
So BeautifulSoup is trained to actually recognize
these tags, and then render them in a
visual way for the human eye.
However, this is just for demonstration.
Normally you'll not have to actually use the
prettify, method a lot, because a better method
to see this code, as I already mentioned
before, is to let me delete the cell.
We don't need that.
So a better way to see that source code is,
to go to your webpage and go to Inspect.
[No Audio]
And here you see a better syntax of the html code.
So here you'll see that, we have
three divisions here, with a cities class.
We have some more divisions here, but
this is what we're interested about.
[No Audio]
So, and the body is everything.
And if you expand one of these divisions, you'll
see that, we have an h2 tag.
So a heading tag, and also a paragraph tag.
So p tag and h2 tags.
And also the other division, which is this one here
has this h2 tag and the paragraph tag.
And Tokyo also has the same thing.
So our duty now is, to
extract the names of these elements.
So that should be, the h2, the text of the
h2 tags, inside the cities tags.
So naturally you start thinking about iterating, through
these boxes, which are actually divisions, so you
want to go through here, here and here
and extract what you want to extract.
So we go back to the code, and what you
want to do is perform a method called find_all.
And what you want to find is divs.
So, divs, but, there may be lots of divs in the webpage.
So, for instance, we have two more divs here.
And we don't want these to be found,
we only want these three.
So, but these three, as you see, they have a
common class attribute, which is equal to cities.
So we want to make use of that.
And we pass here a dictionary, which
should be class equals to cities.
Okay, and let me create a variable here,
and call it all and execute it.
Now, if you print all,
[No Audio]
you'll see that the divisions have been extracted, from the source code.
So from the soup, which was the entire source code.
And I'd like you to actually see closely here.
You can see that the first
division, is divided by comma here.
And then the second division starts up for Paris.
Paris is a second, and it ends here.
And then Tokyo starts here.
So we've got a list with
three elements, one for each division.
Now, if you want to find only the
first element, with this class attribute of cities,
you'd want to use the find methods, all.
So in this case, you don't get a list, but you get the,
code for the division, for the first division only,
which happens to be,
[No Audio]
a tag element of BeautifulSoup.
So it's not a plain string, but it's a
special, let's say a special BeautifulSoup string.
So that BeautifulSoup knows its structure, so it
knows what are elements, so where the tags are
and where the text is and so on.
So that BeautifulSoup is able to give you,
the information that you are looking for.
So, all again.
So you extract the first element.
Now, an alternative way to extract
the first element is logically.
So we have all elements here, is to use list indexing.
So this object that I just showed you,
the tag object of BeautifulSoup supports indexing.
So you execute that.
And in this case, as you see,
you extracted the first item of the tag object.
Or you could do it like this.
So you grab all of them.
So here you have all of them,
and zero is the first one.
You get the idea?
Okay, but what if you want only the
h2 tags, from this div class?
Well, in that case, what you'd want to do is refer to
the all object, and then apply the find_all method again.
And this time, you'd want to get the h2 element.
And in this case, you don't have a class attribute, so you'll
have to leave it like that, and you get an error.
Because, what I did here, is I didn't
point to this division, but I pointed to
actually the list, containing all these divisions.
So Python is trying to get the h2, but
this ResultSet method doesn't have this h2 element.
So what you want to do is, you want to
point to the first element, element of the list and
that gives you the h2 element with a tags
and text, which is like a list.
So you want to perform a zero indexing there.
And if you want London only, you
apply text, and you get London.
[No Audio]
So this is what we wanted,
right, to extract the cities.
So we extracted London. Now,
how about extracting Paris and Tokyo?
Well, as you might guess, we need to use a for loop.
But first, let me summarize what we did here.
So we loaded the content up
here, which is this one here.
And then we loaded this
content, in the BeautifulSoup method.
And BeautifulSoup makes this soup beautiful,
so that it recognizes these tags.
And so what we did then is we found, we
extracted from this content, we extracted all the division elements.
So together with the text, and the attributes and
the text inside them, so everything inside these divisions
with a class equals to cities, then we can
perform for each of these
[No Audio]
elements of this list.
We can perform again a find_all method, so
we can find subtags of these division tags.
And in this case, we found the h2 tags.
And then we grabbed the first item of the list, which in
this case, happened to be a list with only one item.
So each of these divisions have, one h2 tags.
Or alternatively, you could just use find
here and without using this indexing.
But this is a general method.
And then we apply the text attribute there.
So to extract the text out of this element.
So we got London.
Now we need to do the same,
but, in this case by iterating.
So for, let's say, item in all, you want to print out.
So item here is, this one here.
So this would be the first item.
So you want to print out, the item.find_all,
and you want to find the h2 tags from this first item, for example.
So h2 tags. And then you need to apply this zero indexing there
and you want to grab the text from this.
And that's it.
Here are the data.
Alternatively, you could just pass p
here, and you get the paragraphs.
So this one's here, the text.
So that's the idea of loading webpages
in Python, and parsing them with BeautifulSoup
and extracting text out of the webpage.
So, sorry if I was a bit repetitive
in explaining this stuff, but I really want
to make sure you understand the core concepts.
On the other hand, if you found this
very basic, I would say let's move on
to the next lectures, where we'll be extracting
some information from a more advanced website.
And we'll be extracting links and not only text.
So that's a real world program,
and a very interesting one.
So I'll talk to you later.