Great, so let's go to our page.
[No Audio]
So we have searched for Rock Springs there.
And what we want to do now, is
to understand the structure of the web page.
So we use the Inspect tool there.
[No Audio]
And the logic is that you want to
iterate, through all of these boxes.
So to say. So we grab the HTML of these boxes and
then we go through inside of those HTMLs.
So we iterate through those HTMLs, and then we find
the tags, for the data that we want to get.
So that brings us to the point that we should
be looking for the elements that identify these boxes.
So if I go Inspect here again, I'll see that
this whole box has this div with this id.
And actually I want an upper level.
So I want the entire box there.
So if I go here now, this
looks like the entire box here.
So inside here should be the data.
This is the picture and this is a Price.
So, here is the text for the price.
And we have this propPrice, as a class for
the h4 tag, which creates this number here.
[No Audio]
So, this is the first div division.
Then we should have the next div somewhere there, here.
So propertyRow, propertyRow here,
propertyRow, propertyRow again.
So the class, propertyRow.
And again here.
[No Audio]
And let me put this down here.
So docking it at the bottom ,so that we
can see the entire box there,
[No Audio]
like this. So here we go.
So what I would like to scrap and save them in
a CSV file or Excel file with Pandas later, is I'm
going to get the Price, the Address, the number of Beds,
the number of Baths, the Area of the Property, and also
the Lot Size, if there is Lots.
So some properties don't have a Lot there.
So we have a trick there, and you're
going to learn how to crack that.
And so these are the data that
I'm going to grab from each property.
And let me put this up here again.
[No Audio]
And you can either click here and Inspect,
so it goes directly to the elements or
to the Price or go manually, which is
probably better because, it helps you understand better
the website, the structure of the web page.
So this is a Price.
And then down here, inside here should
be the address elements, primaryDetails.
These are the Beds.
So if we expand this, you'll see that
here is the text for the Address.
So it has a span, with this class name.
And this here is the name of the town, the
code for the state and the zip code as well, great.
But first of all, as I said, I need to
go through this propertyRow, class of the div.
So let's do that here.
And you know that you have a method called
find_all, which applies to the soup object.
So find_all, and this will generate a list with all the div
elements that have a class of, what was it, propertyRow?
Yeah, it's propertyRow.
propertyRow with capital R and that's it.
So Alt+Enter, execute that.
And so what can you do?
Well, all, print it out and maybe see what you get.
[No Audio]
So it starts at the very beginning,
of the very first propertyRow division.
So that would be the first price, which
was this one here, up here, here.
[No Audio]
Then there should be a command
here, after the first division ends.
[No Audio]
But anyway, if you don't want to
find that manually, you can do something.
You can find the length of this
all object, which is like a list.
It's not exactly a list.
Actually, it's a ResultSet element of bs4, of the BeautifulSoup library.
But it has a length function, just like List too.
So len(all) and you get 10, and
we have exactly ten results for each page.
So first page here, second page
has ten results and so on.
Now, this is like a list, so it doesn't
have a find_all method, but its elements.
So let's say the first element,
this element has a final method.
So just like you do with BeautifulSoup, so
you applied a method, a final method to BeautifulSoup
to find Tags elements, you can do the
same for the elements of the all List.
So to say, let's call the result, set a list.
And, so
And, that means you can apply a find_all method, to this source code.
So let's look for the Price.
Well, you can go to the Inspect or just
look through here, if that is yeah, this is
not much code, so I found the Price here.
So we have the h4 tags, h4, and
it also has a propertyPrice, a propPrice class.
So let's pass that.
You can choose not to pass that, but the problem
you may run into, if you don't pass the class
name, is that if you have other h4 tags
in the code, Python will extract them as well.
So you need to specify which h4 we want.
So propPrice there find, and here is a price.
So this is a Price, but with the tax as well.
And this is actually a list, as you see.
Now, because we have only one price for each property, in
this case, we are allowed to use the find method.
So that would give us not results
at least, but the actual tag element.
That means we can now apply a text object
here, and we get that funky string there.
So things are not that simple in real life, as you see.
But luckily, all this object is actually
[No Audio]
a string, so it's a plain Python string.
That means you can apply string methods to that object.
So let me Ctrl+Z to remove the type.
[No Audio]
So in this case, what we want to apply
is, we want to remove all these characters.
And a way to do that, that I think about is to replace.
So you want to replace the \n, with nothing.
So just pass an empty string
there, and see what you get.
Okay, these guys of this century21 website
have decided to make our life difficult.
But, how about applying?
So we've got some space there white space, as you see.
So we replace the white space with
nothing and we get the actual string, great.
So we sort of know that things
are working well, at this point.
And for now, I'm just printing out the results.
So, as I already mentioned, it's good to first
use print statements, when you're building your programs, and
then you replace those print statements with other functions
that you want to use for this data, you're
getting or other objects that you're working with.
So in our example here, later on we will
be adding some Pandas methods, to grab these values
and send them to a CSV file.
So that's one of the first thing, I would like to say.
The second thing is that, here now we
grab the value of the first element.
Now we start to think about being efficient.
So we need to instead of going and extracting
all the other elements like the property address and
property state and so on, maybe it's good to
actually start building our for loop.
So we now know that individual values are
being extracted correctly, but now we want to
make sure that a loop, that iterates through
all these properties is also working.
And let's go ahead and start writing
the loop in the other lecture.
But for now, let's actually organize this code.
So here is a trick you can use in Jupyter.
So I go to the first cell and I'm in command mode.
So you press Escape to go to the command mode
and Shift+J, you select the other cell.
J again, select the other cell.
Or you can go up with K, so J, J, J.
And what I want to do now, is I want
to merge all these cells in one single cell.
And to do that with Shift pressed, you press M.
So Shift+M and you get all
the cells, merged into one single cell.
So this is more an issue of preference, but
it's good to have a clean notebook there.
So this is what we did so far.
Let's go ahead and run the loop in other lecture, see you.