Hey there, I'm glad if you're watching this.
It's great that you have made it this far
in the course. And in this lecture, actually throughout
the few, the next few lectures, you'll learn how
to scrap data from this website.
So this is a real estate website and what it
does, it lists properties for sale or for rent.
So basically let's say Rock Springs.
There are actually a few Rock
Springs there, so let's say Wyoming,
Rock Springs in Wyoming.
So we are looking for some properties there
[No Audio]
in Wyoming, and it says it found 28 listings.
So it's quite a small city, small town.
So the idea is that you learn how to scrap data
of each of these properties, so which can be price,
you have the address there and the number of beds
that the property has and baths and so on.
And you also get the square feet
of the properties, if that is available.
Some properties don't have that, so we
have to count for that too.
And also, you will scrap data from multiple pages.
So we have ten properties here in this first
page and then in the next page we have
ten other, and then in the third, the last
page, we have the rest which should be eight.
So 28 in total.
Now normally, I assume that you know about
the requests and the BeautifulSoup libraries.
So you should have taken the previous lectures, where
we scrapped data from a simple web page.
So that was a trivial example and I believe
after that example, you end up with that, now what?
So for that reason, I want you to
learn how to scrap some real data.
So this is one of the real world
programs, that we are building in this course.
And in the script that we are about to write,
you face some real programming issues there, which is very
important to build up your skills, your Python skills.
Just one issue, before scrapping data from a
website, it's good to read the data policies
of that website, so they may have some
policies against using or getting their data.
I'm using this for educational purposes, so
I believe you'd be doing the same.
So that shouldn't be a problem.
So let's go ahead and write it program.
And I'll be using the Jupyter notebook.
I suggest you do the same.
[Author Typing]
So this will create jupyter notebook file.
So I'm using Python 3, call this century21.
[No Audio]
Great, so you now know that the very first thing
you want to do when writing a program is
maybe import, the libraries that you'll be using.
So you'll be using requests and BeautifulSoup, so from bs4 import,
[Author Typing]
great. Now go to the next line and
let's go back to the website.
So now the first thing you may want to think about is,
how do you load the source code of the web pages?
Now actually this is a bit complicated, not much
complicated, but it's different from the static web page
that we scrapped in the previous lectures.
So here we are going to be
scrapping three pages, as I said.
And the good thing is that, every one
of these pages has a unique URL.
So, when you are in the main page here,
you see the URL is just plain simple.
But then when you search place, so look at the URL
now, when I search, the URL will change.
So it went to real-estate/rock-springs and Wyoming.
So this is the string for the place
that you search, rock Springs and wy.
And you also got something else here
that you need to be aware of.
So the idea is that, now you get this URL.
So first we'll be scrapping the first page only.
And once we grab that, then we think about the
next pages, because the next pages are the same.
So the structure is the same, but we'll just be
writing a loop to iterate through the next pages.
So let's go ahead and load the first page.
Let's say requests.get, here is the URL
[No Audio]
and we want the content of this request object.
So r.content and let's print it out.
So a simple test there.
Hmm, Hmm.
[No Audio]
Alright, so my Internet connection is working.
That's all we know about this
code because we can't read it.
So what we want to do is, go to
the next cell and make that code more readable.
So we need to use a BeautifulSoup library here,
[Author Typing]
c and the parser, which is html. parser.
[No Audio]
And maybe print out soup.prettify.
Now let's see, so this is the page.
Sometimes you may get kicked out of the web page.
So you may want to make sure,
that the page has loaded correctly.
So maybe I could go here and search for something.
So Winchester, I'll search for Winchester here.
[No Audio]
Yeah, so the page seems to have loaded correctly
and we don't need the prettify there.
So let me clean the notebook there.
So we were able to load
the page correctly with requests.
And now what is next?
Well, next is we need to understand
the structure of the web page.
So we need to use the inspect tool in our browser.
And we'll do that in the next lecture, so see you.