Hi, in this video, I'm going to show you
how to creates a plot of daily average
ratings. So our graph will show the
average rating for each day. To do that,
I am not going to work on the previous
Jupyter file, because I like to keep things separate.
So for the visualization part, I'm going to create
a new notebook, I would suggest you do the same.
[No audio]
So first of all, we need to load the DataFrame. So I'm
just going to copy that cell from the previous notebook.
[No audio]
And copy there,
[No audio]
and just print quickly the head
of the data just to double check everything is
alright. Yes, so these are the data, the row data,
then press Escape and press B on your
keyboard to create a new cell, press
Enter to enter the new cell and let
create that graph. But before we create
the graph, we need to do some sort of
data aggregation, right. So we are
turning data into information, but our
data are quite a row, you see that. For
one day, we have multiple ratings left
by different students, and for example,
on the 2nd of April, so 2nd of
April, we have this rating, we have
that, and that, and that, and so on. We
want to aggregate those numbers into one
average rating. How do we do that? We do
that, we can do that by using the pandas
groupby method, which produces a new
DataFrame. So it's going to be a new
DataFrame. But with aggregated data, so some new
data which are averages of the row DataFrame.
Let me show you how groupby works. So I
said that groupby produces a new DataFrame, therefore I'm
going to create a new variable where the new DataFrame is
going to be stored. So day_average is my
new variable and that would be equal to
data, so that DataFrame, groupby.
[No audio]
I really like this method, its very intuitive.
So you're grouping these data by, what do we
want to groupby? Well, by Timestamp.
[Author typing]
Let's see what that gives us.
So I'm going to print out day_average in
here, a head of day_average, execute.
And let's see what we got.
[No audio]
We got basically the same DataFrame.
That is because groupby was not able to group
the data by timestamp, because the way
groupby works is that groupby tries to
find identical values in that given
column. In this case, timestamp doesn't
have any identical values, because each
value is unique, you can see it's the
same day but this review was left this
time, the other review was left this
time, and so on, so each value is
different. Therefore, before we apply
groupby, we need to do some processing
here. We need to add a new column in the
data DataFrame. I'm going to name this
column Day with capital D, I'm just
trying to be consistent with the names
of columns. Since these are with capital
letters, they start with a capital letter, I'm going
to create this new column with a capital letter.
[No audio]
So data Day equal to data Timestamp
[No audio]
.dt, dt is a property, which gives us access to a
number of data time attributes such as date,
sorry, date, you can do month,
[No audio]
and so on. For now we need the date. Let me
comment this out. So I'm going to select
them and press Ctrl / or Command /
to comment them out. And I'm going to show you
the new DataFrame. So that is the data DataFrame.
[No audio]
So what I just did is that, I extracted from
this timestamp, I extracted the date only.
Therefore what I got is that for each
timestamp, I got the date. So for this
timestamp is 2nd of April, 2nd of
April, and so on. So this way we got
some identical data. You know, if you
want it month, you'd get the number of
the month. So 4, 4, 4, and so on, but we
need date. So I'm going to keep date there.
Now we can, uncomment these, and let's
delete the data.head because we don't need it
anymore. And now, we can try this groupby
method again. But be careful, this
time we need Day here, as we said, we want to
groupby Day. Let's execute and see what we get.
[No audio]
Maybe it's not the result we were waiting for.
So you see what, day is not yet aggregated. Because
we need to give another command here, we
need to tell Pandas, the method of
aggregation. So do you want to aggregate
based on the mean or the count, in this
case, is the mean. So we'd say .mean the method.
Execute, and to this time, this is what we got.
[No audio]
So that's just the heads. But if you print out
the entire DataFrame, you see that it
goes like that upto this date. So for
each row, we have one day 1st of
January, 2nd of January, 3rd of January,
and so on. So that is the average
rating of all the courses for that day.
[No audio]
Now, you need to understand this
product. This is, as I told you, it's a
DataFrame type, so Pandas DataFrame, but
[No audio]
this has one column. So that is not a column,
Day is not a column, Day is actually the index,
you see that if you say day_average.columns,
[No audio]
Rating is the only column. And if you
want to access rating, you do like that.
Right, and you get this series. Now,
what is this? This is the index.
Therefore, if you want to access the
Day column, you don't do it like that,
because that is the syntax to access
columns. When you want to access that special column,
so that index column, you want to say .index.
[No audio]
And then we get this series which is
[No audio]
type index, but you can easily convert it into a list.
For example, if you like to plot them,
so just like columns, indexes, such as
this one are also lists like types. So
arrays of data. Let's now do the
plotting. To do the plotting, we are
going to need the Matplotlib library, so
I'm going to import it in here.
So import matplotlib.pyplot, we
need that module of the library. And a
good practice is to use as plt. So you'll
see on the web that everyone uses plt.
So if you want to be consistent with
other programmers, you want to import
that as plt. That also makes your job
easier because you don't have to type
that down, but you can just say plt as
we will do here. So plt.plot is the
method and this methods basically gets
two arguments, the x and the y. So, we
are building a graph with an x and a y
axis. Along the x axis we are going to
have the days that means we want
day_average.index. So that array there.
[No audio]
And along y we want day_average
Rating column. Execute.
[No audio]
I got an error plt is not defined because I forgot to
execute this cell. So that the input is valid.
Now I can execute this again. And this
is a product. So along the y axis we
have dates, which I know they are a bit
unvisible but we are going to fix that.
And along the y axis we have the Ratings
column. And so you'll see that it starts
from 3.8, somewhere here, up to 5.
Now Matplotlib picks that range
automatically by looking at the data. So
our Rating column, if you take a look
on it, you see Day_average Rating. If
you extract the max, the maximum value,
you'll see that, it is 5.0. And if you
see the minimum, you see that it's
around 3.8. So one day, students
left a 5.0 average rating, so all
of them, and another day they left 3.8.
So Matplotlib is putting those two
as the limits of y axis, instead of
starting the axis from 0 up to 5,
that would make the plot less readable.
So that's a good thing of
matplotlib. The bad thing is, as you can
see, these graph is not interactive. So
it just an image file, you can not have
popup capabilities. So you could see
some values if you hover your
mouse somewhere, so matplotlib cannot do
that. However, we can improve this a little bit
by declaring a figure object
and give it a figsize argument of,
let's say 25, 3 that is the width,
and that is the height over the plot. So
now you can see that we have a longer x
axis, and the shorter y axis. Now, if
you don't agree with this graph, if you
think that this is still not readable,
it doesn't tell you much about the
trend. So either rating has been
increasing with time or not. Then what
we could do is we could down sample the
data. So instead of extracting daily
averages, we could extract weekly
averages, and therefore we would have
less points along the x axis and the
smoother line. So in my opinion,
these are too much data, it's a
reasonable, it's not useful. Let's
down sample them with a better graph in the
next video. So this is what you learn
in this video, you'll learn how to group
data, and you'll learn how to plot them.
Now let me make a small revision of the
groupby methods, in case you are still
confused. So let me remove that in
another cell here. And let me print out
[No audio]
the head of the aggregated DataFrame. So
you see we have rating and we have Day
index here. What happens with Course Name
with this column, what's happened with
the Comments column, what's happened
with the Timestamp column? Well, they
disappeared because this mean method
only works with columns that have number
values, such as the rating, so it cannot
calculate an average on that column
Comments, or Course Name or Timestamp.
Therefore, columns such as Comments,
Timestamp, and Course Names will be
automatically dropped, deleted by this
method, and only those columns will be
kept. Similar, you could do a count
instead of that, in that case, you'd get
a different DataFrame. So you see that
we have account of Course Names, which means that
46 rows have been created for
that date. In other words, 46 reviews.
So we have 46 ratings, whatever you
like to call it. We have 7 here
because what pandas does is that it
doesn't take into account non values
such as this one or that one, and so on.
Therefore, we could have this count plot.
So we show how many ratings we had each day.
[No audio]
And that is what I wanted to teach you in this video.
Thanks a lot. I'll talk to you later.