- [Narrator] In this intro to Data Science video
we're going to continue talking about
some of the basic descriptive statistics
that can help you get to know the data
that you're working with.
Now, previously we looked at measures of central tendency
specifically the mean, median, and mode
and those helped us categorize typical values in a group.
So for example, if you're trying to figure out
the average height of a bunch of classmates
that would be the mean height of those classmates.
And similarly, if you're determining what car brand
is most frequently purchased in a given country,
we would have to find out all the different car brands
that are sold in that country,
we would have to calculate the totals for each one,
and the most frequently purchased one
would be the mode for that country.
Now, the group of data items that we're working with
is known as a population, and if you work with
a subset of that population, that is known as a sample.
So for the purpose of our example,
we'll be using the term population
for a very small population of data.
But as you get into big data
and data analytics applications,
your population of data may be billions
or trillions of elements and you may
randomly select a subset of those
and process a sample for analytics purposes.
Now, in this video, we're going to
focus on measures of dispersion,
which are also called measures of variability,
and basically they try to help you
understand how spread out your values are.
Now, you may recall, in an earlier lesson,
we talked about the range of values.
And of course, there's a problem with the range
which is you don't get a sense of how distributed
the values are throughout that range.
And you could have a narrow range of values,
where all the values are at one end of the spectrum,
or you could have a wide range of values,
where all the values are at one end of the spectrum
and just a couple are at the other.
They may not be evenly distributed throughout the range.
So, measures of dispersion give us a better reflection
of how the data is spread out.
Now, we're going to be talking about two measures
of dispersion: the variance and the standard deviation.
And we're going to talk about variance first,
because the standard deviation is simply
the square root of the variance.
So, once we have the variance,
we can calculate standard deviation very quickly.
For the purpose of this discussion,
we're going to use these 10 six-sided die rolls.
And let's talk about how you calculate the variance.
So, to determine the variance,
you start out by summing up these values
and calculating their mean, their average.
And if you add up these values and divide by 10,
you'll see that the average of those values is in fact 3.5.
The next step in determining the variance
is to subtract that average from each individual value,
which is also going to give you 10 values,
some of which will be negative
and some of which will be positive.
So, for example, one minus 3.5 gives you -2.5.
Three minus 3.5 gives you -0.5.
Four minus 3.5 gives you +0.5, etc.
Now, once you have those differences,
you then square those differences.
And squaring the differences has the effect
of emphasizing outliers in your data sets.
We'll talk about that again in a second here.
So, we square all the values,
which gives us positive results,
and then we sum all those values
and calculate the average, the mean,
to get what's known as the population variance.
So, as a result of the original set of values
that we have here, the variance in those values is 2.25.
Now, when you square those differences,
again, that emphasizes the outliers,
those are the values that are the furthest from
the original average of all of the data in the population.
So, when you start getting into data analytics applications,
sometimes you want to focus on outliers.
So, for example, if you are a company
that's trying to watch for credit card fraud,
outliers would be strange transactions,
and you may want to focus on those in a scenario like that.
In other data analytics applications,
an outlier, you may simply want to get rid of
because it's just not a case that is going to occur
and it really has no overall effect on the analytical data
that you want to obtain from your study.
So, it varies depending on
what you want to do with the data,
but it does, in terms of calculating the variance,
it emphasizes those outliers so that you can see
how far they are from the mean in your data set.
So, let's take a moment to switch over
to an iPython terminal where I've already executed some code
that calculates the population variance
for the specific set of values we were just discussing.
So, as you can see here, I imported the statistics module.
It has a population variance function called pvariance
in the module, so statistics.pvariance.
It takes as it's argument a sequence of numeric values,
in our case, it happens to be a sequence of integers,
and you can see that indeed, we got the same result
that we discussed in the slide that we just came from.
Now, like I mentioned, the standard deviation
is very straightforward.
It is simply the square root of the variance,
and taking the square root of the variance actually
tones down the effect of the outliers in your data set.
So, I will talk about why standard deviation
might be a better measurement in just a moment.
So, in terms of calculating the standard deviation,
we can do that a couple of different ways.
We can use the statistics modules function
for population standard deviation, which is pstdev.
P for population, std for standard-
or st for standard, and dev for deviation.
And if I execute that, you can see that the result is 1.5,
and to confirm that, indeed,
this is the square root of the value up above,
we can actually use the math module to help us with that.
So, let's import the math module.
Of course, you probably can tell that 1.5
is the square root of 2.25,
or that 2.25 is the square of 1.5.
But remember the math module has a square root method.
So, let's use the math square root method
and let's pass it the results of this code and
snip it to here, which calculates the population variance.
So, if this gives us the same result,
that is the population standard deviation.
So, some basic descriptive statistics to help you
understand how spread out your values are around the mean.
So, in terms of the standard deviation for these values,
the average was 3.5, and the spread around 3.5 was 1.5
for the standard deviation, and 2.25 for the variance,
which emphasizes the values that
are further away from the mean.
Now, going back to the slides for a second here,
the smaller those values are
for the variance and the standard deviation,
the closer your overall data set is
to the mean value in that data set.
So, smaller values mean closer to the mean.
Now, in terms of the advantage of
standard deviation versus variance,
when you are working with the variance
you wind up with units that are
different from your original data.
Standard deviation has the same units
as your original measurements.
So for example, let's just suppose you're using
temperatures in the month of March, Fahrenheit temperatures.
So you wind up, well around here anyway
where it gets cold in the month of March,
we might have 31 days worth of average temperatures
and they may be numbers like these.
Maybe a little higher, maybe a little lower.
This particular March we had some
pretty cold days around my neighborhood.
But in any case, the unit of measurement
for these temperatures is degrees.
And when you're working with the variance,
the unit of measurement becomes degrees squared
because you're actually calculating squares
of the differences between the temperatures
and the average temperature.
So, you wind up with, even though
those initial measurements are in degrees,
when you square the value, you now have degrees squared.
So, that's not the same unit as your original measurement.
By taking the standard deviation,
which is the square root of the variance,
you actually go back to the original measurement.
So, if we find out that the standard deviation
for our temperatures in March is two,
that would mean a two degree difference from the mean
in a given year of average temperature values.