- In this video and the next couple of videos,
We're going to continue our introduction to data science.
And also continue our introduction to Pandas
by presenting how you can use Pandas
with some Regular Expressions to do a little bit
of what is known as Data Munging or Data Wrangling.
It is said that for data scientists,
about 75-80% of their time
is actually spent preparing their data
for use in data science studies.
So, this concept of getting data ready
for a study is of great importance to anybody
who's going to become a data scientist.
Now, two of the common operations
that are performed as part of Data Munging or Data Wrangling
are cleaning data to get it ready
and also transforming data as well.
So, some operations that you might perform
for cleaning up data are things like deleting observations
that have missing values because you won't be able
to process them correctly.
Possibly substituting reasonable values
for missing values.
And by the way,
which items you choose to do
to clean your data are often going to depend
on the type of study and the type of data
that you're manipulating.
Deleting observations that have bad values.
Possibly substituting reasonable values for bad values.
Tossing outliers.
In some studies,
you may wanna get rid of values
that are far outside of reasonable ranges
and in other studies,
those might be useful.
So again, it does depend on the particular study.
You may or may not want to perform duplicate elimination,
and you may or may not want to deal
with inconsistent data as well.
Now, one example of cleaning would be
let's say we had
a patient's temperature readings in the hospital.
So, we might have something like this
where we have the name of the patient
and some temperature readings.
0.0 clearly is not a valid temperature reading
for a patient in a hospital.
Maybe the sensor got disconnected
or malfunctioned in some way
and therefore the reading did not come through properly.
So, if we were to average out
these first three temperatures,
we'd get 98.57 which is approximately correct
for a person's body temperature.
But if we average out all four of these,
we would get only 73.93 which clearly,
in Fahrenheit temperatures,
is not a valid body temperature
and would be of great concern potentially
to a doctor caring for that patient.
So, a couple of ways we might clean this data
would be to either delete the 0.0,
recognizing that it can't possibly be right
or possibly replacing it maybe with the average
of the other three temperatures in the list.
Now, another thing that we often do while munging,
is transforming our data.
So, we might do things like remove unnecessary data
and features of the data that we don't need
for a particular study.
Now when you get into big data,
you may be dealing with massive amounts of information.
So, removing the stuff you don't need,
can often save time and space.
You may want to combine related features.
So, let's say you had a first name
and a last name but you wanted to combine them into
a single string just as a very simple example.
You may want to take a random sample
of a massive data set to get a representative subset
of that data for more efficient processing purposes.
So for example, when people are doing polling
for presidential elections,
they don't go poll every single person in the United States,
they poll a random sample of people
that is often just a few thousand
and then they extrapolate from that information.
You might need to standardize some of your data formats
or you might need to group data differently as well.
So these are just a couple of the types
of things you might do while preparing your data
and in the next couple of videos,
we would like to take a look at some basic manipulations
of data in the context of Pandas using Regular Expressions.