- [Instructor] So before we get into an actual example
of Hadoop let's talk a little bit about its background.
Now Google launched back in 1998
as a web search company.
And at that time there were about 2.4 million websites.
And already that was a tremendous amount of data online.
But just to give you a sense
of how much the web has exploded since then,
there are now about 2 billion websites,
which is approximately at 1000 fold
the increase over the number of websites
that there were when Google first started handling search.
So nowadays Google is actually handling
over 2 trillion searches per year.
So early on when Google was working on their search engine,
they knew that they needed the ability
to return results quickly
and they determined that the only practical way
for them to do that was literally
to store the entire internet
and index it for fast search.
So unfortunately the computers of that time,
were unable to store and analyze
that volume of data in an economical manner
and in a fast enough manner.
So what Google did is they developed
their own hardware and software for that purpose.
They developed a clustering system
with massive numbers of computers connected by the internet,
those nodes on the network
were where all the searching was actually performed.
And because there were so many different nodes
on the network there was a much higher chance
that those network elements
could fail the hardware around the world.
And so they built in a lot of redundancy as well.
So basically what they did is they took the internet
and distributed in chunks across all these nodes
on the network that they created.
And they implemented these with commodity computers
so if one failed,
they could just swap it out quickly with a new one.
Now when they finished implementing the system
and a search request would come in,
what would happen is they would put to work
all the computers in the cluster
to search their portion of the web in parallel,
then the results would be gathered
and returned to the user.
To implement that system,
Google created their own clustering hardware and software.
And they didn't make
that hardware and software available to everybody else
but they did publish their designs.
And one of the key publications
that they put out there was the Google file system paper
which you can find at this URL here.
And some programmers over at Yahoo took that
and wound up implementing their own system.
And they did open source their work
which the Apache organization picked up
and implemented as what is now called Hadoop.
And by the way that was named for the elephant
of a stuffed animal that belong to a child
of one of the Hadoop creators.
It was also the inspiration for our textbook cover
for the book intro to Python for computer science
and data science as well.
Now there were also a couple of additional Google papers
that contributed to Hadoop's evolution.
One of them was on MapReduce
which you can find at this URL here.
And that's the technique of programming
that we're going to be demonstrating
to you as part of our next example.
And then they also had the big table paper
which is the basis for another Apache project
called the HBase which is one of the NoSQL key value
and column-based databases
that we discussed a little bit earlier.
Now there are a couple of key components
to working with Hadoop.
One of them is the Hadoop file system,
Hadoop distributed file system to be more specific
or HDFS for short
and what that file system does
is it enables you to store
a massive amount of data across a cluster.
Then we had MapReduce
which is a system of programming for processing
the data across the cluster.
And basically what it does
is in a massively parallel manner,
it enables you to implement
functional style filter MapReduce operations.
It's called MapReduce
but the mapping step is really up to you
as to what you wanna do in it
and it can include filtering as well.
So the the mapping step
is where you're going to process the original data
and you process it into tuples of key value pairs.
And then the reduction step combines those tuples
to produce the final results
of whatever the MapReduce task is.
Now when you run your task,
Hadoop divides the data in two batches.
The data gets distributed
over whatever number of nodes there are in your cluster
which could be anywhere from what we'll do
which will be two nodes
up to thousands of nodes
in some of the biggest clusters in the world
and along with the data,
Hadoop also distributes the MapReduce tasks code
so that it can be executed in parallel
across all the nodes in the cluster
on the particular batches of data
that are stored on each node.
So on a given node,
it will execute a copy
of the MapReduce algorithm that you implement
on the piece of the data
that's stored within that node.
Then during the reduction step,
all of the results will be combined
into the final results of your application.
And all of that is coordinated by a tool called YARN
which stands for yet another resource negotiator.
It is responsible for managing
all of the resources in the cluster
in scheduling the tasks that will be performed.
Now although Hadoop began with HDFS and MapReduce
and was followed very closely by YARN,
it's actually a massive ecosystem
with many other components to it nowadays.
So here's a couple of links to articles
that talk about various ecosystem components.
And I've listed some of them for you here
and on the next two slides as well.
So these are all Apache projects
that you can take a look at.
And I've provided you URLs for each of them
if you want to get an overview of what they're about.
So Ambari and drill are two of them.
Flume, Hbase, Hive and Impala the next four
and then finally as well
Kafka, Pig, Scoop, storm and Zookeeper.
And this is just some of the Hadoop's ecosystem components
that are now out there.
And we'll focus in this lesson
on just the basic use of Hadoop
with MapReduce and YARN being used
to execute our application.