- [Instructor] Over the next couple of videos,
I'm going to be demonstrating how to create a cloud-based,
multi-node cluster of computers
via Microsoft's Azure HDInsight service.
And one of its many capabilities is to provide you Hadoop
as a service running in the cloud.
And there are ways to run Hadoop locally as well.
For example, companies like Hortonworks and Cloudera,
which are merging, provide downloadable setups
that you can use, but they have massive system requirements.
So it's actually somewhat easier to play around
with this concept in the cloud if you can.
And for the purpose of this example,
we used the free credits that were provided by Microsoft
with a brand new account that we set up.
If you haven't set up such an account previously,
you could do that as well.
Otherwise, you would have to at least for the purpose
of running the example, pay for using those services.
But as you'll see,
we're going to configure a minimal cluster.
The application, itself, only takes a few seconds to run.
So as soon as you finish executing the application,
you can actually shut down your cluster,
delete all its resources, and potentially only be charged
a few cents if in fact you are not working
with the new account credit.
So once we set up the cluster, we're going to use
that cluster to demonstrate Hadoop's MapReduce capability.
And for our example, what we're going to do is parse all
of the words in "Romeo and Juliet,"
and we're going to be determining for each of those words
what the length of the word is.
Then our reduction step is going to summarize
how many words there are of each word length.
The kind of canonical example for getting started
with Hadoop is word frequency counting,
and we just wanted to do something a little bit different
since we've already done word frequency counting
in earlier examples.
Now once we have the code for our MapReduce task,
we're then going to use Yarn to submit that task
to the HDInsight cluster for execution.
Then from that point forward, Yarn and Hadoop are going
to decide how to use the cluster of computers we set up
to perform that task.
And at the end of that,
we'll take a look at the final results.