- So, next let's go ahead and take a look
at the Reducer script
and just like we did in the Mapper script,
we start out by telling it to use Python3
to execute this script.
Now, for this example, we again define a generator function
called Tokenize Input.
In this case, every line we receive is going to
represent one of those key value pairs
produced by the mapper, but we're going to be reading them
from the standard input stream
which the Hadoop system is going to
automatically redirect for us from our mapper script
so we don't have to worry about that.
Now, what we're going to do in this example
is strip off any white space
at the beginning or end of the line.
Remember, the print function which we used over here
in the mapper to print out each key value pair
inserts a new line character at the end
so we're basically stripping off that new line character.
Then, we're splitting the string at the tab character
which gives us back a list of the two strings
in that tuple of information,
the key and the corresponding value.
So, as you can see here, we have a for loop
in which we're using a built-in function
of the Itertools Module from the Python standard library.
And, it's going to group all of the keys together for us.
And, so what we're going to do here is feed that
the tokenize input call which is going to
grab all the key value pairs,
and we're going to
group all of those key value pairs by key.
And, of course, the keys are the word lengths
and the corresponding group will contain the key value pairs
that represent each word of a given length.
So, if the word length is 10, the group associated
with the word length 10, will have all the tuples that
have the key 10 and the value 1, and then the
code in the body, here, is going to be responsible
for summing up those values into a single key value pair.
So, you'll notice as the second argument of groupby,
we have to tell it for each of the lists
that we're getting back from Tokenize Input,
which item number within that list we would like to groupby
and item number 0 in the list is going to be the key
of the key value pair that was split up above.
Then, we're going to use the sum function
that's built into Python and we're going to
basically iterate through all the word-length count pairs
in the current group that we're processing.
And, we're going to sum up those values for the counts.
So, we're going to total up the counts, and then
the final resulting output of our reducer
for that given word length will a new tuple
of the word length a tab and the string representation
of that total number of words of that length.
And, so we'll do that for all the different groups
and as soon as the reducer completes its task,
then our Hadoop Map Reduce application
will be complete, and we'll
be able to look at the output.