- So at this point we're now ready to take the files
representing our two scripts,
and the text file representing Romeo and Juliet,
and load those into our cluster to get ready
to run the Hadoop MapReduce task.
Now you'll notice that I've switched back over
to my web browser window and you can see here
on the portal.azure.com site,
in my notifications tab that it now says that
the deployment has succeeded.
If this has not happen for you yet you will have to
wait to continue on with this part of the presentation.
So I have a couple of options here;
I can either go to the resource, which I'll do in a moment,
or I can pin it to the dashboard,
which just makes it conveniently available
in the Dashboard option,
over here on the left side navigation
on the portal.azure.com site.
So, let me go ahead and click go to resource.
Once I do, let me expand this as well,
once I do I get the management page for my new cluster
that I configured, and for the purpose of this demonstration
the only thing that we really need to be aware of
is this SSH + Cluster login selection.
And the reason that it's important is
when you go to the Hostname and select the new cluster,
it gives you the command that you'll use with the SSH tool
to log into the cluster remotely
using a command line on your system.
Now, if by any chance you're a Windows user,
you may not actually have SSH on your system initially.
So, let me just pull over another browser tab here,
so this is actually a blog post from the Windows 10 team,
they have created an OpenSSH tool that you can install
into your Windows system and if you google this blog post,
or if you go to the website that you see here,
and as you can see the URL is kind of long,
so let me just also scroll over
so you can see the end of it as well.
Let's try that one more time, there we go,
now you can see the end of the URL.
But in any case if you read through this post you'll find
the URL where you'll need to go to install the
OpenSSH tool, and then you'll be able to use
the commands that I'm about to show you in this video.
Okay, so what I'm gonna do here is, for the moment
I'm going to copy part of what I see here,
specifically the SSH user @ and my cluster name,
so let's copy that.
And of course you're going to have to replace
"deitelpython" with whatever name you chose
for your cluster.
Now in order to copy the files over to the server,
I'm going to go out to my terminal window
here on my Mac computer, if you're on another platform
you'll have either a terminal window, a shell window,
or on Microsoft windows you'd use the Anaconda prompt
possibly, or a Command prompt for this particular task.
You'll notice that I have switched into
the ch16 examples folder
and specifically the HadoopMapReduce sub folder,
which has several files in it.
The first three of which are the files that I want to copy
over to the server and I'll use a command called scp,
secure copy, for that purpose.
The first arguments to this command are the files
you wanna move over to the server,
so I'm going to do a length_mapper.py,
length_reducer.py, and the RomeoandJuliet.txt file as well.
And then the next piece of the command is
the SSH user login ID,
so sshuser @ my host name - ssh.azurehdinsight.net.
And I'm going to add a colon onto the end of that,
which indicates that I want it to prompt me for my password
once I connect to the server.
So when I press enter the very first time you do this
you'll get this authenticity warning,
indicating that it can't authenticate the server,
it's unaware of our deitelpython cluster that we created.
So it's just a warning for you to be aware
that you may be connecting to a server that is dangerous,
in this case we set up the server so we're going to go ahead
and say yes, and hit enter.
So at this point that warning won't come up in the future,
if we connect back to this server,
but at the end of this example I'm actually going to
delete the cluster all together,
in which case the next time I recreate it
I will have a similar message once again.
So let me go ahead and enter my password,
and at this point you can see that it has now copied
each of the three files on to the server.
Now the next thing we want to do is to log into that server
using the ssh command, and again I'm going to use the
same ssh user that I used up above with the scp command.
And when I press enter it will prompt me for my password
once again.
Hopefully I typed that correctly.
Let's do it again there we go.
Okay, now I'm into the server and if I do an ls
we can see the files that we loaded up to the server
a few moments ago.
Now, in order for us to allow Hadoop to feed the contents
of RomeoAndJuliet.txt into our Hadoop MapReduce task,
we do have to move that file in to the Hadoop file system.
Now for that purpose we're going to use
the Hadoop fs command,
which is a Hadoop file system command,
and we're going to be copying from the local directory
that we're in right now, the file RomeoAndJuliet.txt,
and we're going to place that into a folder
that already exists in this cluster called /example/data,
and we'll still name the file there RomeoAndJuliet.txt
as well.
Now the reason that we're using that folder is
it already exists, it's one of the folders
that Microsoft provides for doing their own
Hadoop tutorial as well.
You could of course use other Hadoop file system commands
to create your own folders if you needed to.
So I'm gonna go ahead and execute that,
and it'll now move that file into the file system,
and at this point we're ready to execute our Hadoop task.