- [Narrator] In this video,
I'm going to talk about a tool called Docker
and the Jupyter Docker Stacks.
Now, the software that we're dealing with in this chapter,
things like MongoDB,
clusters in the cloud.
Hadoop clusters in the cloud, Spark clusters in the cloud.
All of these require a ton of setup
and they're complex configurations that are really difficult
to get through,
especially if you're somebody who's just getting started
with these technologies.
And that's why it's super handy
to work with pre-configured environments such as,
being able to set up a cluster
conveniently in Microsoft's HD Insight
or in similar services from Amazon or Google
or IBM, et cetera.
Or you sometimes will take advantage of pre-configured,
what are known as containers that have all of the software
set up done for you already, and you can run them,
on your local computer, and that's what we're going
to start with, as we get into Spark here,
we're going to use a tool called Docker,
which basically enables you
to take a pre-configured container
and run it on your local computer.
And to do that, we're going to take advantage
of a Jupyter team Docker Stack.
The Jupyter team has configured a bunch of these for us
for different development scenarios,
one of which is already configured
with the Spark application framework
and the PySpark module that we need
to access Spark application.
The Spark application framework from Python.
Now, in order to take advantage of this,
you will have to install Docker on your local computer.
And in order to do that,
I've provided you with a number of links here.
I don't go through the setup
and installation of Docker itself.
Generally, it's pretty straightforward,
especially if you're on Windows 10 or macOS.
You'll simply go to this link,
download the appropriate installer, and run it.
For those of you who are on Windows 10 pro,
you must allow the installer to make changes to your system,
otherwise Docker will not run correctly.
So, do make sure that you follow the prompts carefully,
as you're doing your install.
If you're a Windows 10 Home user,
unfortunately you can't install the same version of Docker.
In your case, you will have to take advantage
of a tool called Virtual Box.
So, they tell you how to set that up at this link,
and if you're a Linux user,
you'll need to use the Docker community edition,
and you can learn about installing that at this link here.
And then everybody who's getting started with Docker
for the very first time should take some time
to read their getting started guide,
which will give you an overview of the fundamentals
of working with Docker.
And in our case, we're running on a computer that has
one CPU with four cores,
generally you're going to need at least two cores in order
to run something like Docker because it is a form
of virtualization software.
Now, once you've installed Docker,
then you will be able to proceed with what we're going
to show you next, which is how to get the Docker container
up and running.
Now, as I mentioned, the Jupyter Notebooks team
has already pre-configured a bunch
of different Jupyter Docker Stacks
for various common Python development scenarios.
One of them is called the jupyter/pyspark-notebook,
and it has the Spark application framework.
It has PySpark, it's got Jupyter lab, the interface in,
which we've been running Jupyter Notebooks a couple of times
in these Python fundamentals videos specifically,
we used them back in the deep learning lesson,
and you can see a full list
of their pre-configured Jupyter Docker Stacks at this URL,
and this run a command this Docker run command
that you see here is meant to be entered,
as one long command without pressing enter.
Until you get to the very end, and I do want to point out
that you don't want to type it exactly as is
because you want to replace full path to
with the exact location
of our ch 16 folder on your computer.
The full path to that location is going to be required
for this to work correctly.
And what's going to happen is if you type in
your full path to the ch 16 folder correctly,
you will be able in the work folder of the environment,
the Docker container, to see your own local files.
In that ch 16 folder, you'll be able to open files,
from there, you'll be able to create new files there.
And that's important because if you ever delete
the Docker container, you don't want to lose your work.
So, this is a critical aspect
of launching the Docker container.
Now, the let's just talk briefly about the different pieces
of this and what's going to happen the very first time
you do this.
So, the Docker run command is used to execute
a Docker container.
The a run command has a lot of different options
in this case, we have a couple of - p options,
which means that we're going to open a port
in the container.
So this first 8888:8888,
is going to enable you to go to your web browser
and go to localhost:8888,
and be able to, for example, view Jupyter lab.
The second one, a 4040, is where you can actually view
a Spark monitoring webpage,
where you can see what the Spark applications are doing,
in your Docker container and the next couple of options
are going to enable us to log in as the root user
in that container.
They're basically Linux containers, behind the scenes.
Next, the - v option is what's going to help us mount
our local ch16 folder on our system
into the work folder inside the Jupyter Docker Stack.
And this is the actual name of the Jupyter Docker Stack,
followed by a colon, and the specific version number,
that we would like to download to our computer,
on the Docker site, they have a repository
of containers the Jupyter Docker Stacks are in that
repository as well,
and the very first time you run this command,
it's going to go find this specific version
of the Jupyter PYSpark Notebook Stack
and download it onto your computer.
If I remember correctly,
it's about five or six gigabytes of information.
So, it does take up space number one,
and it does take a while to download,
especially if you have a slower network connection.
So, the very first time you run this, it is going to perform
that download to bring this onto your computer.
Then we're going to execute the start sh script,
and we're going to launch Jupyter lab,
as part of that Docker container.
So, let's switch over to a terminal Window here on my Mac
and just show you that I executed the Docker run command.
Notice that I did specify the full path to the ch 16 folder
on my machine.
If you're a Windows user,
this is probably going to be a path that starts with
C colon backslash and then has path, the path separators,
as backslashes leading up to the ch 16.
Now, when you run this command
the very first time, you're going to see some information
here about the Docker Stack downloading to your computer.
I already did this a while back, so it just went ahead
and launched the Docker Stack for me,
and it started to display some log information here,
and then this key piece,
which is where you're going to go in your web browser,
to see the Jupyter lab interface and be able to work
with this a Docker Stack.
Now you can copy this,
but you are going to need to modify this piece.
So, let me go ahead and copy this here for a moment
and I'm going to switch over to a web browser
and oops and paste that in.
And before I hit enter, I'm going to go ahead
and change this parenthesized piece to just say, localhost.
And then so, this token here is just a security token,
that you need only the very first time,
you launch a Jupyter lab in that environment.
So, you notice it shows you a work folder,
but if you navigate into that folder.
You're now looking at the contents of the ch 16 folder,
and we're going to be doing the Spark word count example,
So, I'll go ahead and navigate in there.
Your folder will only have one notebook in it.
I have two, the version that I'm going to be using
during the presentation has some additional text bullets
in it for discussion purposes,
yours just has the code in it.
So, I'm going to go ahead and open that up,
and while that's happening, I'm going to talk about a couple
of other key items with respect to the Docker container.
So first of all, once the Docker container is up
and running, we are going to need to install
some software into it.
And the reason we need to do that is we're going to take
advantage of some libraries that are not
by default, part of that Docker container.
So, I'm going to actually switch to another terminal Window,
where I already did this and I just wanna show you,
what I did to install software into my Docker container.
First of all, again, this is a separate terminal Window,
you would use a separate terminal shell or command prompt
on your own system,
and you'll notice I executed the Docker PS command,
which lists out a bunch of information
about the container that's running
or the containers that are running.
If you have multiple ones running on your machine,
the most important thing I need from this list is the name
of the container, which it randomly assigns.
It's a two word name, normally separated by an underscore,
and the reason I need that is for this command,
which is going to allow me to log into the container,
so that I can install software into it.
So, the Docker exec command with the -it
means interactive mode.
This is the name of the container I want to log into,
and I would like it to execute its shell,
so that I can then interact with that container.
And again, this is a Linux-based interaction that you are
going to be doing in this case.
So, this launches the shell for the container,
and then I was able to execute a conda install command
to install the text blob module and the tweepy module
and down below here, it went through the process
of figuring out all the different packages that I needed,
and if I keep scrolling down,
you'll see it tells me what's going to be installed,
what was going to be updated,
and it did have to actually downgrade a couple of modules
in this case as well.
I said yes, because I needed to install
those for our demo purposes.
It then installed all the software,
and now I'm at the command prompt for my Docker container.
Now, with that said you don't want to have to go
through this process every time you launched the container.
So, it turns out that Docker has this cool little tool,
which you can access through your Docker menu
called Kitematic, which I'm going to go ahead and launch,
it's not installed by default.
I'll talk about the interface here in a second,
let me go back to my browser for a moment.
You can download this from kitematic.com,
it's the Docker toolbox,
as they call it, and what's nice about it is it keeps track
of every container you've ever executed on your machine.
And you can see this green circle to the left of the one
that we were just talking about,
which indicates that it's currently running.
These are some other containers that I had
executed in the past that are configured
for various things, separate from the demonstration
that I'm going to do for you here
in the next several videos.
Once you select a particular container, you can stop it,
and you can restart it.
This is important because if you simply stop and restart
your container, rather than going back out
to the command line to do that Docker run command,
you will not have to go through the process of reinstalling
all the software again.
So, when you next come back into your system
and want to rerun the PYSpark Notebook
with that software installed,
you would go to your Docker menu, open up Kitematic,
select the particular container that you want to execute.
And in your case, it would instead of saying stop
and restart, you would have a run option
because the container presumably is not running
at that time.
And you would then be able to launch the container,
and separately you also have the ability to click
this button to open up a command Window on your system
that's already logged into that container,
and you have the ability to click this icon
over here to open up the main webpage for your container,
as well, so you can basically access your container
and manipulate it right here
in this nice little kitematic tool.
So, at this point we've got our Docker container
up and running, and we are actually ready to start jumping
into our Spark example,
which I'll do starting with the next video.