- [Instructor] One of our biggest goals in this lesson
is to give you experience with a variety of cloud
and desktop-based big-data software.
Now, the cloud vendors are going to be focused on
providing you with what they call
service-oriented architecture.
So, as a service, they provide you with these
complex capabilities that enable you to write Hadoop apps
and Spark apps and pub/sub based IoT applications
by registering for, potentially paying for,
usually paying for, and working with all of their services
in the cloud and as part of this lesson you're going to be
doing that a number of times.
You'll be using a MongoDB Atlas cluster
to do a NoSQL database in the cloud.
We're going to do a Microsoft Azure HDInsight
cloud-based cluster for our Hadoop example.
We'll also do in that same service, a Spark cluster
and we'll do a Spark cluster running locally on the desktop
as well and we'll call that "a cluster," because
when it's running locally, it's one computer
but it simulates the concept of a cluster with threads
of execution.
We'll also be using other types of online services
like the Freeboard.io dashboard visualizations
I mentioned in a couple of the preceding videos,
the PubNub publish subscribe service,
and the Dweet.io publish subscribe service as well.
And there are many other options out there for you.
We chose the particular ones that we're showing you
for their ease of use and sometimes for their
free or free tier or
credits that they give you
so that you have a chance to experience
these different capabilities and work with them
but there are plenty of similar services from companies like
Amazon and Google and IBM Watson.
There's also, for the purpose of Hadoop and Spark,
free desktop versions
of the Hortonworks and Cloudera platforms
and they recently merged so those will eventually become
one platform most likely
and they too have paid cloud-based versions
often running on the clouds of other providers
like Amazon, Google, IBM, and Microsoft.
And then finally, you also have the ability
to work with a Spark cluster, a single-node Spark cluster
in the context of the free Databricks Community Edition.
The folks who founded Databricks are actually the people
who originally created Spark, so it's an interesting website
for a number of reasons, not the least of which
is that they created the software in the first place
and they also have a lot of great learning resources
for those of you who are interested in pursuing
the Spark platform in more detail.