- Welcome.
In this demonstration,
we're gonna take a look
at Kinesis Data Firehose Delivery Streams,
and we are going to create one
using the console with a destination of S3.
There are a couple of different ways
to reach the Firehose dashboard.
So we could search here for Kinesis,
and it will take us to the Kinesis console,
or if we type Firehose instead,
it'll take us to this specific dashboard
within the Kinesis service.
But let's go to Kinesis first,
and we'll see that we have bright orange button
with a couple of different options:
Data Streams or Firehose are two of those.
We can also select on the left-hand side,
and we could pick delivery streams,
and notice that it doesn't say Firehose.
It just says delivery streams
because those are the specific resources
that are created within that Firehose offering.
And so if we go over here,
click on Data Firehose and create,
it's gonna take us into a little wizard
that we have to fill out.
It gives us an explanation of how Firehose works.
We'll make that smaller so we can fill out our form.
We have a source.
We can allow for direct put,
that way we don't have to create a data stream as well.
And our destination is going to be S3,
but you can see that there are other possibilities,
third party offerings, other AWS services and so on,
but we're gonna pick S3.
Now that we've done this,
we can scroll down a little bit further
as this expanded the form for us.
It's gonna give us a random delivery stream name.
We can make that a little bit different.
We could call this test-stream if we want.
We have an option of transforming or converting records.
And so if we enable data transformation,
then it's going to ask us for a Lambda function
that's going to perform that work.
So we'll leave that disabled.
If we want to do a format conversion,
we have a couple of options here,
both Apache Parquet or ORC files.
We'll leave that disabled as well.
Although we can pick here schema,
we're gonna talk about AWS glue
in another lesson in this course.
So if we just wanna pass the data through
in essentially the raw format,
now all we have to do is figure out the destination
as an S3 bucket.
And if we didn't have one,
we can go ahead and create one here.
So we can click on the create button.
And it's gonna take us to the S3 service
where we will then be allowed to create the bucket.
So we can call this,
let's call it livelessons-firehosetest-01.
We'll leave it in us-east-1
because we're gonna match the region
of the delivery stream itself.
That's one way that helps to optimize for cost,
and we really don't have to change anything else, all right?
The only thing that I would add here
because it's not enabled by default
is object encryption.
Other than that,
we just click on create bucket.
And now we go back to our Firehose dashboard,
and we can then browse our S3 buckets here
and it will pull up a list.
And so we can scroll down,
keep going here.
Let's see, there we go.
Let's refresh livelessons-firehosetest-01,
choose that.
And if we wanted to pick a prefix,
we can do that here.
We can enable data partitioning,
which we're not going to do.
We don't need any error output prefixes.
And then we can start
to configure the actual delivery stream itself
in terms of the buffer size for buffer and batch,
as well as the interval.
And so regardless of how much data has been collected,
whether we meet the minimum size or not,
every 300 seconds it will automatically perform a delivery
to the back end.
We can pick different types of encryption
for the data records themselves,
and then we can come down here to advanced settings
where there are a few more options,
like error logging.
We can enable server side encryption
on the delivery stream itself,
and it will ask us about permissions.
And this is an important thing to note.
When we are going through and doing this,
using the AWS console,
there are a number of tasks
that are being accomplished for us behind the scenes
that would normally have to be achieved
as an individual piece of work.
Instead, the dashboard is combining all of this
into one form,
and the creation of permissions is one of them.
For a delivery stream to deliver to an S3 bucket,
there has to be an IAM role in place
that allows that delivery stream permissions
to place objects in that S3 bucket.
And that's what this is doing for us.
So we can add a tag,
we'll give it a cost center tag,
and click on create delivery stream.
This process usually takes a handful of minutes to complete.
And so if you are planning on automating this type
of workflow to create delivery streams,
you just need to be aware that there's going to be a delay
after the provisioning
while it creates that delivery stream for you.
And that completes this demonstration.