- Now let's talk about AWS Data Pipeline.
AWS Data Pipeline is another very useful service
that helps us move data from one service to another.
Perhaps we have data in S3 that we wanna get
into Redshift, or we have data in DynamoDB
that we want to copy into RDS,
but they're different data types.
They're stored in different ways,
so we need to transform that data
from one data type to another data type,
and Data Pipeline can do that as well.
So again it helps us move data between various services,
including those outside of Amazon web services
such as on-premises.
Maybe we have data stored in some type of a local database.
Perhaps it's Oracle or an on-premises sequel server,
and we want to run queries on that data,
and then store that data somewhere else within AWS.
So we can use Data Pipeline to execute sequel queries
and then take the result of that and store it somewhere.
We can use Data Pipeline to execute custom applications.
So perhaps we have custom applications running on EC2
that pull data from somewhere
or generate some kind of data from a external API.
Does whatever processing that it does,
and then Data Pipeline can facilitate
the moving of that data from our application
to, perhaps, Redshift or back to DynamoDB.
We also get the ability to schedule these operations.
So perhaps we want to, every day, move data
from a DynamoDB daily table to Redshift,
and then perform daily aggregates and the like.
It does have built-in air handling,
and then of course the execution logs
of whatever happened will be stored in S3
so that we can go back and analyze,
perhaps something went wrong.
We can go back and look
and do our troubleshooting from there,
or we can just look at those logs
and see yes, indeed, things, our data was pulled
from DynamoDB.
Yes, it was written into Redshift.
So you can see that AWS Data Pipeline
is yet another very powerful, very flexible tool
that can help us move data from one place to another.
It can help us transform that data
from one data type to another,
and so it's a very useful service,
and that is AWS Data Pipeline.