So another important tool in the quiver of
tools that a data scientist has is Apache. Iceberg,
Sam redai is a senior software engineer on the
experimentation platform and Netflix Samuel responsible for telling me all the
amazing things that I should or shouldn't be watching. I'm guessing and
obviously you have a background Beyond just media in medical research and
hospitals by chains and so on, we talked a lot about data,
you know,
If you're listening to what give you two says, maybe you choose an algorithm, you think is right,
but ultimately garbage in garbage out. So data, quality is going
to drive whether or not your models are successful. Why don't you talk a
little bit about how we test data quality at scale?
Awesome. Thanks a lot for the introduction. So I'm excited to be here. I'm
happy to talk to everyone about Apache, Iceberg. So we can, we can jump
right into it. I'm going to cover something called the write-audit-publish pattern, which is a
general pattern. But it's specifically what? Well, implemented
via, Apache iceberg.
So high level overview and just you know what, I'm playing the cover in the talk,
I'm going to talk about data quality so you know what is what exactly does data quality mean?
You know, in the modern modern era of data, you know what are some common
patterns that that are inspired by the goal
to achieve data quality? Then I'm going to cover, Apache, Iceberg
just hi-low introduction on what it is and specifically its integrated
audits feature, which I think is makes it very
Very easy to implement this, this exact pattern at data
very large scale and I'm going to talk a little bit about the hard part which is
automating. This feature via your orchestration system so that people can use the
write-audit-publish pattern in a much more declarative way. So,
let's start with, you know, data quality, right, what exactly is
data quality. So if you look at Wikipedia, it says people's views on data quality.
Can often mean disagreement even when discussing the same set of data for the same
purpose,
it was on this day is the number of data sources increases the question of internal
data consistency becomes significant, regardless of the fitness for
use for any particular purpose and this is my favorite
sentence is that defining data quality. In the sentence is difficult,
and in reality, this is what we're trying to avoid, right data.
Quality really means that our consumers of our data don't
feel. Don't lose trust in in what is actually contained in the
data that we're delivering.
All right. So you know, the big question then is
how can I get people to trust my data, right? And here
are just a couple of solutions that, you know, if anyone's work with data has tried one
or more of these Solutions, at some point, right? You can write your data to
production and you can leave it to your consumers to run. Validations
another solution is you, you know, you write it somewhere else. You have like, a test data
warehouse that you write everything, sort of no one knows about. And then you
Audit your data there in private before you rerun your stuff to move to production.
And then, you know, some people are clever. They write data quality metrics and they
sort of ship that as along, with their data. So that all
consumers can sort of go. Look at metrics here, investigated for
whatever, you know, whatever their definition of data quality, is they
can sort of frame it in the context of these metrics. You provide and
maybe, you know, some combination of these three options.
Some of you are really, really expert data engineers, and you look at and you say, oh, it's not that
complicated. I have my own mechanism here. And many data Engineers, do many
dangers have really, really fantastic. Bespoke data, quality solutions
that work very well. And so, you know,
the big thing that I want to cover is that copying
data when you're working with data at massive scale, copying data is
essentially not an option. And that's the big thing that
That changes that sort of knocks off a lot of the solutions that you know
maybe were valid Solutions along time ago, once you
reach the scale of data that pretty much exist at any organization
you can't really copy the data and so you try and find out how to how can I
maintain data quality? How can I share data quality without having to do this
duplication of my data separately to actually run my
audits?
So I want to talk about Apache, icebergs integrated audits feature and just
to start at a high level. You know, what is it that you guys work for anyone? Who's never
heard of it? It's a high performance format. This is sort of the long
description from the iceberg. Doc, site my
description. I usually give to people is that it is Iceberg provides. Massive scale for
called native, SQL tables, and it's accessible by many compute engines.
And that list of compute engines as it is continuing to grow over the
past year, has been extraordinary.
Both in different compute engines that can access you know
commentator tables that rest Iceberg tables.
So just a high level overview on what the integrated audits Feature
Feature is. So the big core of it really is that
it allows you to write your data to production in an unpublished
State and by unpublished it means Downstream consumers. Can't see
that table. When they just typically query it directly as, you know,
writing a sequel query to select from this table, they won't actually see the
data that you've written in its unpublished state.
The other thing that it does, and this is particular, and a lot of this is going to be specific to this park
implementation of this. But this integrated, all this feature is that the core
Iceberg layer. So any compute engine can actually add support
for this and they're only a few now. But the featured the raw
material does exist there to implement in any compute engines and Spark in particular. The
way it's implemented is this spark web app ID value from your
spark session tags. The unpublished snapshot so I see
Has this concept of snapshots and when you write data unpublished that
snapshot is tagged with this write-audit-publish ID
from your spark session time, travel is a big feature, an iceberg.
So time traveler, lets you select any historical snapshot
explicitly. So that's a core feature. Not
specific to integrate audits but specific to Iceberg for
time traveling to select these. This unpublished data
And then when you have confidence in your data, Iceberg has a metadata
only cherry-pick operation, which lets you take the difference between this
unpublished data and the current, you know, sort of head of your
table that people have access to and sort of cherry picking the
metadata making a new current snapshot
to sort of make those unpublished changes published that's sort
of the equivalent of it. And you know, another one of the best
parts is that
Iceberg, snapshot expiration, cleans up a lot of
data that's unused, it's not referenced by the
current state of the table. So if you have a weekly or monthly snapshot expiration
process, any data that you don't end up cherry-picking or any data that
remains unpublished is automatically cleaned up and so this is an overview
of the core features of iceberg that really enable everything I'm going to talk about
in the next slides.
So let's cover like the three stages, right? So write-audit-publish
and and what what it entails right, what it contains. So for
write this, this right? That wept at enabled is actually a table
property. So by default swap is
not enabled on Iceberg tables but you can enable it simply
by setting this table property to true on your table and you only need
to do it. Once you don't necessarily need to do it. Every time you do a write-audit-publish
session.
And then the second one is in your spark configuration for your, your
job that you're running. You just need to set this uuid
in the spark session configuration. So when that
once that ID is set, then you're sort of it's the
the entire spark session that you're running in, you know, sort of
it's a signal to Iceberg that this is going to be followed the write-audit-publish
pattern. And then the best part is you just run your production
HDL code. So
So it sounds scary to a lot of people. But once you see this
enough and you use this enough, you'll gain that confidence that
Iceberg won't publish your data. Because you've had, you have a web ID set
on your spark session configuration. So you run your production ETL
code. AS is, you insert into your production table, you change. Nothing about your
production code except the setting the swap ID in your spark
session configuration.
So then what is the, how does the auditing happen? So the auditing is,
you simply have to find that snapshot ID for the
production table that's tagged with the the web idea that would set in your spark
session configuration. So that point that tag essentially is a
pointer to say. Okay, now I this data is unpublished, so I can't access it by
just clearing the table. I have to actually use the time travel feature
to select a, an unpublished, a different. Snapshot, that's
unpublished.
And the whap ID is sort of the link that lets you look up, okay. What is the
snapshot? That was created by my production, right job
and then you can perform these validations against the data using any data, auditing
tool. So anything that can that actually has support for Iceberg tables and can use time
travel, you can use for auditing. So if you have a spark based auditing tool, if you have
a tree, no based auditing tool, you have a fling based auditing to anything
that can actually select a specific snapshot of Iceberg.
You can utilize for
for your audits.
And then what happens after you're finished your Audits and you want to publish, what if your audits
fail? You can just go back to the drawing board. Remember there's the automatic
snapshot expiration, clean up that that is part of the maintenance of an
iceberg warehouse and so you can, you can be assured that any of this
unpublished data will be picked up automatically and cleaned out.
If you have a, you know, a number of sessions
that fail, you have data lying around, you don't have to concern
yourself with it but that will all get cleaned up. And if
Your audits pass then publishing that unpublished data. You
run. This cherry-pick operation of the specific snapshot ID that was
tagged with that write-audit-publish session. This is a metadata
only operation. It's super fast. This goes back to the cardinal sin of
copying data when you're working with data at scale. You really
you know it's just not feasible when you have like maybe
jobs that take 56 hours to run your moving massive amounts of data,
the great part about this,
This is the publish here is just a cherry pick operation. It just takes the metadata of the
unpublished snapshot looks at the current head of the table and then Cherry
picks those changes to to be available. Just a very
fast operation and immediately all of your data is
published and available to a downstream consumers.
So it might, it might feel like this, right? You're seeing all these
steps and you're saying, okay, I have to set this sparked rap session ID. I have to,
you know, look it up. When I went to audit it, right? I have to make sure that
I, you know, once those audits complete based on some conditional, I
publisher or don't publish. And
it, you know, let's zoom out on sort of what, what this feature is
aiming to do, right? So iceberg is a data quality in a blur. This is a high level of you in
What, what it's trying to solve here, with this particular feature, it's a data
quality enable. Enabler in that, here's a gross simplification
of a data Engineers pipeline, right? You have some data source?
You ingest some pipeline, you know, you have an ingestion pipeline which you
run your code for and then you do some manipulation, maybe some
joins and maybe some filters and then you put this data in your in your production
data warehouse. Now, if something goes wrong, if
something is wrong Upstream with your data sources, you know, that's
not the best thing in the world but your ingestion pipeline fails and things,
you know, you just tell your Downstream consumers. Oh the data is on the way, it's
not, it's not.
Ready. We're debugging some issues with Downstream data sources if the data
sources are fine but your ingestion pipeline, something's wrong with your code.
Similar ingestion, pipeline fails, you tell your Downstream consumers, you
know my pipeline failed, I'm working on figuring out what's going
on with the code and everything is you know
sort of people are waiting for the data but
you know it's it's not the worst case scenario which is
When something goes wrong and bad data is published to your production data
warehouse. And this this portion right here is where all of the anxiety
lives for. Well, I wouldn't say all most of the anxiety
lives for a lot of data Engineers is that you don't want to actually
production analyze or publish bad data and you can
think of that as we're this integrated audits feature lives,
it allows you to run your ingestion pipeline. It allows you to run its the case
when everything goes wrong.
When the data sources are solid your ingestion pipeline
runs and complete successfully, but you want to include
a set of auditing or checks before that data is actually made
available in the production data warehouse and you want to do that without
actually copying your your data, your massive amounts
of data that's actually stored in your pipeline
or process by your problem. So what it really means is no more running your data twice,
No more having to remember to clean up artifacts like test tables
no more having to remember to keep testing Pratt schema synced, right? So even if you could copy your
data twice, there's this thinking that you have to do every time you
migrate schemas for tables, which you always gets
challenging makes you want to change them less, there's no more locking yourself into a single
auditing tool. So anything that supports Iceberg, which is a very long list
of compute engines that are growing everyday anything based on
those tools.
Can actually be used to audit your Iceberg tables. And there's no more
coupling of your ETL logic with your validation logic, right? So you
can actually run your keep your production code as is,
and, you know, that that sort of allows you to store your validation logic as a
separate separate component.
So, you know, the hard part, really,
and this is sort of the last section in the talk. But,
you know, there are a couple of steps there, right? We see, you have to set the web
ID and your spark configuration. You have to ensure that you utilize that
when you're running your audits, you also have to cherry pick the right
snapshot ID when you go to publish. And so the hard part is
really automating this feature, right? As part of your orchestration system. So
I want to cover
what that looks like and how I've seen it done and also talk
about some of the challenges there as well. So
let's start without a high level. You have your orchestration system could be anything
could be air flow, could be a custom scheduler there many. Many
orchestration systems out there that come with scheduling your workflows
on the right. Is just you know simple version of a production table, you could have multiple
production tables but for the sake of this example, to assume it's one production table
and then you have it.
Auditing tool, right? So this is your generic auditing tool that you can use
to run the audit process of the write-audit-publish
pattern. So what's step one, right? So Step
One is, we mentioned verifying that, right? Dot web. That enabled is
true. Then that's a very cheap thing to do every time. So if you wanted to, just
have a check that says, oh, is this enabled on the table? If it's
not, then then set it to true. You can actually do that on every
workflow as part of
Of as an automated part of the orchestration system. You can also leave this
outside of the orchestration systems job and just have the users enable web on
their table before they start using this pattern so that that's that's
another option there as well. But it is a cheap enough metadata operation that you
can. You can check for this every time.
Then the second step is you run the spark application, right? That the user scheduled as part of
the orchestration system and here, the orchestration
system should take the responsibility of generating the Run
ID, all right? So you shouldn't have to actually think of some
you some ID for this web session, the orchestration system should
automatically generate it and usually orchestration systems. Have some concept of
a workflow instance, ID or run ID for this particular
Execution and oftentimes, that's enough to just utilize that.
As the as the, the Run ID solar system to
grab that it should actually inject it into the spark session
configuration for that. The users job is going
to run in
And then, you know, once that job finishes
successfully the registrations and also should trigger the audits, right? It should trigger the
auditing tool. Now the auditing tool has to, you know, has to
look up. What's that, what's? What actual snapshot should I be auditing?
So you need to have this very declarative way to make this as easy as possible. You need
this declarative way in the auditing tool to say hey run these audits for this
specific snapshot of the table because then it's a matter of just finding
out what snapshot is.
Is tied to this execution ID and then providing that snapshots of the
auditing tool to the actual audit executions so
that they can run. And then when the audit is complete,
it will give you a sense of, should I publish this data or should I not publish this
data? And that signal should be returned to the orchestration system
which if the auditing tool gives a published signal, it runs the cherry pick operation
and specifically for the snapshot. That's
Packed with this, run ID that orchestration system itself
in created and set on the spark session.
So this is kind of like high level how an
orchestration system can really Orchestra system sort of the
core component in this, how we can orchestrate this and automate this feature. So that users can
actually just provide their their ingestion Pipeline
and provide some audits as well. Defined some audits as part of some
auditing tool and have this taken care of
Italy for them, by by really the system is orchestrating the execution of their
of their pipeline. One thing I want to say is this part right here is
super tricky, it seems to
be very easy, right? Publish or don't publish, right? If the
audits past published the
data, if the audits failed don't publish the data. But there's a lot of nuance there
and this is usually where human intervention is.
Often required or requested explicitly.
So to give some examples of that. Oftentimes, if
the audits fail, there are different. They're
different hierarchies to audits, right? Some are called week, what we
like to call blocking audits, right? These are audits that under no circumstances, should
the data be published, right? So this is something like, if you're expecting millions of
records and you get under 100,000
records in the status in this,
Ian, something's wrong in the should never be published. And then there are
non blocking audits which even though they fail their more for notifying
the ingestion owner that they want to maybe look historically and Co this. This
check that I do fails every Friday and Saturday
when volume picks up. For example, that that is
very different from
That is the reading room. Something you want to actually stop the pipeline. You may actually want that
publish every so this different category of audits
but then there's also the blocking audits that
Phil where users actually want an ability to override that failure.
So that's another level of control that users will request. Is
that? Hey, when my audit my blocking audit fails, you know, I
want to be able to do some investigation you sort of pause the
At that at that stage let me do some investigation. Maybe, I've talked to some
Upstream, some owners of the Upstream tables that I use
and there's a good reason why this audit failed. I'll adjust the audit next week for
now. Let me just skip it and so a lot of a lot of
nuance exist in this stage right here, the rest of this not so
much, although there is some as well. But but this in particular is usually the
one where you really want to curtail that user experience to something that's
intuitive to
To really the people who are creating these jobs and using
this pattern.
Some of the gotchas I want to cover some of the gotchas to be aware of so
pipelines that, that both right to and read from the same Target
table, something to be aware of their. So Iceberg
has, it's actually a really cool feature where when you have that
spark session, that whap ID set for the spark session,
when you write to the table that's unpublished. And then you read from that
table, not specifying a snapshot. But just reading from
Latest view of that table, and understands the concept that you're still within this
write-audit-publish session and it will actually return
the unpublished data as well. And that's useful when you're, when you're
sort of having Justin pipelines that are multiple reads and writes from
to the same table or Reason rights as part of multiple tables
and it's very useful in that sense in that it allows you to do those read and write
cycles and actually publish and actually
combine them as a
Unit of ingestion or unit of rights that can actually publish together or not publish
together where that's tricky. However, is the with the web ID,
right? And in success, there's no issue there. But when the,
when the pipeline fails, you have to ensure
that you don't use the same write-audit-publish ID
when you run that next batch, because then that feature can
sort of be unwanted, right? Because when you're starting a new Fresh session,
you want to start
The new fresh whap ID when it refresh that web ID because this is a new lab
session in in situations, where you're not writing and reading from the
same Target table. You can reuse the same web ID for multiple
succession is because you're not actually reading from that from that data.
But keep in mind that this can, this is a potential.
Gotcha for this you for this particular unique scenario. Another one
is are running parallel parallel whack jobs, so
parallel
I'll whack jobs right now and this is a great example to think of our back feels right. If you're running
if you're doing daily backfills for the past year, for example,
you may want to launch these 365 jobs in parallel.
Each utilizing a write-audit-publish session
and that all works and particularly
they all work. When you're auditing, each of those individual
day separately, one area that is
still has room for improvement.
Me is when you're running parallel whack jobs and you want to actually audit the
combined view of it. So you actually want to write right? Run the backfield for
365 days. But then have a combined view of all of these
multiple web sessions and do an auditing. There it is possible.
It's actually it's very possible to do if you're sort of using the the web
ID yourself. But when you're automating and orchestrating this, it
becomes very tricky because you need to actually have a higher level
trucked of a web session of a collection of web sessions that are
related and need to be audited together,
the new branching and tagging work that's happening. In open-source Iceberg, what will really
solve this problem? Very well. So, that's if this is a particular use case that
you're interested in, it will be great to keep an eye on what's happening there in the in
the open source community. And the idea there is that, you know,
if you actually, you know, you can create a branch of the head of your table and actually
becomes sort of
Just as you would think of a branch and get it becomes a branch off of the main and it's a named
Branch, right? So you give it an explicit name. So instead of actually
tagging wet unpublished snapshot IDs, you can actually
create a branch at the beginning of your session and you can utilize that Branch as
part of these parallel whack jobs. So what all these parallel,
web jobs can actually run their own rap sessions, but published to the same branch
and then you can actually utilize that Branch name or that branch.
Eid to get that Collective View at the end when you run your audits.
So Branch contains is super exciting feature. That's part of the iceberg
spec and the implementation is is currently actively being work on
work on. And this last one is,
is really more for the orchestration engine. So the overhead of the web
step is small. And when I say about five minutes,
really, the like checking for the table properties about 200
milliseconds,
Running the actual publish is probably two minutes or less.
So where the five minutes really comes from? Is if you do this as part of separate
steps and you're actually running this on running these
containers on a, you know, through via containers on a spark cluster, the
overhead of sort of starting up each of these steps individually
are unnoticeable 43, our
batch ETL jobs, but it can be significant when you're doing
parallel job. So that's something to keep in mind.
Mind, especially if you're doing smaller running jobs. That are about 10 minutes,
the overhead of the additional five minutes, backfilled over
365 days and you're running these in each of your back, each of your
10 or 15-minute backfills end up becoming 20 minutes as a percentage,
there could be some overhead there to consider.
So that's something to think about there are ways to sort of optimize it by keeping that keeping
the the web operational steps closer to the actual Sparky.
ETL run that we can utilize some of the same resources to perform these
steps but that's a small got you? I figured it would be good to
list here.
So that's that's all I just put a bullet here to like, check
out the iceberg open source Community. There's lots of ways to
contribute. It's a big Community, that's growing lots of people from different organizations.
So there if you go to the iceberg that Apache
dot org slash Community page, you'll see lots
of ways to join the slack channel. The weekly saying there's a
community Google Docs where you can add different agenda items.
So there's a number
number of ways to join and start contributing there.
Thanks everyone. Thank you so much Sam. And it
looks like we have a little time to discuss some of these things. So
fortunately we can jump to Q&A here. Really. Appreciate you going through a lot of this
stuff. I have a bigger question about the, what you're seeing, as the
sort of business requirements around auditing and data
as companies start to rely more and more on automated models and more and more
of those decisions are being scrutinized by the public, obviously, recommending the wrong
movie is
Not bad, recommending, the wrong drug is probably really bad. What are you seeing
happen in terms of Industries like law and
governance? Starting to look at the output of these systems to
understand whether data is valid or what there's liability involved.
Yeah, that's that's a great question. I think that the, you know,
in my personal view of, or my personal experience has been
before as data velocity, as data scale, started to grow
tremendously. There was a big question on just, how do we process
this data, right? And it was much less about. How can we pop
process this in a way that ensures data quality? It was just how, can we process
this period, right? And so, there was a big sort of
You know, revolution in how we manage databases right? The
data warehouse was born, Open Table formats are
you know, here to stay. And so a lot of these new things were sort of created
on just, how can we process data at scale in a way that
we're not we're not falling behind on the amount of data that were processing.
I think now that those technologies have matured data quality
at scale is sort of the new the new problem that's trying to be
solved. And so you see a lot of new things.
They're showing up, you know, data quality tools being one. But even like these
concept of metrics layers or semantic layers that have these data
quality built in dire the data quality checks
built in. So, I think that right now, this is
probably a moment where we're sort of taking all those mature data processing
technologies that are that are really, really made incredible, incredible
things, capable at truly, a
ridiculous skills.
And finding out, how can we provide, how can we have that same Revolution for
finding how data quality should be done? And how can we do it,
reliably and inject that as with the same level of engineering that
people are creating these ingestion Pipelines?
So let's talk a little bit about
what Iceberg can do to backfill stuff when there's a downer
downtime or an outage at some point in the data structure, kind of lives that leaves a hole in the world, right?
Like your bass lines, don't work anymore because there's a gap there,
there's obviously a difference between like not having any data versus having
anomalous data and you know, it's hard for code to
deal with or four models to deal with.
And changes are gaps and data. Can you talk a little about backfilling? And and
like, how you would go about back filling a gap in data once that
data is restored and then like redoing the analysis
to fix your models.
Yeah, yeah. So backfilling is a, you know, is
is really probably one of the most complicated Arts of data engineering.
And there's a lot of challenges there, but Iceberg and
particular. Its snapshot feature is really core
for for backfilling will two things I'll say so. Snapshot feature is very core
for for auditing and also the ability, the fact that the
rights are Atomic so you can actually run these
These these jobs in parallel and the
actual publish step when they're made available happens instantaneously through
through sort of a metadata commit. So that allows you to sort of
parallelize these various backfills at super large scale
on the other thing about you mentioned sort of like having,
you know missing data or bad data, right?
You can you can actually roll back super easy with
with Iceberg as well. So these snapshots
Owners make in those cases where maybe you do have bad data, right? Maybe you did perform a
backfill and it, you know, ran for eight hours and then you found out that
something about your backfill logic was actually off and that may be
injected some some nuances in the data that makes it fundamentally
wrong very easily with Iceberg with a metadata
operation that runs and under you know probably a few seconds definitely under a
minute, you can roll back, you know, petabyte size tables
to to the
Vyas to the previous snapshot that existed before the backfill.
Are there competing formats? Like Delta hoody or Iceberg
and do you think we're going to see, you know,
they're valid reasons for those three things Beyond sort of each vendor or each
group wants their own format? Or are we going to see interoperability between those down the
road?
Yeah, that's an interesting one and a harder harder one to predict.
I think these were created, you know,
at a time when neither existed, right? So, I don't think necessarily that there was, you know, a
mature product that exists in and someone made a another mature product, as sort of a
competitive competitive product. I think they were all
created at separate times from, you know,
organizations with really really strong requirements.
Sort of building things that happen to sort of converge on this concept of an Open
Table format. So I think the concept of an Open Table format is
here to stay. You know, I know I'm much more familiar with with
Iceberg. And the thing that I would say that Iceberg has a
pretty good Advantage. There is, that's always been open sourced from the beginning. And it
has a, the table spec separate from the implementation of
the actual Java implementation. The table spec is completely laid out
on the dock side.
So you can see the like every Nuance of the spec which is very
solid and sort of all implementations follow that spec. So there's a little bit
there that I think Iceberg has in terms of adaptability and
integration has a little bit of a better story, but as far as you know where
where where this all goes, we'll just have to wait and see.
It does feel like convergent evolution as you said rather than sort of
specific differentiation a couple more questions.
It's the data would probably be event-driven in your data lake or Lakehouse.
Then auditing would be natural. And I'm presently, Iceberg
could help.
Yeah. So, for adventure, and so you could, you could actually use this for like if
you're running streaming streaming data, if you're for
examine the Flink application, you could implement this. And I think that that adds the other
dimension of your checkpointing strategy, and when do you want to run
your Audits? And over what interval? Right, if your checkpointing every two minutes, you maybe
don't want to run your audit sweet every two minutes, but you
absolutely could use this as part of as part of a
yeah.
You're an adrenaline, event-driven pipeline.
Yeah, it's fascinating to think about, you know, the data Ops person is now wearing the pager
we used to think of that as like, you know, someone unless the hardware is
true, unless there's a problem with the hardware or like, the database itself goes down. But
now, you have people instrumenting and like everyone's on pager GT. Now, they're all
getting you know, wherever the data, flows are two more quick
questions. Any tips on implementing automated testing for Enterprise data warehouses?
so,
Yeah, so at Netflix, we have a homegrown tool
that we use for that. There are tons that are out there
that are available. I think it really is, it really
is unique to the type of data quality that you're doing and the type of data that you're doing
and the particular engines that you use. So, that's sort of the
compatibility that would look for, is that it has, it can integrate
with the particular compute engines that you're using and that it has the right
Suite of
It's that that you need and that that really is the nature of the data,
right? If you're running, if you're, for example, doing ml data, and you want something that's
more statistical statistical, heavy with statistical, auditing
functions. You may use that if you're using doing financial data. For example, you might want
something that's more, has more forecasting features and maybe you can
audit based on forecast, on on each daily
partition or something along those lines. So I think that one is pretty
depends a lot, but the good thing about
Iceberg is that in this particular pattern, is that you can inject really any auditing tool. It
brings no opinions about that.
Awesome, last quick question. You've
obviously had worked as a software engineer in the
very what sort of link for the very like life
important medical stuff, where they were or false
negative is really bad. And you know the
sort of luxury world of watching content, where a false
positives, not that big a deal. What has changed in your approach to
software engineering.
At those two extremes of data science.
Yeah so that's a great observation. I think
you know manual
intervention manual human intervention is always
always been critical in you know clinical settings
for sure. So that so that that factor has been
always present that it's not enough to say oh you can Define these
Audits and code and they'll run automatically, you always have to think of that
review process. You always have to think of that. You know, how can someone look at this and
Sign off. And then the human element is always much more present in those sort of
Life, critical environments when it comes to a
product. When it comes to something, that's not life critical, but more
a user experience that you're really trying to drive
latency and, and
velocity sort of being able to run these automated at High
Velocity is sort of the bigger requirement. So, the difference that I think is you can be much
more Innovative and creative and there's a
a lot to gain from getting you know audit audit checks there for ninety
percent ninety percent of the time, if it increases your velocity and increases
your productivity to a, some order of
magnitude,