No, I'm excited to introduce Matt. Welsh here for a talk, on
building applications using large language models. Matt is the
co-founder and CEO of fixie, .a I a
start-up developing a new Computing platform for AI based
applications. Previously was senior vice president of engineering at octo ML and
he spent time at Apple and Google and as a professor at Harvard
and he's also the co-author of one of the first O'Reilly books ever bought
running Linux. So,
Really happy to have you here, Matt, go ahead and take it away.
All right. Thanks so much for having me. Yeah. It was gosh, maybe almost
30 years ago that I wrote my first
O'Reilly book, I guess it was. My also my only O'Reilly book but it was the best
O'Reilly book. So once you've written one, you're pretty much done
today. What I'd like to do is talk to you a little bit. I mean, Maltese
talk was a great overview of some of the
challenges around building applications that use large language.
Models. What we're doing at fixie is building on
top of some of those same ideas, but also enabling people to
connect their own applications using llms
with their own data and not just static data, but also live
data coming from things like apis, as well as using the language
model to to drive application logic and so
we're seeing a lot of interesting potential applications here.
So I think everyone here is probably freaking out
about chatgpt and these Technologies
this guy in particular, he should be freaking out that he's using a cell phone
from 1997, I suppose. But I think one of the reasons that
the world is kind of going crazy around this Tech stack.
Right now is that these large language models are a bit
like an alien technology that just landed in our backyard and we're just figuring
out now how
How to make the best use of it. You know it's not like
we've seen this very slow, gradual Improvement and
in natural language models over the course of years
and now they're just starting to get fairly good. This
this transformation has kind of happened overnight. And
you know, this would be as if, you know, computer
Graphics went, you know, in the span of something like six months from,
you know, having the quality of
Of pong to, you know, Red Dead Redemption to.
And so I think this is the reason that most of the world is going a little bit
crazy here. So
the interesting thing about large language models is that
you can think of them like a virtual machine that you can program
in English. And this is a really interesting
characteristic of these models. The interesting thing about it,
one interesting aspect is
is this was been discovered empirically that is
that large language models were not previously known to have
a kind of reasoning ability. In fact, they were, you
know, designed to autocomplete text. And
then, you know, people were discovering empirically
that you could actually get the language model to follow
instructions and to take a complex problem statement and
Break it in two steps and then execute those steps one at a time
and manipulating a kind of world model as the execution precedes.
So
You know, let me show you an example of how that might work
when you program a large language model as a virtual machine,
you can effectively teach the language model, a new
trick on the fly. If you've seen the movie, The Matrix, it's like that scene
where, you know, they need to learn how to fly the helicopter, all of a sudden. And
all of a sudden they have learned how to fly a helicopter.
So by sending the model small
number of examples of a task,
Ask that you want to perform you can teach it a new trick you can say. In this case
I'm going to teach the language model, how to look up stock prices and how to answer
questions about stock prices. In this case, what I'll do is I'll say
what is the current stock price for Apple and instruct the model that what it
needs to do is call some external function to
actually get the data that it needs. Because the language model does not know
internally what the stock price is.
But assuming that the model can do that and it gets back a response
from that function, then the share price can be shared
back as part of the response. And so the missing piece
here is the ability to connect the language model to
some external API or data source and
this turns out to be quite easy to do and it turns out to be
an extremely powerful abstraction for building
new software systems,
So where I kind of think a lot of this is going to be going in. The future is
we are going to start treating large language models like
a new kind of computational substrate. That is
the language model itself is the code that is we're not using
the language model to generate code and then compiling the code and running it on the
machine but think of the language model as the machine that you're
programming but you're programming it by teaching,
Get a new skill in English and so you're teaching the language model how
to interface to an API, you know, maybe how to pull data from a
database, how to transform data in certain ways and and
also potentially how to use software that was originally meant for humans and we've seen some
really interesting demos and products built up around this
idea of interconnecting, the language model to Conventional software
systems. And this is the reason that I think the
industry is going berserk.
Around these ideas right now is primarily because this is such a
powerful and new way of instructing machines
that it's, it's having a dramatic impact on our entire industry.
Okay, so the question is, how do we get all these things? How do we get there right
today? You could start by going and grabbing
an API key from openai and starting to feed
prompts into a model and getting back responses. And then,
you know, you need to do something with those responses. But this turns out to be,
very cumbersome, very low-level, way of doing things. And where we see the world
going, is kind of a new programming abstraction that's built up
around all of these ideas.
He's so say you've got an amazing large language, model
app that you want to build. How do you do it today? Well,
the first thing that you need to do is figure out which model you want to
work with. And today, of course, most people are using
openai as models. But, you know, there's also anthropic
models and there's some open source models, like llama and alpaca.
And there's Google is coming out with its models and other companies are coming out
with theirs. I actually
Fact that there will be a potentially hundreds of open-source
large language models that are competitive with today's gpt-3,
for model in the not-too-distant future because the technology
for training these models and the datasets that you need are becoming
much more widely available. So you got to figure out which model you want to use,
okay? Then of course, you've got to build connections between
your application and the model and whatever data sources, you
want to interface with, whether it's
It's databases, web pages, S3, buckets, external
apis and multi is talk showed examples of how to do that.
But then of course you've got all of the, let's say fun,
Cloud crap to deal with and what I mean by that is where are you
hosting all of this logic? How are you interconnecting? All these pieces together.
Many of these things require specialized
infrastructure like a vector data base like we V8 or pine cone of
course you need to have you know, your
Chattering, you're testing your evaluation, your authentication, all of these
things need to be built up in order to use this stuff in a production environment.
So I hope that, you know, I'm kind of setting this up for you
is you know of course, Matt isn't there a better way in? This is maybe
where this talk takes a left turn and sounds a little more like an infomercial.
But I do want to show you how we're thinking about this
at fixie because this is an example of how I think
the platform.
Form for building large language models is going to evolve in the future. We
have a particular way of thinking about this at fixie, but we're also seeing
plenty of other open source projects and Commercial companies
building out their own kind of platform.
So the idea with fixie is this is a cloud hosted
service, it integrates all of the pieces you
need to build large language model applications and in
particular it incorporates a wide range of models under the
hood as well as all the cloud infrastructure pieces that I mentioned earlier
and the key idea in fixie is that, you build? What we call
agents, where an agent is a large language model, plus,
a little bit of
Code that knows how to talk to some external system. So for
example, a database agent would go from English queries to
SQL queries and fetch data from a database. The GitHub
agent would go from English to the GitHub API and
fetch information about your pull request, or issues, or whatnot. And
our idea is that as you build these agents, first of all the
agents are super easy to build as I showed you just a few slides ago.
The idea of instructing
the language model to do a new trick is not that hard
and once you plug these agents into the back plane, if you will,
they can communicate completely with each other in English. And so,
the idea of building up an application becomes a matter of
composing, the abilities of these different agents, that could be built by different
people and even running in different places on the internet and having them
communicate with one another in English.
I'm going to show you a quick example of what that might look like in a customer service
context. So in this case we have
a user who's gone on to a website and you know maybe there's a little chat
widget on the website that they can type in a question
we've all seen these things pop up. So in this case the user saying
it, you know, I ordered the wrong size t-shirt, I'd like to get one size larger,
this ticket, plus some other relevant context would
Flow into the fixie platform and fixie. Using the large language model, would figure out
what are the sequence of steps that need to be taken in order
to handle this particular ticket because it's a return
ticket. So they're asking about an exchange. So
in this instance, the ticket would flow through a set of
agents that individually, use the language model to do things
like look up the customers order history. Check the
stock to see if the
The t-shirt is still available, generate a return label by calling,
like, the ups apis for that. And then, finally,
drafting an email reply, that could be sent back either directly
to the customer, or potentially reviewed by a human before
it's sent out. This is just an example. And I think the
important thing here is that the sequence of agents in the
sequence of steps that are invoked really, comes down to
the
if Akin put that was provided and this is the kind of Rich powerful
symbolic manipulation that large language models are so good at
Another thing about fixie is that we allow agents to process
any kind of data or any kind of media. So it's not just text,
think about this, a little bit like email attachments,
where, you know, I can send you an email with a image in it or a
PowerPoint deck and you as a recipient of that
email know how to deal with those attachments in most cases.
So in this case, if we were to say, you know, generate an image
of a red panda and
get on top of the night sky fixie would build up a workflow
that first generates the red panda image. Second generates the
night. Sky image, third uses a different
agent to mask out the background of the red panda. And then
a final agent, that Composites, the two images together to give
you the final result. And again, you're doing all of this directly
in natural language and the individual agents. Are executing those
steps.
So, as far as you know, how do
you build agents in fixie, the
key idea in fixie, is that you use a set of few shot,
learning examples. Those were exactly those few shots that I
showed earlier for the agent that generates
stock prices. But then you simply augment it with
a little bit of code, that knows how to call out to some external API
or access some external data.
De and the idea is that this is just so easy to do. Most
developers can build an agent in something like 10 minutes, the
GitHub agent that we've built is took me something like 20 minutes
to build. It's just incredibly easy to put these examples
together and then just provide those little low level function hooks that
need to be invoked. The reason this ends up being
interesting is the language we're leaning heavily on the language
model to do the heavy lifting here.
R, right? It's responsible for interpreting the query from the user and
kind of transforming it into these underlying function
calls and those are, that's what the language models are good at. So we're using
that as much as we possibly can.
Few people have asked on the, on the chat here
about Lang chain and a lot of people ask me this question so I'll just address it here.
Lang chain is another way to build agents in fixie. So
Liang Shan is a this fantastic Library. It's in Python and typescript
that let you build up large language model applications
fixie supports it. We have a simpler model here for
building agents and we kind of think that Lang chain is useful in the cases where you
want to have full control.
All over what's going on and have more sophisticated agent behaviors.
But if you want to build simpler agents than this approach here
is pretty easy to do.
The other thing we can do is build agents automatically without any
real code. And in this case, I'm showing it as code. But you can pretty much think about
this as just configuration here. What we're doing is
we're providing a list of web pages in this case.
The Wikimedia Wikipedia articles about the TV show,
Silicon Valley and now I have an agent that knows how to answer
questions about Silicon Valley so I could go to this agent without
writing any additional code. Just this
Configuration here and ask questions about
about the TV show and what happened in it.
What I'd like to do is just in real few quick minutes. Here is
just show a live demo of this working in action
and I want to emphasize that this is all very
Rough and Ready, kind of early beta quality software. So you'll forgive me if things
don't work out as expected. So, what we're looking at
here is the fixie developer dashboard.
Keep in mind, fixie is a platform. It's not a website.
It is not.
You know, going to a different web page and chatting with fixie, instead of say
chatgpt instead, it's a set of apis that you can integrate
into your own application, but we do have this developer
dashboard to make it really easy to experiment with it. And what you're going to see
here is we have a bunch of agents that have been built. That
each of these agents knows how to talk to some different
kind of system. For example, we have a
pillow, a
That invokes the python pillow library for manipulating images. The
Google Calendar agent can answer questions about your calendar and create
events on your calendar. We have the stock agent that I mentioned
earlier and so what I can do is, you know, pick any one of these
agents that, you know, I might want to talk to, and get
it to answer some questions for me. So, in this case, I'm going to
chat with the GitHub agent and I'll say, you know, how
How many issues are assigned to in be
Welsh which is my GitHub username. And what you should see
here is fixie is sending this query into the agent. The agent
is converting that into a query against the
GitHub API. In this case, we happen to be using a
SQL database. It's not a database,
a SQL interface to the GitHub API, so we convert the
English to sequel.
And then when we issue the query against GitHub, I get back the answer, they're
21 issues assigned, him D Wells which actually makes me nervous because I
didn't know there were so many and then I can say, you know, show them to me.
And when I do this, i-i'm sorry,
this is the downside of doing demos live is that we're changing things all the time,
and sometimes things don't work. As expected, let me show you the version of this that I had done
previously.
And what I say here is, you know, show them to me. And it says,
okay, I will go and fetch the open issues, assigned to MD, Welsh and
return them as a list. So this worked pretty well
in this instance. So now we're going to come back and look at another
example, where what I might want to be doing here is
fetching some data from the web and then visualizing it.
So in this case, I said make a pie chart of the top 10 countries
by population and fixie takes
in this query and it says, okay, the first thing I recognize is this is a
multi-step tasks. I first need to find out what are the top
10 countries by population and it sends
that query into the router, the router in fixie is responsible for
deciding which agent should get the call and sends that
into a search agent the sir
Agent gets the query says, look up the top 10
countries by population does a Google search for that information
and comes back with the list of each country, the top 10 countries and their
relative populations. Then we send this data
back up into the router and say the second step is to generate
a pie chart from this data, we send down the data
into the chart agent in the chart agent. Generates a pie
chart
Next, if I were to say, make a bar chart instead, in
this case, I didn't say make a bar chart of the top 10 countries by
population. The the
conversational context is present here and therefore, it's really,
really easy to just understand that I need
mean to do exactly the same thing but just using a bar chart instead and so the
final result is a bar chart.
So I think this is just intended to show a little bit of the
power of building out.
These individual agents, getting them communicating with one. Another in English,
hopefully showing that it's really easy to do and all these things
are running in the cloud. So if this has gotten you excited, you know, definitely
head on over to our website. It's fixie F IX, I E.A I
and you can sign up for the developer preview. It's completely free to
use. You can build
You can build client applications, you can interact with the existing agents in the
system and everything that I just showed you should should
be working. Okay, so takeaways here
basically. We're just looking at large
language models as this new kind of computational engine.
And what we're seeing is that it's still pretty hard to
tap into their abilities. There's a lot of Machinery that you need to build up,
but once you build up that Machinery, we think that it be
Comes extremely compelling to start replacing conventional software with this
concept of natural language agents. And so, you know, my
general message is if you haven't been freaking out yet. Now now might be a
pretty good time to start. So with that, thank you very much
and if we have time for questions, I'm happy to answer any questions.
Thanks, Matt. We've got time for one question, which is come up quite a
bit here in the Q&A. It's about testing obviously with the current
error rates of large language models. Now, how are you going to test the applications that
you would build with fixie? Yes, exactly. And I think, you know, Maltese
talked earlier also spoke to that. I think that my expectation is
that any company, any platform, any system
that's building around large language models is going to have to have a solution for Quality
evaluation and testing.
The sexy the way that we're thinking about that is first of all, you can effectively
unit test the individual agents and that's really helpful
because an agent itself is fairly constrained and what it should be allowed
to do. And what kind of answers is going to give you, and now,
you need at a second level above that kind of rigorous unit
testing of One agent. You need to be able to have the same kind of integration
testing. Make sure that working multiple agents are going to work together,
effectively.
This is obviously an ongoing area of research for us and for many others.
And so, we are integrating a lot of capabilities into the fixie
platform to make that super easy and automated. But this is going to
evolve a lot over time.
Just like anything else in the space. It sounds like moving fast. It sure is.
Thanks very much for spending some time with us today. Great presentation, really appreciate it.
Thank you.