To illustrate how powerful a lot of the go concepts are such as channels, go
routines and interfaces. I've got a 62 line program here
which is actually a scalable work system. So what it can do is
read line by line from standard in a list of jobs to do,
and then run those jobs in parallel, collate the
output and output that on standard out. And all of
that is in 62, lines, it makes use of go routines.
Channels for fan-out interfaces for defining, how work can be done and I'm going to show you
how it works. Then we'll make an actual program, using it, that does something.
So, first of all, I've defined an
interface called task, a task
is able to do two things. It has
a process interface or a function which is called to process
that task do whatever is actually needed and it has a function
called output which is used to write the output.
So will actually send out send out the output. There is also an
interface which I call a factory and that is a thing that can create a
task given some string. Now that should be a string that's
received from standard in. So the idea is, this program will read from standard
in call, create in your factory, make tasks and then run
them in parallel and use output to get the output.
So look at how this operates. So the Run function takes a factory, which is a
An interface and it runs a couple
of go routines. So the first go routine is using buff
IO new scanner. Now the Buffalo scanner is a very nice thing. Let's
let's bring that up here.
So there's a scanner which is a thing for receiving,
essentially, new line delimited pieces of text and it
automatically wraps. All the usual things you do for reading. It doesn't actually
scan Fields out. That's up to you, but it gives you the line by line and you just
make a simple for Loop like this. And at each time that for Loop
is called if there's a line available, it will be available here in s
text and you can do something with it. And in this instance,
What we'll do is we will call the create method on the
factory that we have to create a task and pass it down a
channel. And that channel is this guy is a channel that are so they'll just be this channel
will get filled with tasks to do and it's an
unbuffered channel. So somebody loves you have to be willing to do that to handle
that task. Until before, that can actually be transmitted
if anything goes wrong, then we'll send an error and when we're done we close that channel to say. There's no more
work.
And I'm using a white group to count go routine, just so I get
clean termination of the end. So I've created a weight group for every go routine, I
add one, go routine and every time, one terminates, I call
done. Now, to do the work, I actually need some
workers, and I've hard coded in here that I'm going to create a
thousand. So I'm going to create a thousand go routines and I'm going to use the weight group to count
them and each of these go routines is really very simple. It's going to pull a
task off of
The input channel in the task is T. And if you remember that
task is an interface that has a method called process. So for
each task it is going to call process to do whatever that task requires
and then it's going to Output that down and output channel. So
what's going to happen? He completely naturally in go is load balancing across a thousand. Go
routine. So a thousand go routines are going to try and listen on in
that's what this range in does. And one girl routine, this
one that's really
new standard in is going to write to in. So imagine a thousand, go routine start.
Somebody starts writing and they just get picked up automatically as there's work to
do. And when in is closed, then what will happen is?
This will terminate. This Loop will end will call down on the weight group. The goal team
will terminate so as soon as we close the in channel to say that we're done, there's no more
work to be done. All those go routines, will terminate
You also have a separate goroutine here and it's waiting
for all of the go routines to terminate, and then it closes the out Channel
and what's happening in the output. Is finally the program reads from the output so
you can imagine. Now this is the fan inside, those thousand, go routines,
all writing to the channel and this output will output all of
the output from all of the tasks that been processed. So right
here if you just Implement your factory and your process
and output method,
It's on some type, then you're done, you have something you can scale.
Obviously, a thousand is hard coded, it could be any other number. We could use the flag package to set
that and typically you might want to do that because you might want to control, how many simultaneous tasks
work. But really in sixty two lines here. You've got something, which could be quite
complicated to write in a language. There's no explicit synchronizations
and no need to worry about how that's going to work. There's no load, balancer. That's all
part of the program itself. So now that I've got this flexible work system where
they can scale up,
Up or scale down. Let's do something concrete with it. And what I'm going to do with it is except on
standing in a list of URLs web pages. Go get those web
pages in parallel and output, whether they were successful,
not to whether there was any sort of error getting those pages and to do
that, I'm going to have to define a task and a factory. So the task I've
started here, the task is going to have a URL. That's the URL to get and a
Boolean which indicates whether getting that page, worked correctly, and
you've got a couple of functions. You got to
The output function is going to be pretty simple. I'm just going to print out the URL and I'm going to say
truth value. Did it work or not and the process functions a little more
complicated. We're going to have to use get to some Network stuff here. So
we can have a look at in the net HTTP package. There's a really nice little function called
get. And what get does is it gets a URL gives
you a response and an error. So I'm going to use that. So I'm going to import
that up here.
So we're pressing a task going to say response, error equals
HTTP, get the URL. And well,
if there's any sort of actual error occurs, then clearly
things are not okay, and we're going to return from there.
And then going to take a look at the response because an anything other than an HTTP error 200, we
my indicators it problem. So we can have a look at that and let's just use
go doc to look at that.
So, the response structure actually has the status code here is an
integer. Okay, we can do that so we can say response
dot status, code goes 200.
And in fact within the net HTTP package there's
actually nice defines for all those things you can
find in here. There's so let's be, let's be good. Program is not using
magic numbers so status. Okay. So if
it's okay,
I'm going to set it to true, and we'll be done.
Otherwise.
I can set it to false explicit. He's been necessary to do that because the fact that by
default will be false, but there you go. So that's the process that I think. So, an
individual task me process like that and because it supports process and
output. It's a task and now all we need to factory and if you remember the
factory is the thing that's going to read from every line. So
let's just make one of these.
This isn't going to be very interesting cuz it doesn't have any anything ready to do. But I wanted to
find a function on it.
And that's remember it's this thing create. So I'm going to get a line given to me
and I'm going to return a task. And in this case, I'm going to return an
HTTP task. So we're just going to say
tasks
And we do this. So, very simply with your say,
let's create an HTTP task set,
its URL, to be able to the line and
return it okay.
Talk back. Let's see if we can type things correctly and I'm going
to need for Matt as well.
Okay, so that compiles nicely. So now it's just a question of hooking, this whole thing up. So I say run
and the Run thing requires a factory. So let's make ourselves a factory.
Okay.
That guy's running. So it's running now, waiting for some input, so, let's actually try
something out.
And if I let us build it as a as a program and then it's
good.
So he went off and did that, it started a thousand go routines around the whole thing and it
managed to get that particular page. So that's good for one thing.
Now, let's have a look at doing something like this and that one false because in fact
that would have created a 404 error on that particular web page. But will be
more interesting is if we had a number of URLs. So let's just make ourselves a quick little
file here.
And Okay. So
Okay, so there's some files and we can do this.
And so out, they come as it works its way through these things. See if I
Riley
Okay, so there they all are now. Those were all run in
parallel, but we could actually slowed things down. Let's just change this to
only have one routine.
And I was going to have to do them one by one and this would be much more obvious with many,
many thousands of Euros. So just to cheat a bit, I'm going
to do this.
Let's clean this up.
All right.
I got all those timelines Bliss. Just in here if the URL is blanket, and we're just
give up like this.
if stroke URL equals blank, then
That way in case is anything silly in the data and let's
build it,
and how it's going to go through fairly slowly, because it's still doing only one at a
time.
So let's do this. I say this in
value in which is stop that expended while doing that
and let's make n. The number of things were going to
run simultaneously as use the flag package
and
This account by default. 10 of those.
Number of workers like that.
And n equals talking. So we just take whatever that is and then we'll be able to use
that here with in here. So
Okay, so go back to that.
Now we got 10 running so you can see things are spinning along a bit, a bit
faster.
And what I can do is I can go in here and say,
Count equals 1000. So now it's running a thousand. Go routine simultaneously
running through those things as fast as you can. As you can see, some sites are actually
faster to reply to us than others. So it's actually done everything in the
entire list. That was in there. Let's throw in a bad
one.
To make it a bit more interesting. So you can see there, it was right there
and you can see it's running them in pretty much any order either coming through and just getting
executed as as they're available. So, sixty
two lines of code and you've got a completely flexible system. You could obviously
change this to not take from standard in, but the basic
concept is here. And in fact, this program, I've used hundreds of times for different different
purposes because it makes a very very quick.
Parallel system.
So if you look at the output from before, you might have been surprised that everything seems to be in
order in some way, all the cloudflare was coming first and I think it was Google and O'Reilly in
Yahoo and you may have been wondering if there was some sorting happening inside.
In fact, the ordering is coming about because of the speed of those relative websites and to find
that out, I can actually change the program a little bit. So what I've done is
I've added to the task, a couple of other
values, these are time durations one is elapsed which will be how long it
actually took to.
The web page and the other one is starred. Which is when did this particular task
start processing? It should be the case that given the number of URLs we
have and we have lots and lots of workers. They should all start pretty much the same time
and then what I'm doing is when I output it, I'm going to say when
did it start relative to the beginning of the program? And also how long did it take to
run? So if we just build that
oh and also I keep track of when the program started here. And when the factory starts up,
Up I keep track of what the time now is as if we do that, and
then go back and run through those URLs or with the
Thousand workers we can run through. And again, you can kind of see
the ordering. So they started around between
three and six milliseconds, into the start of the program, various things started
running. So they all started running with a few milliseconds of each other couple of
milliseconds, but the times to actually finish downloading varied a lot.
So for cloudflare, it was
In the sort of four to six hundred milliseconds to get the complete page. Google
about a second or Riley little bit longer. You can see there was one Google that
was long there, Yahoo a bit longer etcetera. So the
ordering actually came about just because of the speed of downloading the pages and you can see
the faster, the page of the nearer, the beginning was. But looking at the timings here, you can
see that it was running simultaneously. That all of those
requests. At the same time we're executing without really no effort on our part.