- One of the most important things you must do
when benchmarking is validate that the results are accurate.
I cannot stress this enough.
Use your gut if you have to,
but you can't just blindly trust what you're seeing.
Remember, we were just talking about, that your machine
must be idle, and there's lots of reasons why
your machine might not be idle.
I'm on my MacBook.
Even though I've got most of the things turned off,
there's still background programs running,
there's still network being used.
Then, a lot of times, you've got your CPUs.
If they have thermal scaling or they do things
where they don't go full throttle all the time.
All these little things are gonna matter when
you're running benchmarks, and you've gotta validate
your results at all time.
There was this article I read one time
that was using merge sort.
Merge sort is like the worst sorting algorithm
you could use in Go, because the way merge sort works
is you take this list of, let's say, a million integers,
you cut it in half, you cut it in half, you cut it in half,
you're cutting every half in half til you get to the bottom.
Then, you do your compares.
You might swap, but on that swapping,
after that you're merging.
Merging lists back, merging lists back, merging.
It's really an allocation hell type of algorithm,
and really one of these algorithms that you don't wanna
be using in Go unless you're gonna write some
very, very clever code.
The author said we wanna use merge sort,
and what I wanted to see, the author's like
"What I want to see is that if I do a merge sort
"over a million integers and I use one goroutine,
"how fast is that?
"If I use a goroutine every time I split that list
"in half, I throw a different goroutine at each half.
"An unlimited number of goroutines, depending on the size
"of the number of things that we're sorting.
"Is that faster, or if I use just enough goroutines
"that I have cores.
"I have an eight-core machine,
"never use more than eight goroutines.
"Is that the fastest?"
The author wanted to know what was the fastest way to go
and the conclusion of the post was that,
at least when it came to merge sort,
that only one goroutine was the fastest way to do sorting.
Now, I love the messaging, because the message is
just because you throw more goroutines at a problem
doesn't necessarily mean it's going to run faster,
especially if you're not leveraging mechanical sympathy.
Some of the things that we talked about before.
But, it hurt my brain, because I could not fathom how
one goroutine could be faster than eight goroutines
on my machine.
It's just, my brain said there's no way.
Some level of parallel processing for what is really going
to be CPU-bound work had to be faster.
I said, you know what, I don't get it.
So I went ahead and I wrote merge sort.
Now, I'm not gonna go through the merge sort
code here, that's non-import.
What's important is I have an implementation
of merge sort, just like the author's implementation,
and what I did is I went over and,
we're gonna start with that million integers.
I wrote three benchmarks.
Single, unlimited, and NumCPU.
One goroutine, unlimited number of goroutines,
a goroutine for every split, and then just a number
of goroutines for the number of cores that I have.
I reconstructed everything that the author did
and I'm gonna go ahead and run this benchmark
just like the author did in the article.
I am in the right place.
Go test.
We're gonna run no functions,
run all the benchmarks,
increase the bench time to three seconds,
and we're not looking at allocations
because that's going to be nasty.
We're looking at CPU performance here.
There I am.
We got a single goroutine sorting a million integers
and every three is that millisecond.
This is 82 milliseconds it took.
We were only able to do this 50 times.
82 milliseconds on a single goroutine
to sort those million integers.
This was 635 milliseconds to do the same sort
with an unlimited number of goroutines.
Look, when I just kept throwing goroutines
at the problem, it ran significantly slower.
Then, when I gave it eight goroutines,
dash eight, we've got eight cores,
when I gave it eight goroutines,
it actually took longer than just using one.
This was the conclusion that the author
had, as well, in the post.
I guess we should just walk away.
My gut is wrong.
I guess, truly, when it comes to merge sort,
that those allocations are causing so much pain
that running it, even in parallel is worse
than running single goroutine.
OK, we're done.
No, no, no, no, no.
We're not done.
I still don't believe it.
This is what I'm talking about with validate.
You must validate your results, and this is killing me.
This has killed me for like two hours at this point
and I won't let it go.
I've done enough multi-threaded software development,
I've written the algorithms, I know I've got
the mechanical sympathies, to the extent I can, in place.
This is impossible for me.
I'm not willing to let this go.
Then, I do the following.
Let's just isolate running NumCPU by itself.
Look at the result we get.
Suddenly, we went from 95 milliseconds to 55 milliseconds.
Look, NumCPU is now faster, from my perspective,
30 milliseconds, significantly faster
than a single goroutine,
which is exactly what I expected to see.
What happened here?
Why is NumCPU suddenly faster?
Because, we're gonna go back to the idea that your machine
must be idle when you're running these benchmarks.
Think about it.
We ran single on my machine while it was
as idle as it could be.
Then, we ran unlimited on my machine when it was
as idle as it could be.
But when we started running NumCPU in this batch,
the run time in the machine was still busy cleaning up
the mess we made with the unlimited test.
In other words, we had created so many goroutines
that hadn't died yet, the scheduler and the run time
was still dealing with all of that
when we started the NumCPU profile.
Suddenly, now, the cost of the clean up is affecting
our performance numbers with NumCPU.
When I bring our machine back to an idle state
and I just run NumCPU, suddenly we get
a much more accurate result.
I like the idea that throwing goroutines at a problem
doesn't mean it makes it necessarily run any faster.
But, when we can run things in parallel efficiently
with mechanical sympathies, it really should
be faster than just using a single goroutine on one core
and now we've really proven that.
My point here is that we must always validate our results.
We should have some kind of understanding or be able
to predict in some way what we expect to see,
and when we don't see it, we've gotta make sure
that we understand it and that it's not us doing
something wrong, but it's really accurate.