X
    Categories Application Performance ManagementPerformance Management Tech

Basic Performance Comparison Between Koa and Express

AppNeta no longer blogs on DevOps topics like this one.

Feel free to enjoy it, and check out what we can do for monitoring end user experience of the apps you use to drive your business at www.appneta.com.

There’s been a lot of chatter lately in the JavaScript community about the latest, greatest ES6 features shipping in recent browsers and the V8 versions in node.js 0.12 and io.js 1.2.0.

One such hot topic is generators, a special function that produces an iterator. Several modules on npm, such as co, have taken to hacking this mechanism to produce sync-looking code that actually runs asynchronously by pausing the function with yield statements.

This experimental approach is at the core of a hot, new framework called koa, which takes an extremely minimalist approach to the middleware chaining functionality of connect or express.

app.use(function* (next) {
  this.requestBody = yield parse.json(this)
  yield next
})

It looks very clean, pausing the function to yield an expression and resuming when the iterator consumer is done doing whatever it needs to do with the expression result. But how does it perform?

When benchmarking a readFile call using a callback versus promises or generator yields, you will typically see results something like this:

1 concurrency
  • 19,644 op/s » generator with promises
  • 19,564 op/s » generator with thunks
  • 33,691 op/s » promise.then
  • 41,160 op/s » callbacks
10 concurrency
  • 21,469 op/s » generator with promises
  • 20,992 op/s » generator with thunks
  • 37,512 op/s » promise.then
  • 72,284 op/s » callbacks
100 concurrency
  • 21,442 op/s » generator with promises
  • 20,996 op/s » generator with thunks
  • 37,565 op/s » promise.then
  • 72,540 op/s » callbacks
1000 concurrency
  • 19,775 op/s » generator with promises
  • 19,203 op/s » generator with thunks
  • 32,857 op/s » promise.then
  • 60,864 op/s » callbacks
10000 concurrency
  • 16,217 op/s » generator with promises
  • 15,960 op/s » generator with thunks
  • 26,855 op/s » promise.then
  • 58,262 op/s » callbacks

Callbacks are indisputably faster in this benchmark. Generators are a brand new feature and not much work yet has gone into optimizing their performance in V8, but they perform better than you might expect.

Micro-benchmarks are not a very good representation of real-world performance, as they don’t do real-world tasks. In a real world node.js application you’ll likely be serving many concurrent requests with more complexity than a simple readFile call.

While a single callback is certainly faster than co running a generator that yields a single promise or thunk, the performance advantage of generators comes with operational complexity, which is also where they become a code clarity advantage.

Let’s try a more realistic benchmark, http requests to a koa server compared to requests to an express server. I’ve created a basic todo app with CRUD routes for storing tasks, and a basic list route that renders the list of tasks.

koa
  • 583 op/s » create
  • 1,037 op/s » read
  • 50 op/s » list
  • 923 op/s » update
  • 1,104 op/s » destroy
express
  • 573 op/s » create
  • 862 op/s » read
  • 25 op/s » list
  • 712 op/s » update
  • 834 op/s » destroy

As you can see, the more realistic benchmark varies wildly from the performance you’d expect, given the micro-benchmarking results. Koa has turned out to be noticeably faster.

The reason for this is reduced operational complexity for the VM. The VM simply has less stuff to juggle to make your app function. With each function you create, the VM has to do more work to keep track of things like variable availability in different scopes or what this is bound to. With all the JS in your dependency graph, that can result in a rather large number of parts interacting with each other and complicating the runtime behavior. VMs are complex and unpredictable, so it’s generally best to keep things as simple as possible to avoid further complicating the work for the VM.

Another trend emerges if we look at the distribution of requests. I captured some latency distributions while running both of these. In the chart above, you can see 10,000 requests to Express, then a break, then 10,000 requests to Koa. The different bands represent the different operations, but beyond the separation, Koa is noticeably more consistent. The reduced operational complexity shows up not just as better real-world performance in the average case, but also as much better performance in the slow case. Tracking 99th percentile or slowest requests captures this variation more starkly than just the average.

This just shows that the only sufficient benchmark to evaluate real-world performance is to measure on real-world code. The more data you have about the state of your app over time, the more clearly you can understand the runtime characteristics. While one method might be faster than another in a naive benchmark, it could also consume a lot more memory, or create a lot more work for the garbage collector, or hold more file descriptors open. A clear understanding of real-world performance requires constant monitoring and evaluation.

View Comments

  • Thanks a lot for this write-up! I was surprised to see generators performing so poorly, but you make a good point when showing the opposite results in the overall benchmark of the whole framework...Have you had the chance to try out newer versions, since co 4.0 has switched to promises? I'd be very interested to see the same benchmark with the new version!