TraceView’s Oboe Symfony No. 1

by James Meickle on June 11, 2013

Here at AppNeta we’re still recovering from an immensely productive DrupalCon Portland 2013. Our partnership with Acquia and our continuing support for the contrib TraceView Drupal module made sponsoring the event a natural decision. We’re glad we made the long flight out West, as there was a lot of interest in TraceView. In fact, on the plane ride back to Boston I brought back not only three Voodoo Dozens but a stack of business cards two inches high. (I’ll leave it to you to guess which one ran out first!)

But for as much fun as I had exploring Portland’s unique culture, my favorite part of the con took place well before it. I spent a lot of time coding up improvements for TraceView’s Drupal support based on customer feedback. My personal goal was to submit enough patches to the TraceView Drupal module to warrant a version bump before DrupalCon, and thanks to the hard work of module maintainer Peter Drake, we were able to get there with room to spare. If you’re running Drupal 7, check out the changelog, and then go install it on your site!

TraceView Drupal Module

The TraceView Drupal module now supports full-stack application tracing using drupal_http_request, including via the Apache Solr module.

I was able to code up more than just improvements in the months leading up to DrupalCon, though. If you stopped by our booth, you’ll have heard all about this, but we have a second important release to announce: we have a beta Symfony bundle and a (very!) preliminary port of our module to Drupal 8! While I had initially planned to just work on Drupal 6 and 7 improvements for TraceView, the announcement that Drupal 8′s SCOTCH and WSCII initiatives had resulted in the incorporation of Symfony components into Drupal 8 prompted me to take another look at what I’d lumped into ‘yet another PHP MVC framework’. It quickly became clear that I’d horribly misjudged Symfony. Drupal 8 hasn’t quite hit code freeze yet, but the adoption of Symfony has already let it pull in some beautiful pieces of architecture.

My favorite architectural decision in Symfony is its robust sub-request framework because of the degree of support that it lends to one of my favorite technologies, Edge Side Includes (ESI). Many large-scale sites leverage CDNs to deliver cached pages and static assets, but they don’t cope with logged-in users well because the content served to them is so heavily customized. ESI allows you to offload mostly cacheable pages to a CDN and only request the uncacheable portions of the page from the origin server, but it’s currently underutilized because most web frameworks can’t render just a portion of a page. In Symfony, though, any component of a page (such as a block or a sidebar) can be built as a sub-request for little additional developer effort. A year from now, large-scale Drupal 8 sites may entirely bypass common performance pain points by taking an ‘ESI Everywhere’ approach and leaving the CDN servers to do the work of assembling a dynamic page from a static cache entry.

Symfony had caught my interest, and my first instinct towards interesting software is to hack on it. That meant I had to create a Symfony site, so naturally, I chose Drupal 8 as a starting point. The codebase is still in flux, but it’s stable enough to install and it already makes use of several core Symfony components. All that I had to do to get an environment running was spin up an Amazon EC2 server, set up a LAMP stack and install TraceView (by far the fastest part of the process). The Drupal 8 installation required installing the trunk version of Drush, but after that I was only a `drush qd` away from a functioning Drupal 8 site.

I hit a wall fairly quickly, since like most Drupal developers I’d never tried writing Symfony codeor using Composer to manage packages. Instead of getting frustrated, I took a step back and began to read articles about both Symfony in its own right and how it’s being leveraged to rewrite Drupal. I made heavy use of the Drupal core issue queue and the Symfony API documentation, but I also turned to blogs written by Drupal and Symfony developers. Of particular note, chx wrote an introduction to Symfony in two articles as well as an article that helped me fix my first Drupal 8 performance issue:

TraceView Drupal MOdule 2

Every request to the Drupal 8 server triggered well over a hundred `file_get_contents` calls.

Before writing any integration code, I pulled up some Symfony traces because I was curious what they would look like. Even without code modification, TraceView picked up on Drupal 8 making a huge number of `file_get_contents` calls on each request. While individually fast, the calls often added up to more than 100ms; worse yet, that many disk reads has severe negative implications for scalability. At my last job, a sudden increase in traffic took down our entire distributed filesystem just by repeatedly fetching a single JavaScript file!

When I examined the `file_get_contents` calls in TraceView and realized they were all fetching module-related files, I immediately made the connection to Symfony’s dependency injection container (DIC) (one of its major design paradigms) thanks to another post by chx about how the DIC is stored on the filesystem. The filesystem calls were due to Drupal not finding the DIC on the filesystem and instead recompiling it using a dependency-checking mechanism similar to a package manager but made more complicated by the ability of classes to define additional compiler passes and other overrides. Like most problems I run into on Linux, I was able to trace it back to incorrect permissions:  because I had run Drush as a normal user, I had changed the permissions on the relevant directory and prevented the webserver from writing the compiled DIC so that it could be loaded on future requests. One `chown` later and my testbed site was running more than five times faster.

I kept an open mind to Symfony as I continued to read, but coming from the perspective of a Drupal developer, I was quite surprised to hear that the familiar hook-based event system has already been partially replaced with Symfony’s event listeners (bringing it more in line with some other PHP frameworks that we support). It’s new territory for me, but I can understand the rationale as there are some serious advantages to the switch. Unlike hooks, events don’t need to check every module on the site for an implementation; they can also have their propagation stopped if no further work is necessary. This means that developers will have much finer control over how modules respond to events, rather than having to use unusual hooks simply because they get called at the ‘right’ point in page loading (I’m looking at you, `hook_footer`!). Interestingly, a total switchover has been postponed until Drupal 9, which means that these overlapping systems will both remain in place for the foreseeable future.

The differences between Drupal hooks and Symfony event listeners were quite relevant to building out TraceView support, so I spent a lot of time reading about them. It’s there that I stumbled across another excellent addition to Drupal 8: the `terminate` kernel event! Thisimport from Symfony is the replacement for the recently removed `hook_exit`, with the key difference being that it can be run after a response is returned to a browser. This lets expensive cache setting operations be performed in a way that won’t harm end user performance, emails be fired off only when the request is complete, or disk writes take place after an upload confirmation screen has been shown to the user.

It took about a month to get up to speed on Drupal 8 and Symfony to a degree where I felt ready to stop reading and start writing. That left me with about a month to go before DrupalCon, and in that time I was able to turn out not only the Drupal 7 module improvements but a really awesome DrupalCon booth demo!

TraceView Drupal MOdule 3

The top tier of Drupalception included both our beta TraceView Drupal 8 module and our TraceView Symfony bundle.

I couldn’t think of any better way to show that we’re serious about our Drupal support than to demonstrate every version of Drupal we support at once, and so I built Drupalception: a Drupal 8 + Symfony AWS instance using cURL (via Guzzle) to call into our Drupal 7 demo site, which in turn uses drupal_http_request to call into our Drupal 6 demo site. TraceView catches the entire stack as a single trace even when blocking calls are nested fifteen layers deep!

But wait, aren’t I skipping a step – how did I get from reading a bunch of blog articles about Drupal 8 in Symfony to a functioning Drupal 8 module and Symfony bundle? Well, that’s a story in its own right – and for now, one that you’ll have to wait for another blog post for!

TwitterFacebookLinkedInRedditEmail

Slow Web Apps?

Web pages are complex. Download this free article to discover the four different ways you’re keeping your end users waiting.

Download the article