Modern Users Need Modern Metrics
by Christine Cignoli on

Sometime in the past ten or so years, we’ve come to treat the Internet as a revered state of mind, rather than a repository of information. I say “we” knowing I’m not alone here. When I first started using the web, I remember happily tolerating near-daily Hotmail outages and being really delighted when a coworker (the IT guy, of course) sent me Peanut Butter Jelly Time, my first taste of how fun and distracting web videos are.

But my patience has eroded as the Internet, and now my phone, keeps getting faster and smarter and more useful to me. Now, whenever I like, I can order an Uber, find out what song is playing at the café or shop for discounted women’s wear. Who’s got time to wait for pages to load? Not me, that’s for sure. And this trend toward Internet impatience—er, higher expectations—has obviously become part of what we expect from applications used for work, not just play. Measuring how users experience the web is a newer concept in the tech world, and one that lots of businesses are trying to understand and implement. 

Some tried-and-true metrics still have an important place in a modern IT environment. Quality of Service (QoS) was a concept born in the early days of video and audio streaming, and the idea is that network performance could be measured and planned for various workloads. As applications started to use more resources and bandwidth, IT teams had to start looking at ways to improve network performance, which meant reducing data loss and jitter and adding bandwidth strategically. Then they could measure improvements using QoS, which also more broadly can refer to the requirements of hardware and software, and whether those requirements are met.

QoE Is So Right Now

These days, QoS serves as a necessary baseline for infrastructure performance. A newer, more elusive way to measure overall use and performance is with Quality of Experience (QoE) metrics and techniques. “Experience” is the word that’s more commonly thrown around these days when we’re talking about users and their apps. It’s more nuanced than QoS, and aims to capture what an end user expects and what that user actually sees and does when using a service or app.

With nuance comes subjectivity, of course. It’s easier to apply benchmarks and expectations to a piece of hardware, but much more complicated with actual humans. But it’s something app providers and businesses offering SaaS applications would do well to tackle to find out which metrics they should be using. As we’ve all become web consumers, our app expectations are high at work as well as at home. QoE metrics consider how a user moves through an application, and whether they become frustrated or stop using the app altogether.

Here at AppNeta, we use Apdex scoring in our TraceView product to get a view of what end users are experiencing at any given time, as well as Transaction Groups to monitor groups of endpoints. The industry-standard Apdex measurement takes into account several user metrics—one of which is the very human “frustration level”—to rate the overall experience.

Gartner suggests a few key metrics for measuring end-user experience, one of which is the super-technical “keyboard-to-eyeball response time.” Clearly, user experience itself is subjective, and so are the metrics that are established and still emerging to measure how good that experience is. The metrics aim not to lose this user experience information in the noise of more technical numbers like jitter and latency—or ideally to tie those user experiences to overall infrastructure statistics.

Lots of developers at AppNeta have considered the QoE challenges, and you can see here the 14 end-user experience metrics we recommend. How are you measuring QoE—or how would you like to, if you’re just getting started? Leave a comment and let us know.

Filed Under: Industry Insights

Tags: metrics , QoE , QoS , SaaS