How to track metrics that improve user experience (and increase revenue)

Written by Christophe Limpalair on 08/29/2015

Want to increase revenue and user satisfaction?

Monitor more than just system performance.

Monitoring isn’t just about measuring the performance of your servers, it’s also about giving your users the best possible experience which will in turn increase revenue.

Here are some interesting examples:

At Kickstarter, they care about users being able to pledge money and back projects, as James Turnbull told me in his interview.

At Netflix, they make sure users can watch movies. If there’s an increase in errors when users press play, something’s not right. Same thing if the number of ‘movie starts’ drops.

At ScaleYourCode, I care about my audience being able to watch, listen, or read interviews.

Remember, the reason you’re monitoring server performance in the first place is to make sure users can always access your app — so they can go on to consume your content, use your service, or buy your product.

Those are your business metrics. They are the reason you even care if your servers go down. They are what brings in the money that keeps your servers running. It’s equally as important to make sure your users can find/use what you offer as quickly and as efficiently as possible.

In his interview about Netflix, Jeremy Edberg says they can always kill an instance and bring another one back up if system metrics (CPU, memory, etc…) are acting up. What they can’t easily fix is user frustration when they are paying a monthly fee and unable to watch their favorite show.

Step #1: Determine your business metrics

What does your application do, and what business value do you provide?


But Chris, I’m already making sure people can access my videos, products, or services, and it always loads very quickly for me and the people I know.

That doesn't mean everyone else has the same experience. I thought the same thing but I was wrong.

A few months back, I thought everyone could stream my interviews with no problems. I tested it myself, I asked friends in various locations, and they all reported that the streaming was great.

But the more viewers showed up, the more complaints started coming in. It was only 1–2 complaints at first, but this number grew with every episode. It was always different people, too. 

To figure out what was going on, I tracked a few things:

  • How many times did people run into video loading errors?

  • How many times did the video stop to buffer per user (and per video)


The numbers were way higher than I wanted them to be. Most people had no problems, but I didn’t want any users to have problems because of my setup.

The excuse of ‘it must be your internet’ was no longer viable. 

How many returning viewers did I lose? I’ll never know the real number, but I’m sure there were some. Catching this was all thanks to the few people who took the time to let me know.

I would have caught this earlier with proper monitoring. Not CPU usage monitoring, but business metrics monitoring.

What are you unaware of that could be costing you revenue?

It’s up to you to find out but here are a few to get you thinking:

Step #2: Act on your metrics

Track errors you don’t know about


This is a recent improvement for me. It is a big improvement.

Have you ever gone to a site and run into an error? You refresh the page a few times, but it’s still not working. What do you do? Go back to Google and click on the next result.

Errors pop up all the time. There are ways of finding many of them before they hit production, but some end up slipping through from time to time.

Instead of waiting for someone to complain (like I did with the streaming example), instrument your code so that you receive alerts any time a user runs into an error. 

This can be especially useful with client-side JavaScript errors. But there are tools out there that work with any language, and integrate with pretty much any reporting tool. (PagerDuty, Slack, Logstash, email for example)

Some tools are free for smaller sites. There is literally no excuse to not be using error tracking. Read more about that here, like how to set it up in a few minutes.

Tools

Things to look for


  1. Catch and log all application exceptions. This is easy thanks to frameworks.

  2. Try/Catch JavaScript code and log in the Catch

  3. Go in your reporting dashboard and look through stack traces. Who caused the error and what made it happen?

  4. Keep a close eye on this after deployments


I bet you’ll find some interesting errors after a few weeks.


Monitoring Transactions


If you run a really busy site, there are a number of transactions you could keep an eye on to detect early signs of issues.

Logins/Registrations
If you never get below x amount of logins/registrations during certain periods of time, having the number dip below your minimum with a margin of error could be a big indicator that something is wrong.

Or maybe login count isn’t going down yet but it is taking far longer to process authentications. Trigger an alert and get an engineer on it ASAP.

Purchases
Similar to logins, you expect a certain amount of transactions but it dips way below or takes longer than usual to process them.

Page load times
This one's probably the easiest of all because it's a standard with most tools/services.


Tools


Things to look for


  1. Drop in number of transactions

  2. Transaction time


Track user experience


This one is pretty fun ;)

How can you track what your users are doing without slowing down the experience with heavy JS and yet have as much accuracy as possible? There are a few ways:

Tools

Things to look for


  1. How many steps users had to go through before finding what they were looking for (or what you wanted them to find). Can you reduce this number?

  2. How long it takes for information to load. For example, my viewers shouldn’t have to wait for the YouTube video to load on page if all they want is to read the interview.

  3. How many users click on what you want them to click vs. how many users landed on the page. Add an A/B test with different sizes, positions, or colors, and see how results change.

  4. How janky is your front-end? Your server could be blazing fast, but how about the scrolling and navigation performance?

Conclusion


By listing various tools, I hope I’ve helped you take the focus away from finding the right tools, and instead on what matters most: giving your users an awesome experience.

If you haven’t already, take a moment to think about your metrics and how you could do a better job of monitoring them.

The larger and more complex your app, the harder this is to accomplish. But focusing on your business metrics can drastically help. Good luck, have fun.