First Contact Resolve

Customer Support: Career, Community, and Culture

Menu Close

Author: fairyqueen

Measure What You Value: Building Lightweight, Effective Support QA

How closely and consistently do the interactions you have with customers align with how you think they should be? Training for and maintaining a high quality support experience, especially for high growth teams, means you should be measuring much more than volume and response times. I’ve already discussed on this blog ways to think about a more holistic set of support metrics. One of the things I mentioned, but didn’t dive into, was measuring the quality of your interactions. Support QA is something I’m really excited about.

How can support teams set clear expectations around what a high quality interaction is? How can the qualitative be quantitatively measured? How can you integrate your expectations around quality into the very fiber of your team culture? Support QA is a big topic and could easily fill a book. This will not be a book. In this post, I’m going to try to cover (briefly, but hopefully effectively) how to distil values into scorable metrics, figure out tracking, cadence, and feedback loops. I’ll also go into calibration and program roll-out with integration into hiring and training.


What is a High Quality Support Interaction?

 I was at a workshop for customer support leaders recently. One of the things we discussed was support QA. We broke up into groups and were asked to list all the things we valued in our support communications. Not specific communication action items – like following an approved greeting script or other company protocols – but the core values we wanted displayed in our communications. Then each group was asked to pick their top four values. Members of the teams represented many companies spanning many different markets. As much as we all like to believe that what we value is unique to our companies and teams, the top four values were remarkably consistent from group to group. What we converged on at the workshop is, I dearly hope because I place a lot of emphasis on these things on my team, likely what customers value, too:

  1. Correctness/Accuracy
  2. Completeness
  3. Empathy (top of the list for every team!)
  4. Professionalism and Tone

Welp. That’s great. But how does all this fluffiness get us to a QA score? Let’s start first by defining very clearly what each of these things means in the context of an interaction. For the four values above, I might start with something like this:

  1. Correctness/Accuracy: Did we correctly interpret the core question? Did we give a direct and correct answer to that question?
  2. Completeness: Did we cover all the bases? Did we answer all the questions asked? Knowing the questions an answer is likely to prompt, did we answer those potential follow-up questions, too?
  3. Empathy: Were we being kind humans talking to humans? Did we acknowledge the likely emotions around the question? Or connect with personal details brought up during the course of the conversation?
  4. Professionalism and Tone: If written, was everything spelled correctly? Formatted in a way that was clear and made sense? If spoken, were we confident and clear? Overall, are we signaling that we are intelligent, trustworthy professionals?

Now take this back to your team. If these are to become your core communication values, you best get some consensus.


How Does One Score Empathy?

You’ve reached quorum on the values and what they mean for your team. It’s time to figure out how to, as objectively as possible, see if those values are present in your communications. Let’s hammer out a scorecard! I like to keep forms like this super dead simple. Here is a sample email QA form I have successfully used to show just how dead simple you can make this:

qa form sample

As long as you are clear on what your values mean, your form does not need to be fancy and complicated. Don’t make it fancy and complicated if you can avoid it. If your intention is to closely monitor quality, you want your form to be quick. And easy. And allow you to do samples of multiple tickets per agent without the person doing the scoring wanting to die. And you want all this and to still get actionable information out of it.

The form above scores a sample of 10 email tickets for a single agent. When checking to see if a ticket is “Correct”, run through the questions we set next to “Correctness” in the last section. If we can answer all of them with a yes – that box gets a 1. Otherwise a 0, and a note to the right detailing what was off. No half-points or scales to muddy things. Simple binary. Add the columns to understand where this person is performing beautifully and where they might need more focus. Get an average ticket score for the overall. 4 is amazing! 0 is very not amazing!

We have a form! But wait, there’s more!


Testing and Calibration

Now that you have a form, how do you know that it’s good and it works? Before this form becomes canon, it must be tested and calibrated. You want to be sure that what you’ve put together adequately captures what’s going on in what you are scoring, that your scoring is consistent, and that the results are useful for you and your team. There are a number of ways to do this. Things I have done in the order that I prefer them:

  1. Test the form on your team applicants’ writing samples: I love this approach. I require writing samples from anyone interested in joining our team. When I’ve tested and rolled out QA during a high volume hiring cycle, giving team members the form as a framework to evaluate writing samples was a great way to test. You can also do multiple scorecards per candidate to see how consistent your scoring ends up being.
  2. Test the form with peer review: This can be a great way to get feedback on gaps in the form, broader usability, and give people a structured way to practice constructive feedback with each other. This can be a high bandwidth approach, depending on the channel you are scoring, and people will likely be scoring only one or two people. With so many people scoring and little scorecard overlap, checking for score consistency can be hard.
  3. Choose a core team: Identify in advance who your quality scoring folks are and have them run a sample. Make sure your team is aware and be very clear that this is just for testing. Do a few runs and make sure that your core team is making similar scoring decisions for as little variation in scores as possible.

QA Program Roll-Out and Team Integration

You’ve defined values. You’ve figured out how to score them. You’ve tested the form and found it good. Now you need to turn this whole regular quality testing thing into a thing. Hopefully, with lots of team enthusiasm and buy-in. How? I’ve found an incremental approach useful:

  1. Start with new hires: Develop training assets that detail your support communication values and involve existing team members in communication best practice training. QA becomes a valuable resource to the new hire to figure out, with their manager, where to focus effort.
  2. Include more tenured team members: Once QA is an established part of training and onboarding, broader team roll-out becomes a more natural extension of your team’s quality program.

Cadence and Feedback Loops

How often should you score quality? The real question is how often is it useful? When beginning a quality program, it can be helpful (once you have an idea of your baseline) to set performance goals, score with more frequency, and use the information gained to coach your team to those goals before tapering down. For reference, I generally start with once a week until we’re hitting the mark, then taper off to once a month or so. Unless we’ve just launched something big and new – then back to once a week. As simple as I hope you have made your form, QA still takes time and resources.


Taking the time to put together a thoughtful support QA program is worth the effort. It forces you and your team to think deliberately about, and reach some consensus, and what you value and how those values manifest. It keeps those values top of mind and makes them real.

Where Do I Even Begin with Customer Support Metrics?

Every company is different. No approach to customer support metrics will fit every product or business model. Of course. Duh. But in some ways, support metrics – and good ways to start poking at them until you’ve refined an approach that fits you more exactly – can follow some general guidelines. This approach I’m about to lay out borrows heavily from Chief Customer Officer, a book I love and highly recommend.

When I talk to companies about customer support metrics, especially if they are setting up support as a new function, they tend to focus on ways to measure how quickly and efficiently the team is answering questions. Which is a great thing to think about, but is a very narrow slice of what a good support team should be focusing on. When you are thinking about what to measure and track, you should make sure you are looking at things under each of these three areas:

1. How many customers are we gaining and losing as a result of how we do things?

When I say “how we do things”, I’m not just talking about how support is handling stuff. How many people are you gaining, maintaining, and losing because of how the product works? Features you do or do not have? Your policies and request lead times? Pricing? Quality of support interactions? Other stuff? You will likely have to dig for some of this information. But it’s worth it.

Good things to start with:

  • New customers – volume and value
  • Lost customers – volume and value
  • Reasons why customers left (as specifically as humanly possible. DIG!!!)
  • Reasons why customers chose to renew (if you have a pricing model where this makes sense)

11. How good are we at helping customers who ask for help and rescuing customers who need to be rescued?

 This is where you get into the quick and efficient question answering metrics! More than just how many emails you are responding to and calls you are answering, there are several other things you want to know. What can you do to get better at what you are doing? Are you getting better over time? And are the interactions you’re having with customers good interactions?

Good things to start with:

    • Volume
      • What do we expect to be able to complete per agent per hour?
      • What do we actually complete per agent per hour?
      • What is the reason for the delta (if there is one)?
      • How are workflow/process/tooling changes increasing or decreasing what we are able to get done?
    • Efficiency
      • Median first response and full resolve times
      • If you have set goals or expectations for first response and full resolve times, what percentage of things are getting done within those timeframes?
      • What kinds of things take the longest to resolve? (Why?)
      • How many interactions does it generally take to fully resolve things?
      • What kinds of things take the most interactions to resolve? (Why?)
    • Quality
      • How do customers rate the quality of their support interaction? (CSAT surveys are a great way to start figuring this out)
      • How closely and consistently do the interactions we have with customers align with how we think they should be?

111. How well do we listen to customers and make things better?

All this digging and hard work and number stuff is great. But how are you doing at acting on all this information? Are the actions you’re taking making any difference at all? Has anything you’ve done made things easier for customers? How do you know?

Good things to start with:

  • ALL ABOUT ALL THOSE INTERACTIONS
    • Clear organization of all that awesome interaction information
    • Volume and trends
    • Close tracking of volume and trend changes in response to actions taken to (hopefully) resolve root issues
      • You’ve identified an issue with all this customer data and tried to fix something. Fixes can (and should) run the gamut of copy changes, FAQ improvements, new features, improved support messaging, etc… Make sure it was the right fix by checking to see that ticket volume related to the issue has dropped, or the CSAT surveys for tickets associated with that issue come back happier, or funnel conversion has become more efficient…
Beyond talking to customers and generating reports your company would also likely benefit from involving your customer support leader in the product planning process, if you don’t already. This will help your team intelligently plan for feature releases. In addition, support can give deep insight into what customers are actually asking and how what you are planning is likely to perform in the wild.

IMPORTANT NOTE: As you dive into tracking, don’t be sad that you don’t have all of this information right away. Setting up how you want to collect and report takes time. Collecting enough data for the numbers to actually mean something takes time. Resist the urge to use your first numbers as benchmarks and start drawing firm conclusions. Take your time in the beginning to just notice the numbers until they start manifesting trends.

To sum up, your customer support metrics should be about more than just response time. Develop a clear set of data that addresses more of the things your customer-driven company should be measuring: customer retention, support bandwidth and quality, and how successful you are at improving the overall customer experience.

No One is Bleeding, We’ll All be Okay

UserConf (now Elevate) 2016 is coming up soon and reminding me of the wonderful time I had presenting at their Portland conference last year. The conference is fantastic and I highly recommend it. I talked about my transition from birthwork to working at tech companies and how they ended up not being as different as you may think. Video below! And transcript below that!

“I went into the interview for what would be my first technology startup with the vaguest idea of what a technology startup was. I had spent the last few years as a doula – it was my job to help women while they gave birth. Whitetruffle was very early, bootstrapped, and risky. The CEO asked if I was okay in high stress situations. I told him that as long as no one was bleeding and everyone was breathing, that I’d be okay. And I was. Since Whitetruffle, I’ve worked in customer-focused positions at Mighty Spring, GitHub, and now Earnest: a financial startup lowering the high costs and barriers to credit faced by millions of financially responsible people.

“While the experience of working at a startup is nothing like assisting a woman in labor, I find myself drawing on the lessons I learned from birthwork every day: First, that everything will be okay; things are rarely as urgent as we think. Second, as deeply as I care about everything I do, I need to keep firmly in mind that this is not my baby and not my birth. Third, that it’s important to fully understand and plan for what progress is really like.

While everything is important, not everything is urgent.

“Everything will be okay. Things are rarely as urgent as we think. I remember my first births as a doula. I wanted to do everything just right. A mother would call me, excited and nervous and absolutely sure that she was in labor and she wanted me there and I wanted to be there and I would drop everything and rush to her side. As soon as I arrived we’d sit and chat. I’d quietly time any contractions I noticed and they would get further and further apart as we both felt calmer. After those first few births, I learned how to more accurately assess real versus perceived urgency. Instead of rushing to the rescue, I learned to have those chats over the phone. Things felt calmer and contractions would slow. We’d both agree it was best to stay home and get as much sleep as we could. Labor was coming soon – but it wasn’t here quite yet.

“Similarly, understand in advance what likely is urgent for your customers and your product. Keep a close eye on as much of the customer experience as humanly possible. Have a plan of action to respond quickly and appropriately when urgent issues do arise – but admit to yourself that, while everything is important, not everything is urgent.

“Know when to question the rules. Once a woman in labor is admitted to the hospital, this is often the only thing she will be given to eat until after she gives birth: ice chips. Why is that? Not so long ago women were routinely knocked out for birth, and of course before any planned surgery or procedure in which a person is given general anesthesia, it is accepted practice to prevent the patient from eating or drinking because there is a small chance of aspirating vomit while unconscious. Of course, it is also standard practice to put a breathing tube down a person’s throat when they’re put under to reduce that risk of aspiration even more. As a result, the actual likelihood of anything terrible happening is incredibly low, especially with local anesthesia and epidurals being much more common than general anesthesia during birth. So, the risks of eating and drinking during labor are pretty close to zero. And birth, calorically, is just like running a marathon. Knowing this, the mother and I would find ways to sneak her calories. Clear, sugary liquids to pour into her ice chips. Grapes, honey, and other high calorie, easily digestible foods to give a much needed extra energy boost before pushing.

What rules would your team most like to break for customers?

“What are you doing that maybe doesn’t make sense? Comb through your support team’s current rules and policies. Are any of them unclear or unspecific? Outdated? Do they jeopardize relationships? Ask your team for insight into which rules they would most like to break for customers. See if it makes sense to change them.

“Labor is painful – and so is some aspect of your product. The biggest difference is that everyone expects labor to be painful. Give your customers strategies to work through that pain. Earnest is a financial technology company that helps people who behave responsibly with money, but may have thin credit profiles. Personal finance is an interesting space to be providing support in. It’s something that people simultaneously know to be incredibly important – and harbor a lot of confusion and anxiety around. Knowing that, we’re working incredibly hard to reduce that anxiety for people. Our loan terms and disclosures are written at a 10th grade reading level to make them as easy as possible to understand. When someone is declined for a loan, we provide them with reasons why, make ourselves available for followup questions, and will always reconsider an application upon request. We emphasize communication and clarity in every part of the our user experience.

“Care deeply about your company and product, but know when to let go. Pregnant moms and I would work together on a birth plan well in advance of labor to help both of us better understand her vision of how she’d like her birth to be. While birth plans are never set in stone, mothers generally hired me to help them follow through on those plans and balancing that against what might be needed when labor was actually there could sometimes be challenging. A woman I worked with who was set on not using any painkillers during birth ended up in very painful labor that lasted for days. She was exhausted and having a hard time coping. I let her know that whatever choices she made for her birth, they were hers. She had proved herself strong, and capable, and no one would think any less of her whatever she chose. She ended up taking a mild painkiller and was able to get a few hours of sleep. When she woke up her energy was back, her confidence had returned, and her baby finally came. Not considering a deviation from the birth plan would have been terrible, so we let go.

Especially if your team is growing rapidly, the regularity of your weekly team syncs, bimonthly reporting, quarterly goal planning, and whatever other rituals you have established all provide reliable structure, a built in outlets for needed communication, and keep momentum flowing.

“I found myself in what felt like a similar situation at Whitetruffle. Whitetruffle is an employment matching service and used LinkedIn’s API to make it easier for people to create user profiles. LinkedIn decided that they no longer wanted us to use their API. In addition to that, they asked that we get rid of any data we had gathered using it and gave us a deadline for deletion. We quickly created a way for people to easily port information if they wanted to keep their profile active. Over 75% of our user profiles had been created using the LinkedIn API. Everything was going smoothly, and then Hurricane Sandy happened. We had a large userbase on the East Coast. As important as it was to us that we preserve those user profiles, it was nowhere near as important to the many people displaced by floods. Continuing to push people to port their employment data would have been terrible, so we let go.

“If a woman is free to move during labor she will usually find a spontaneous ritual, usually adapted from something we went over in our prenatal appointments, or another childbirth preparation class, which she will repeat precisely the same way contraction after contraction. If a partner is a part of the ritual, that person must also consistently repeat their part and over with her. The ritual changes from time to time in labor. If a woman feels overwhelmed and unable to carry on a ritual, it was my job to help her find it. Breathing, rocking, pacing, moaning, or – in this case – deep hip compressions through the apex of each contraction.

“As you build your support team, establish a regular rhythm of reporting and check ins that team members can rely on. Don’t skip your one on one’s if at all possible. Especially if your team is growing rapidly, the regularity of your weekly team syncs, bimonthly reporting, quarterly goal planning, and whatever other rituals you have established all provide reliable structure, a built in outlets for needed communication, and keep momentum flowing.

“Anticipate the ugly. Progress is not always pretty. I always brought grapefruit oil to a birth. As the baby’s head moves down, it was not uncommon for a little bit of poo to escape. I was always ready for a quick, discreet cleanup and would spritz a little of that oil around to brighten both the room and the mood in late labor.

Progress is messy, bloody, loud, painful, and sometimes ugly. And you want to be – because that means you’re getting somewhere.

“I’m sure we can all think of times we’ve had to clean up a little bit of poo. An unfortunate bug, a delayed feature ship, the stresses of keeping up with exponential demand. Growth can be painful – but the pain of growth is good for you. You’re moving in directions you haven’t moved before, you’re learning, and you’re figuring out the ways all the pieces work best together. The important thing is to anticipate that things will not always work smoothly, learn from it, and have your grapefruit oil ready the next time.

“This is me. Minutes after giving birth to my son in my living room. For most people, creating a quality customer support experience may not have obvious parallels to assisting women in labor – but birthwork is actually the most intense, high feedback, full contact customer support role you will ever find. Birth teaches you to identify and prioritize what is actually important – do that, and everything will be okay. This is not your baby, not your birth. You are building the experience for someone other than yourself. Progress is messy, bloody, loud, painful, and sometimes ugly. And you want to be – because that means you’re getting somewhere.

“I want to leave you with what I told my mothers preparing for birth:

“This work is going to be hard. But it’s also going to be worth it.”

© 2017 First Contact Resolve. All rights reserved.

Theme by Anders Norén.