An alternative to NPS for dev tools

This article was originally published on Medium.

When I worked at Snyk, we started experimenting with NPS (Net Promoter Score) shortly after forming a customer success team. We talked with our users a lot, so felt like we had a pretty good grasp on what they thought of our product, but using a sentiment analysis tool like NPS meant that we could quantify how perceptions were changing over time.

Many companies use NPS, and there are some good reasons to use it:

However, as a dev tool startup, we found it had the following drawbacks:

Because of these drawbacks, we tried a another approach that still had benefits of NPS, but also addressed the drawbacks. The method we adopted is a combination of feedback scoring and gap analysis.

We compiled a list of statements and sent them to users, asking them to rank their level of agreement on a scale of 1–5. The linear scale for the responses is 5 because 10 feels far too broad. 5 gives a good range of responses from strong-medium approval/disapproval, to being agnostic (3).

The responses to these statements were then averaged and plotted on a radial graph. This helped us clearly see the areas that users believed we were performing or underperforming in. I’ve mocked up an exaggerated example so you can see it in action. Each survey statement corresponds to a line on the chart, and the average score for the responses to that statement is plotted on each line.

This gave us a good indication of where users felt we were performing well, and where they felt we were underperforming.

In my example, for the statement “the product is easy to ingrain into my existing workflows”, users have given an average score of 3.75. But for the statement “the product is intuitive and easy to use”, they’ve given a score of 1.75. Looking at this, I know I need to do some work to improve the product’s usability, so I’d be pulling in my design team to do some more user research and prioritise improvements in this area.

You’ll want to set a goal for the next timeframe, such as the next quarter, that feels like enough of a stretch but still achievable. Jumping from an average of under 2 to over 3 would be a good goal for a quarter, so the goals have been set to 3.5. You might find areas where you’re getting such good scores that you can deprioritise work in this area, giving you more time to focus on the areas that need attention.

Ensure teams are coming up with their own ideas for how to improve the scores, and aren’t relying exclusively on the data — the data is a good foundation, but they should be reaching out to users for further validation.

Users will have different expectations and responses based on the plan they’re on, so be sure to capture this information. You can then group the average scores by plan, and if you ask the same question over a set of time periods, you can compare those results with previous ones to see whether you’re improving in that area.

At Snyk, we not only segmented by plans, but also had a segment for our design partners, who would respond to the full survey every quarter. Another segment we captured was new vs mature users so we could see if changes we were making were improving the experience for people who were just onboarding and didn’t have knowledge of what the experience was like before.

If you’re ready to make your own, take time to craft your statements as you’ll need to ask them in exactly the same way each time to enable accurate comparison of responses over time.

Have as many different statements as you want to get feedback on, but don’t ask individuals too many at once or they will give up. If I’m sending a survey, I aim to keep the number of statements as low as possible to increase the chance that someone will complete the form — ideally no more than 6. You could ask users a single randomised question via an intercom prompt.

Always give some space for the responder to add more detail if they want to, such as some free-text at the end. You can get some good insight this way that may lead to additional areas to ask for feedback on in your next survey. Make sure the Customer Success team has access to a feed of the responses so they can reach out to the user to address any concerns.

Here are some prompts to help you come up with your own feedback statements:

If you’re looking for something to get started with right away, you can use this example Google Form. I’ve done it as a Google Form because it’s free, has a nice UI, and feeds directly into Google Sheets which I can use to query the responses, but you can use whatever survey tool you already use. The responses feed directly into this Google Sheet and there’s a tab on there that calculates the averages of all the scores for each plan, and generates a couple of radar charts.

Use a range of tools to gather feedback if users interact with your product outside your app. If you exclusively use something like Intercom to pop up the feedback prompts, you won’t be reaching the users who rarely log in and use the integrations and may have a very different experience. You could explore some innovative capture methods like the CLI or GitHub comments to meet the users where they are.

I hope this method gives you some useful insights into how your users perceive your product. For those who are more visually-oriented, I’ve summarised this post in a slide deck you can share with your team.

You can download the Feedback Scoring and Gap Analysis slides here.