Best practices for your quantitative survey

By Owen Hughes

1 November 2021 Experience design

‎

Gain the best possible insights from your user research with this guide to quantitative survey success.

What is a quantitative survey?

A quantitative survey is a form of quantitative user research.

While qualitative user research often gives you the why behind people's actions, you don’t get the data. Quantitative user research gives you data from a much larger audience. It allows you to take an informed and data-driven approach to your design decision making.

Surveys also give you the statistical significance of the results (something we’ll get into later) which allows you to have more confidence in the results,

A survey typically takes the form of a set of questions. For example, you might ask participants to use a sliding scale to rank how they feel about something. Or to select an answer from a predetermined set. In some cases, a survey will include open-ended questions too, which can be really insightful.

Quantitative survey examples and use cases

Quantitative surveys can be used to measure a range of different things, but the most common use cases are to:

Gauge customer satisfaction or an opinion
Assess opportunities for improvement or redesign
Acquire data to support your decision making and other research sources
Gauge demand for new digital product features

The advantages of quantitative user research

Surveys have several advantages over other methods of user research.

They are:

Relatively inexpensive
Straightforward to set up
Fast (you can set up a survey, QA it, and get all your responses within two weeks)
Easy to distribute across a broad range of channels (for example email, social, websites etc.)
Suitable for a broad range of use cases

How to determine your target audience

Just like qualitative research, quantitative user research surveys need to have a clearly defined audience.

It’s important to align on this with key stakeholders at the start of your project. The last thing you want is to skew your results by inviting participants who aren’t relevant to what you’re trying to learn. As a broad example, you wouldn’t want feedback on a new product feature from people who don’t use the product.

Some examples of your target audiences could be:

Existing users
New, potential, or prospective users
Specific user segments (for example, inviting highly engaged or loyal users to comment on a particular feature)

How to recruit participants for your user research survey

Once you’ve decided your research objectives – and the audience that you want to target – it’s time to think about recruitment.

Firstly you need a screener to make sure you get the right people to answer your survey. A screener is a set of questions carefully designed and worded to ensure that each participant meets the criteria of your target audience – without asking potential participants to self qualify whether they fit the criteria.

Calculate your quantitative questionnaire sample

You will need to determine how big your sample size needs to be and a sample size calculator can do the work for you.

Sample size calculators are based on an equation that considers:

Population size: the total number of people in a group whose opinions you want to gauge
Confidence level: this shows how confident you can be that the results represent the attitude of all your users. The industry standard is 95%
Margin of error (%): the percentage that tells you how much of your results reflect the views of the overall population. The smaller the margin or error the better, as this means you’re closer to having the exact answer at the given confidence level

Your sample range should be a minimum of 30-100. Going for 300 is ideal, but your user population size will determine whether that’s possible.

You can learn more about sample size calculators here.

Tip: remember the times you declined to take part in a survey, because you didn’t have time, or just didn’t want to? This is something to consider when looking at sample sizes. You’ll inevitably get participants who don’t respond. A good rule of thumb is to plan for a loss estimate of 10% (participants won’t complete the survey). You can minimise the loss estimate by offering an incentive relative to the survey effort i.e. the longer the survey takes to fill in, the larger the incentive.

How to make your user research survey inclusive

Demographics questions ask participants to share information around things like sex, gender identity, ethnicity, age, location, education, and marital status.

These questions provide good insight into your sample, but they’re potentially a very personal thing to ask participants.

If you need to know this information, ask it in a blank (open) field and allow participants (if they feel comfortable doing so) to enter their own option.

If you’re asking which gender a participant identifies as, allow them to enter their answer, rather than choose from a predefined list. It's a personal topic for some. So be clear, give context, and tell participants why you want this information (and how you'll use it).

Whichever survey tool you decide to use (we’ll explore some examples in a moment), make sure that tool is accessible. Your survey should meet AA web content accessibility guidelines (WCAG) standards or higher. This is to ensure that you’re meeting the diverse accessibility needs of your participants (bear in mind that 1 in 5 people in the UK have a disability, for example).

Write with accessibility in mind. Make sure your questions are clear and concise. If you need to use images, make sure that questions explain or give context to the images. And ensure your colour contrasts are accessible too.

9 tips for writing quantitative survey questions

The quality of your survey questions impacts the quality of your research insights.

Follow these steps to improve the outputs from your quantitative user research:

Inspire participation. Consider how you can encourage people to take your survey. Consider the wording of your email subject line. Does it entice and encourage people to open the email? How can you use social media to encourage participation?
Keep it simple. Keep your survey short to maintain attention span and stop drop offs. Only ask the questions you need to ask.
Keep questions short. Make questions as short as possible. Use short words where possible, and keep questions clear, concise, and unambiguous.
Avoid jargon. Use common terminology. Try and use the language of your audience, and provide explainers where needed.
Start simple. Start your survey with your most straightforward questions to encourage engagement. Leave trickier or more controversial questions to the end, and make them optional if possible.
One topic per question. Only address one topic at a time in a question. For example, don’t ask whether a participant finds a website ‘quick and easy to use’ in a single question. Because these are two distinct topics.
Avoid leading questions. For example, instead of asking: How concerned are you about the current fuel shortage situation? ask: How are you finding the current fuel shortage situation?
Important words first. Put important words at the start of questions. For example, if you want to know how many times someone went to buy fuel this week ask: In the past week, how many times have you tried to buy fuel?
Use neutral language. For example, rather than ask: Do you suffer from...? ask: Do you have…?

How to QA your survey

One of the main downsides of using surveys is that once it’s out there, there’s no way to edit it. That’s why quality assurance is key.

After you’ve written your survey, you need to thoroughly check it. This process will depend on the nature of your survey and your sample size, but it ideally involves these key stages:

Test your survey with colleagues to tease out any immediate issues
Do a trial run with a sample approximately similar to your target sample
Do a trial run with an actual sample size
Address any issues that come up and get ready to hit send!

Then, when you’re ready, send it off and stand by for your results!

How to analyse your survey findings

Most survey platforms include standard analytics for free. Based on the analytics you can draw findings and, with deeper analysis, you can then prioritise your next steps.

Look out for false entries when analysing your results. Some participants complete a survey in the fastest possible way to get to the reward or incentive. These false entries can affect your results if they’re left unchecked.

Tell tale signs of false entries are things like:

Short survey completion times. Look out for instances where the survey was completed much faster than the average time. If you see that people typically complete the survey in 2-3 mins, but one individual took only 30 seconds, that might warrant a closer look.
Look for patterns in the answers. If people select the first answer of every question, that might be a red flag.

Statistical significance

Statistical significance works out the probability of the results happening randomly. The probability needs to be no higher than 5%, which tends to be the cut off point.

This is indicated by the ‘p’ value in the survey analytics. If the p value for your results is less than 0.05, your results can be considered statistically significant.

Survey tools (like SurveyMonkey) provide the statistical significance of your results and highlight ones with strong statistical significance. This allows you to be confident that the results are accurate rather than random.

User research survey tools

There are a ton of survey tools out there, with varying levels of complexity and cost.

Here are some popular ones:

Google forms. It’s free to use and gives very basic analysis for completed surveys.
Typeform. This paid tool is more sophisticated than Google Forms and provides more in-depth analysis, with lots of integrations.
SurveyMonkey. This is one of the biggest paid survey tools on the market, with detailed analysis and lots of resources for guidance.

The value of quantitative surveys

User research surveys provide insightful quantitative data – at an affordable price point. Used and analysed correctly, they’re a very effective way to better understand your audience and replace your assumptions with evidence, in the form of data.

It’s important to remember that surveys don’t often give you the ‘why’ behind the questions you’re looking to answer.

That’s why the best customer insights come from a powerful blend of qualitative and quantitative research.

If you enjoyed this guide, do check out our user testing FAQ and our post on user research recruitment.

If you need support with your user research or recruitment, or you’re looking to grow your team’s research capabilities, do drop us a line. We’ll connect you with one of our user research specialists.