As an industry we tend to use survey length as the cornerstone for how we price surveys, but often the estimated lengths and real lengths of surveys can turn out to be wildly different. Leading as I have experience to potential conflict.
The reason is we have not established in the research industry a common and reliable way of estimating the length of a survey. The most common method in circulation is to assume we answer surveys at 2.5 questions per minute but this technique is fatally flawed. This is because question themselves can vary wildly in length e.g. a survey with 10 grid question with say 50 options may take 50 times longer to answer than a survey of 10 simple yes no questions.
So I have been on a bit of quest to work out some slightly more accurate ways of do this. As a result of some recent work we have been doing to examine in detail how long respondents answer survey I have come up with 3 new alternative methods I would like to put forward to more accurately calculate the length of a survey.
This is the most accurate way of doing it (though I recognise it take a quite a bit of work). This formula will given you the length of an English language survey in minutes.
W = word count: Do a word count of the total length of questionnaire (questions, instructions and options). An easy way to do this is to cut and paste the survey into word but don't forget to remove any coding instructions first and it will tell you the word count. Respondents read English in western markets at an average rate of 5 words per second.
Q = Number of Questions: Count how many questions the average respondent has to answer. Allow 4 seconds per question general thinking time and 1 second navigation time* (assuming 1 question per page).
*this may vary depending on survey platform if it takes longer than 1 second to load each page adjust accordingly
D = Total number of decisions respondents have to make: Count in total how many decisions the average respondent makes in total using this guide below and allow then 2 seconds per decision.
Single choice question = 1 decision
Multi-choice question = 0.5 of a decision per option
Grids = 1 decision per row
T = Open ended text questions: Count how many open ended text feedback questions a respondents has to answer and allow 15 seconds per question. (note this may vary quite dramatically based on the content of the question but on average people dedicate 15 seconds to answering and open ended question).
W= word count
R = total number of row options: Note this is just rows and not columns on a grid. This can be quite easily done by cutting and pasting your survey into excel and then in a side column mark up all the rows and then sort.
This will give you a rough estimate of the length of a survey in minutes. It is no where near as acurate as the above 2 more detailed methods but it will be someone in the correct ball park. Careful though if you spot your a dealing with a particularly verbose questionnaire.
A wisdom of the crowd approach I would recommend would be to use both the 2.5 question and W/150 methods and compare the differences - if they produce just about similar figures well go with that, if they generate big differences it might be worth adopting method 1 to do it properly.
1. If all the respondents don't see all the questions: Skip logic can mean not everyone sees every question in a survey which means it can be hard to work out the average number of question respondents will have to answer which you will need to know to accurately work out the average survey completion time. Most errors in estimating survey length centre around this issue. There is often no easy way of doing this other than manually working it out using a spreadsheet.
2. Not properly taking into account question loops. This another issue that leads to people miscalculating the length of a survey. If for example there is a loop of question that you ask for a set of brands people often forget to include the extra time it take to answer these question and only count one loop.
3. If you are working out the length of a survey not conducted in English: or where English is not the primary language (India for example) you will need to weight for longer reading, comprehension, consideration and survey loading times in different countries. Below is a rough weighting guide if you are working from a translated version of an English survey, (sorry that I don't have time weighting data from every country):
4. If there are a lot of images in the survey: you will need to allow for extra loading times. Allow between 2-10 seconds per mb.
The reason is we have not established in the research industry a common and reliable way of estimating the length of a survey. The most common method in circulation is to assume we answer surveys at 2.5 questions per minute but this technique is fatally flawed. This is because question themselves can vary wildly in length e.g. a survey with 10 grid question with say 50 options may take 50 times longer to answer than a survey of 10 simple yes no questions.
So I have been on a bit of quest to work out some slightly more accurate ways of do this. As a result of some recent work we have been doing to examine in detail how long respondents answer survey I have come up with 3 new alternative methods I would like to put forward to more accurately calculate the length of a survey.
I hope they may be of use to some of you.
Method 1: Survey length = (W/5 + Q*5 + (D-Q)*2 + T*15)/60
This is the most accurate way of doing it (though I recognise it take a quite a bit of work). This formula will given you the length of an English language survey in minutes.
W = word count: Do a word count of the total length of questionnaire (questions, instructions and options). An easy way to do this is to cut and paste the survey into word but don't forget to remove any coding instructions first and it will tell you the word count. Respondents read English in western markets at an average rate of 5 words per second.
Q = Number of Questions: Count how many questions the average respondent has to answer. Allow 4 seconds per question general thinking time and 1 second navigation time* (assuming 1 question per page).
*this may vary depending on survey platform if it takes longer than 1 second to load each page adjust accordingly
D = Total number of decisions respondents have to make: Count in total how many decisions the average respondent makes in total using this guide below and allow then 2 seconds per decision.
Multi-choice question = 0.5 of a decision per option
Grids = 1 decision per row
T = Open ended text questions: Count how many open ended text feedback questions a respondents has to answer and allow 15 seconds per question. (note this may vary quite dramatically based on the content of the question but on average people dedicate 15 seconds to answering and open ended question).
Method 2: Survey length = (W/5 + R*1.8)/60
If you want slightly simpler approach use this formula which is not quite so reliable but will get you close...W= word count
R = total number of row options: Note this is just rows and not columns on a grid. This can be quite easily done by cutting and pasting your survey into excel and then in a side column mark up all the rows and then sort.
Method 3: W/150
If you don't have enough time to add up all the number of questions and row options this is another quick a dirty method (though I would not vouch for it being much more acurate than the 2.5 questions per minute approach).This will give you a rough estimate of the length of a survey in minutes. It is no where near as acurate as the above 2 more detailed methods but it will be someone in the correct ball park. Careful though if you spot your a dealing with a particularly verbose questionnaire.
A wisdom of the crowd approach I would recommend would be to use both the 2.5 question and W/150 methods and compare the differences - if they produce just about similar figures well go with that, if they generate big differences it might be worth adopting method 1 to do it properly.
Where all these formula will fall over?
2. Not properly taking into account question loops. This another issue that leads to people miscalculating the length of a survey. If for example there is a loop of question that you ask for a set of brands people often forget to include the extra time it take to answer these question and only count one loop.
3. If you are working out the length of a survey not conducted in English: or where English is not the primary language (India for example) you will need to weight for longer reading, comprehension, consideration and survey loading times in different countries. Below is a rough weighting guide if you are working from a translated version of an English survey, (sorry that I don't have time weighting data from every country):
| Length Weighting | |
| Japanese/Korean | 0.95 |
| Netherlands | 1.00 |
| Germany | 1.05 |
| French | 1.06 |
| Spain | 1.09 |
| Scandinavia | 1.10 |
| Italy | 1.11 |
| Chinese | 1.13 |
| India | 1.34 |
| Eastern Europe | 1.35 |
| Russia | 1.37 |
| Latin America | 1.43 |
4. If there are a lot of images in the survey: you will need to allow for extra loading times. Allow between 2-10 seconds per mb.
5. If you are including a lot of non-standard question formats in the survey e.g. sorting and drop down style question take longer to answer.
6. If you are boring people to death with a long highly repetitive survey! Respondents will start speeding when they get bored and so average decision making time can drop.
Now I would love to hear from anyone who has some thoughts on this or have come up with what they think is a more effective means of doing this. My ultimate aim is to find an agreed means for the who industry to adopt to use as a more effective trading currency when pricing surveys.
Do you have any thoughts?
Now I would love to hear from anyone who has some thoughts on this or have come up with what they think is a more effective means of doing this. My ultimate aim is to find an agreed means for the who industry to adopt to use as a more effective trading currency when pricing surveys.
EmoticonEmoticon