There are various goals to an open ended question but in most cases it is not about the volume of feedback but about the quality. Whether you are trying to get respondents to be analytical, creative or spontaneous, the biggest challenge you face is encouraging people to think and think in the right sort of ways.
The average respondent spend 15 seconds answering the average open ended question, and you get on average of 5 words. Five words might be enough if it constitutes meaningful feedback, but often it is humdrum verbiage, a bit of a nightmare to analyse. If you were to ask people to watch an ad and write down what they thought of it, the most common response would be "it was OK" . The second and third most common remarks would be, "I liked it" and "I didn't like it" , the 4th would be "I don't know" . These responses are clearly not a great deal of value, so that you might as well have asked a yes no question, “did you like it or not”.
This is a guide to show you how to write more effective open ended questions to improve the quality of feedback and to encourage respondents to:
- Put more thought into their answers
- Be more creative
- Be more analytical
- Be more reductive
- Be more free form in their thinking
Techniques for Motivating respondents to give you the type of answer you want.
The first challenge you face is to motivate respondents to answer the question you pose. Often respondents have little or no reason to write something down in the first place. So the question you should ask yourself is, what would make me want to answer this question?1. Explain the reason why you want to know the answer: It may not always be possible or relevant to do this, but explaining why you want to know can help motivate people to give an answer. Instead of asking "what do you like about this product" you might ask it like this: "This is a very popular product but we are not quite sure why, what do you like about it?".
2. Bring respondents into the problem: I would also advocate bringing respondents into the problem you are trying to solve if you an. So instead of saying "why do you like this brand" you might say "This brand is a lot more popular amongst some people than others and we are trying to understand why."
3. Emotionalise your appeal for feedback: The more you can emotionalise the appeal, the better. Below is an example of the impact this can have. We asked one group of mothers to talk about their experience of using baby wipes and to another group we explained that the reason we wanted them to talk about their experience was to try to improve baby wipes for future generations of parents, and we supported it with this visual. We got over 100% more feedback from this group.
4. Use visuals: Using a well chosen visual to re-enforce questions, as the example above demonstrates, can have a measurable impact on improving the volume of feedback (see my other full blog poste on this). Take the example below, which was part of an international experiment we conducted looking at the general effects of taking a more creative approach to survey design. We asked one group to imagine that they were in charge of the TV coverage for the Olympics games and had to build up a list of sports they would want to cover. We visually re-enforced this request to a second group with a picture of the Olympic logo and a camera. We have tested this out in 15 countries. In every one the presence of the image increased the feedback by upwards of 20%.
5. Challenge them: Using language that challenges respondents to do something can be an extremely powerful weapon to encourage feedback. Instead of asking "what brands come to mind", you might re-phrase that to say "how many brands can you guess". Or instead of saying "what words come to mind when you think of this brand" , you might say "can you think of the most popular words people associate with this brand". This type of approach can sometimes double the level of feedback, we have discovered.
6. Use reward mechanics: Have you ever thought about offering a little prize for the best answer? I can tell you, it is an extremely powerful approach. In experiments we have run we have seen up to 6 fold improvements in feedback. It’s obviously a bit complicated to administrate, but is a somewhat overlooked technique to my mind.
Set expectations on type of feedback you want: Most people use their own standard to assess how much time and effort they should put into answering an open ended question. We have found that establishing some expectations without making them actually conditional can help improve the volume of feedback. There are several different ways you can do this.
7. Use word counters: Asking people to write X number of words on a topic and using a word counter so they can see how much they have written is a great way to extend the volume of open ended feedback. The x limit might be 3 words or 300 depending on the circumstances.
8. Set time limits: Likewise telling them how long they might spend answering a question is another clever technique. If you say "please spend no more than 3 minutes telling us what you think about this topic" this may sound like you are asking someone not to spend too long. In fact you are doing the opposite and establishing a benchmark, that if they come anywhere near to adhering to, they will have spent far than the 15 seconds they normally would spend answering the question.
9.Show examples: By showing examples of what other people have written you can subtly socially condition people to write more. If they see that other people have written several sentences it will encourage them to write a similar amount.
10. Deal with spelling and literacy concerns: One of the reasons why some people are reluctant to give open ended feedback in surveys is because they are worried about their spelling and how well they can write. There are a couple of ways of dealing with this. First, make it clear that you are not worried about this. Secondly we have found that showing a range of informal feedback, with not necessarily perfect spelling, encourages respondents to be less concerned about their personal style of delivering feedback.
11. Foreplay - now's not the right time to get intimate! People are often reluctant to answer open ended questions at the very start of a survey; in fact, this type of question can be the cause of considerable drop-out. You need to warm people up and get them in the right frame of mind to answer open ended questions first.
12. Tackle boredom thresholds: People put far less effort into answering open ended questions at the end of a survey and there are often trigger points you will notice wherein people tend to give up, and their desire to get to the end of a survey cuts in so strongly that their responses to open ended questions dry up.
13. Being overwhelmed by choice of what do or say: Sometimes people are overwhelmed by the choice of things they could say in an open ended question and it leads them to say just one thing. If I ask you, for example, what your favourite bands are, this is a highly complex question to answer as someone may like lots of different bands and music artists and with all that choice your mind implodes and you can’t think of any that stand out. This issue has been observed by social scientists in a famous experiment in which people were asked to name something they might find in a fridge. The average person got stuck after naming 7 items. Yet if they were asked to name white things, or things with sell by dates, or bottled items, people were able to list a lot more. The lesson to learn from this is that when wording questions, you need to think of respondents like oysters: you need to feed them the grains of sand to stimulate the pearls of thinking and help them focus. So you might ask “what was the first band you fell in love with?” or “what was the last?”.
14. Break out into thinking chunks: If you are asking what do you like or dislike about something you will find that if you break this out into 2 open end text boxes ("likes" and "dislikes" ), you will get more responses. I often see open ended questions that try to ask several questions at once all in one question. My advice would always be to chunk it up and separate out to unique questions the separate things you want to know. You can cluster them together so it comes across as one specific topic you are asking about, but you will get much better feedback if you do it this way.
15. Use random stimulation: Giving respondents some sort of random stimulus can be an extremely powerful means of unlocking ideas. Say you are wanting people to think of an idea for a new type of ice cream; you could create maybe 20 or so random thoughts to help people consider things from different angles. This could be anything such as "if Lady Gaga were creating an ice cream what would it be like", "invent an ice cream to eat on cold days", "create an ice cream you think football fans would like" etc - the more abstract the better and to each respondent you present a random set of these stimuli, one at a time.
16. Set rules: I have talked about this quite extensively in some of my work on the gamification of research, but adding a rule to a question can make it a whole lot more interesting and fun for respondents to answer and thus trigger more thoughtful feedback. The easiest way to do this is by placing word limits - e.g. review this film in 3 words.
17. Make it personal: Always remember you are one human being talking to another and most people's favourite topic is themselves, so try wherever possible to frame the question in a personal context. Instead of saying “why do you like this brand?” you might say "if a friend asked you what you thought of this brand what would you say?".
18. Realise the power of the word "imagine": The word imagine is a magical word to use as a means of getting people to think. We did an experiment in which we asked one group of people to make a list of their favourite shops and to another group said "imagine you could design your perfect shopping centre, what shops would you have in it" - this increased the feedback 4 fold.
Understand the barriers for respondents to give feedback
11. Foreplay - now's not the right time to get intimate! People are often reluctant to answer open ended questions at the very start of a survey; in fact, this type of question can be the cause of considerable drop-out. You need to warm people up and get them in the right frame of mind to answer open ended questions first.
12. Tackle boredom thresholds: People put far less effort into answering open ended questions at the end of a survey and there are often trigger points you will notice wherein people tend to give up, and their desire to get to the end of a survey cuts in so strongly that their responses to open ended questions dry up.
13. Being overwhelmed by choice of what do or say: Sometimes people are overwhelmed by the choice of things they could say in an open ended question and it leads them to say just one thing. If I ask you, for example, what your favourite bands are, this is a highly complex question to answer as someone may like lots of different bands and music artists and with all that choice your mind implodes and you can’t think of any that stand out. This issue has been observed by social scientists in a famous experiment in which people were asked to name something they might find in a fridge. The average person got stuck after naming 7 items. Yet if they were asked to name white things, or things with sell by dates, or bottled items, people were able to list a lot more. The lesson to learn from this is that when wording questions, you need to think of respondents like oysters: you need to feed them the grains of sand to stimulate the pearls of thinking and help them focus. So you might ask “what was the first band you fell in love with?” or “what was the last?”.
How to stimulate more thinking
15. Use random stimulation: Giving respondents some sort of random stimulus can be an extremely powerful means of unlocking ideas. Say you are wanting people to think of an idea for a new type of ice cream; you could create maybe 20 or so random thoughts to help people consider things from different angles. This could be anything such as "if Lady Gaga were creating an ice cream what would it be like", "invent an ice cream to eat on cold days", "create an ice cream you think football fans would like" etc - the more abstract the better and to each respondent you present a random set of these stimuli, one at a time.
16. Set rules: I have talked about this quite extensively in some of my work on the gamification of research, but adding a rule to a question can make it a whole lot more interesting and fun for respondents to answer and thus trigger more thoughtful feedback. The easiest way to do this is by placing word limits - e.g. review this film in 3 words.
17. Make it personal: Always remember you are one human being talking to another and most people's favourite topic is themselves, so try wherever possible to frame the question in a personal context. Instead of saying “why do you like this brand?” you might say "if a friend asked you what you thought of this brand what would you say?".
18. Realise the power of the word "imagine": The word imagine is a magical word to use as a means of getting people to think. We did an experiment in which we asked one group of people to make a list of their favourite shops and to another group said "imagine you could design your perfect shopping centre, what shops would you have in it" - this increased the feedback 4 fold.
19. Apply NLP techniques: I don’t pretend to be an expert in this but the concept of the embedded command in a question can be extremely powerful. I could say to you "tell me what you do in 7 words" or I could say "tell me what you do in EXACTLY 7 words". Respondents seem to treat the word EXACTLY as more of a command, so it tend to solicit more responses.
20. Study the form: It is interesting to see which discussion topics posted on linkedin trigger the most comment and involvement. By studying this you can start to get a feel of how to pitch a question that solicits a strong response.
In experiments the biggest uplift in feedback we have ever achieved as a result of redesigning the way we asked an open ended question has been a 6 fold increase from 17 words to 103 worlds. This used a combination of techniques listed above and might be described as my current PB. With the right thought and approach you can easily achieve double or treble the volume of feedback by re-engineering the wording of a question. But note that, as I said at the start, it is often not about volume of words so much as about the clarity of thinking these words stimulate. We find that with increases in volume feedback is also linked to more thoughtful feedback.
All this begs the question about what you do with all this open ended feedback, which can be overwhelming. Some clever ways of automatically processing open ended feedback have emerged with social media text analytic software and natural language computational techniques, and there are some very interesting things you can do with respondent self coding feedback, but this I think is a topic for a future post....
20. Study the form: It is interesting to see which discussion topics posted on linkedin trigger the most comment and involvement. By studying this you can start to get a feel of how to pitch a question that solicits a strong response.
What can you achieve if you get it all right?
In experiments the biggest uplift in feedback we have ever achieved as a result of redesigning the way we asked an open ended question has been a 6 fold increase from 17 words to 103 worlds. This used a combination of techniques listed above and might be described as my current PB. With the right thought and approach you can easily achieve double or treble the volume of feedback by re-engineering the wording of a question. But note that, as I said at the start, it is often not about volume of words so much as about the clarity of thinking these words stimulate. We find that with increases in volume feedback is also linked to more thoughtful feedback.
EmoticonEmoticon