Guide to questionnaires for service evaluation

Introduction

This guide has been designed to be used for reference if you are considering using any form of questionnaire in your service evaluation.

If you are in the Bristol, North Somerset and South Gloucestershire (BNSSG) area please register your survey activity with [email protected].

This guide covers:

1. Choosing to use questionnaires
2. Defining objectives
3. What format should it take
4. Sampling and involvement
5. Questionnaire design
6. Piloting your quesionnaire
7. Coding and scoring data
8. Analysing data
9. Presenting results
10. Ethics, data protection and information governance
11. Disseminating your findings

1. When should I use a questionnaire in my service evaluation?

A questionnaire may be useful to you if…

• Time and resources are limited
• You want to collect data from a large number of people
• You want a few standardised responses
• You want to find out about knowledge, beliefs, attitudes or behaviour
• Anonymity is important
• The Evaluator is unlikely to be present during data collection
• You want to collect data via post or online
• You want simple answers to basic questions
• You need mostly quantitative results
• You accept the limitations (see box 1)

Questionnaires are not the right choice if…

• You don’t yet know what the main issues or themes are
• You want detailed information
• You want to know about a complex or controversial issue
• Your respondents may need probing; or not necessarily have an opinion.
• Your population might struggle with reading or understanding the questions
• Your population cannot use or access a computer (for online surveys)

Advantages of questionnaires

• Quick, cheap and easy to administer
• Can measure behaviour, attitudes, preferences, opinions and intentions
• Can be used to quasi-experimentally e.g. to measure change before and after an intervention
• Evaluator does not need to be present during data collection
• Variety of question formats e.g. lists, grids, scales, ranking
• Can collect qualitative and quantitative data
• Can be analysed more ‘scientifically’ than many other forms of social research method
• Data can be collected face-to-face, over the phone, on paper, or electronically.
• Software such as SurveyMonkey is available
• Lower response rate than some other methods; sample easily skewed

Disadvantages of questionnaires

• Can lack validity and reliability
• Social desirability, poor recall, or misunderstandings can result in response bias
• Questions can be leading or misleading
• Long questionnaires can be off-putting; short ones don’t get enough information
• Data protection, information governance and ethics must be addressed.
• Hard-to-reach groups can be excluded; accessibility issues; language can be a barrier.
• Not sensitive instruments; they will not tell you about emotions, feelings, or relationships.
• Errors can be made in coding, data entry and data analysis.
• You cannot usually go back to respondents for clarification

2. Defining questionnaire objectives

To stay relevant and coherent, you need to be clear what you want your questionnaire to achieve. Every question must justify its place in the final cut.

It may help to reflect on what you are interested in and write an objective for each theme. Common areas to focus on include

• Behaviours
• Attitudes
• Experiences
• Preferences
• Opinions
• Intentions

It is also important to consider what the results will be used for:

• Service design
• Service improvement
• Monitoring data
• Managing a message
• Assessing knowledge and understanding
• Outcome measures
• Staff/ Patients/ Public Experience measures
• Satisfaction measures
• Service usage data

3. What format should I use?

There are four main formats that you might consider when deciding how to administer your questionnaire. You may choose more than one approach. To an extent, the format of your questionnaire will depend on the type of questions you want to ask:

Open-ended questions (e.g. “How do you feel about using questionnaires?”)

• No set response
• No standardised responses
• Can generate discussion
• Mostly qualitative data

Closed-ended questions (e.g. “Do you like questionnaires?”)

• Not sensitive to nuance
• Can standardise responses
• Can compare pre- and post-intervention
• Mostly quantitative data

Postal questionnaires

This is when you send your questions to your sample population and hope as many of them as possible fill it in and send it back to you.

Best suited to closed-ended questions, large (or very engaged or primed) populations, and using close-ended questions that have either been validated or thoroughly piloted for sense and efficacy. Relatively cheap and easy to administer. Include an pre-paid reply envelope with mail-outs.

Data quality can suffer from a low response-rate, response bias or recall errors, or people not understanding the questions. Labour-intensive to code and enter data. Layout, design, ease of use, and question phrasing is critical to an effective postal questionnaire.

Electronic questionnaires

This is when you either email a copy of your questions or use an online tool such as SurveyMonkey (www.surveymonkey.com) or SmartSurvey (https://www.smartsurvey.co.uk/).

The BNSSG Clinical Effectiveness team has enhanced access to online surveys and can help you formulate and set up your online survey, and give you direct access to the results. Please note this service is only available if you are providing a service commissioned by the ICB. Contact [email protected] if you would like to use this service.

Online questionnaires are very cheap and quick to administer and have the potential to reach vast numbers of people. Data collection and analysis is automated, which saves lots of time. They are clear and easy to use, and the software can help with question planning. You can develop your questionnaire and view results online or via an app from any mobile device.

However, you will only get responses from people who can read and understand the questions, as well as who have access to a computer. You are unable to follow up on respondents unless you have specifically requested their contact details, which they may choose to withhold. You cannot follow up on non-responses.

Phone questionnaires

An alternative approach is phoning people and asking for feedback.

This format can be cheap and quick (although you still need to code and input the data so somewhat labour-intensive). You are likely to get a good response rate, and can build a rapport with the respondent.

Best suited if you have a small number of people in your sample, and anonymity is not an issue. Useful for people for whom literacy is an issue, although relies on being able to communicate well on the phone.

Suits both open- and closed-ended questions, although caution should be used when recording or transcribing responses.

Face-to-face questionnaires

This is when you run through your questions directly with the respondent. The response rate will be good, although it must be recognised that people may not answer as thoroughly as they might if their anonymity can be guaranteed.

In an NHS service evaluation context, this approach can be useful if you just have one or two questions that can be added into assessments or provider monitoring data. Face-to-Face questionnaires can be for both open- and closed- ended questions.

If you feel that face-to-face questionnaires are the best method for you, you would be advised to re-visit the criteria in section 1 of this guide as it may actually be that an interview or focus group is a more appropriate method for what you want to do.

Use this table to help you decide what format will best suit your needs. It may be that you want to put your questions into more than one format to get the best and most thorough response.