Guide to Questionnaires for Service Evaluation

Introduction

Writing a concise introduction to your questionnaire not only ensures that you have identified your objectives clearly, but helps secure informed consent from respondents.

Your introduction should detail:

• What the questionnaire is for
• Who should answer it
• What arrangements, if any, have been made to ensure anonymity
• How the data will be stored
• How to access the questionnaire in other formats
• How to return it
• How long it is likely to take to complete
• How long the questionnaire is open for/ deadline for responses
• Whether respondents need to answer all questions, or just the ones they think are relevant to them.
• Thanks for their participation
• When and how respondents can get feedback
• How respondents can get further information

Identifiers

Identifiers are unnecessary if your questionnaire is completely anonymous, but you may consider using a Unique Confidential Identifier if you want to collect data from identifiable respondents to be submitted confidentially. It may be that a service provider collects the questionnaire data from the patient and adds a unique confidential identifier to the data before submitting data to the ICB or CSU for analysis. This would enable follow up or sense checking with the respondent at a later date if there are queries and would advance the quality of the data.

Developing Questions

It would be impossible in the scope of this guide to synthesise the vast amount of theoretical and practical resources that are available on developing questions for surveys. So, we have included some generic principles and recommendations, as well as a selection of links that may help.

Don’t reinvent the wheel

Keep in mind that question development is much harder than it looks, and if you can possibly avoid writing your own questions, do so. Doing some scoping to try to identify tried and tested tools can save you a lot of time. Some you may need to pay to use. You can cherry-pick questions that suit your needs.

Online survey tools such as www.SurveyMonkey.com have a database of customisable questionnaire templates for various measures, supported by a question bank:  https://www.surveymonkey.com/mp/certified-survey-questions/.

In addition, your Library Services or Public Health Evidence team can help you identify what is already available, and give advice on how to appraise it effectively.

Sources of validated survey instruments:

Instructions

You need to be very clear what you want the respondent to do throughout the questionnaire. Specify whether you want them to tick a box, circle an option or write freehand. Do not assume that people understand your scales without explanation. Piloting your questionnaire (see section 6 below) is very important to ensure the instructions are clear and sufficient.

 Structure 

The questionnaire should have a logical structure and sections should flow coherently. Group questions together depending on whether they are focusing on

• Demographics
• Behaviours
• Attitudes
• Experiences
• Preferences
• Opinions
• Intentions

Always number sections and questions clearly.

Question wording 

This is absolutely critical to ensure that your questionnaire makes sense and is unambiguous.

Key things to remember are

• Avoid jargon
• Keep sentences short and uncomplicated.
• Avoid double negatives
• Avoid double-barrelled questions. “Do you agree that questionnaires are the cheapest and best way to evaluate health services?” is impossible to answer with a yes/no.
• For paper/ electronic questionnaires, use mostly close ended questions
• Be clear if you want people to tick a box, circle and answer, or rate on a scale.
• Avoid leading questions that may fall victim to social desirability bias. For example, “Overuse of A&E services are costing the NHS millions of pounds a year. Did you seek advice from any other healthcare service before coming to A&E today?” may not illicit a truthful response.
• Use a mix of positive and negative statements/ questions.[3].
• Limit response options. Categorise if necessary.
• Don’t use acronyms or abbreviations
• Be specific about timeframes – ‘recent months’ is open to interpretation; ‘during June and July 2014’ is better.

The Plain English campaign offers a selection of guides that can help you write clearly for a wide audience: http://www.plainenglish.co.uk/free-guides.html.

Layout 

It is critical that your questionnaire layout is easy-to-read, inviting and clear.

• Using coloured paper has been shown to increase response rate[4]
• Follow your organisation’s style guide for logos
• Use Arial 12 as standard; response boxes should be 20 point.[5]
• Use larger or bold font for headings or instructions
• Use shading to group sets of similar questions
• Leave lots of blank space
• Use comment boxes rather than dotted lines for freehand comments.[6]
• Proofread thoroughly (ideally have someone else do it).
• Check that your formatting is perfect.
• Keep your questionnaire as short as possible.
• Be mindful of raising expectations

 Scales 

Using scales in questionnaires can be helpful as they can measure strength of attitudes, rather than simply ‘yes/no’. Scales are relative and conclusions should be drawn with caution.

Commonly used scales include:

• Likert Scales: respondents are asked to rate their strength of feeling towards a statement or question. Simple to use and easy to code; although you rely on people reading the questions properly, understanding the scale and not simply ticking the neutral option throughout. Use a variety of positive and negative statements throughout. There are techniques to test data quality, such as including two contradictory statements and comparing the response.
• Semantic Differential Scale: Allows respondents to rate statements on a number of different dimensions[7]. Requires concepts to be mutually exclusive.
• Ranking: Asking respondents to rank items in order of preference can be a useful way of identifying local priorities that can feed into service design/ re-design. Best delivered in face-to-face questionnaires with options on a individual cards to be sorted. Can be done via other questionnaire formats by assigning numbers, although need to be careful with wording as easy to misunderstand instructions. Electronic questionnaires have the potential for people to move web-based icons into a ranked order.
• Visual Analogue Scale: Measures a characteristic or attitude across a visual continuum. It’s a very subjective measure and unlikely to be of use in service evaluation as is hard to interpret. Mostly used in psychometric testing to compare changes in an individual after an intervention.