The impact team is a research team, and all research boils down to asking questions. There are the broad questions - ‘Is our service helping people to feel more connected?’ - and the questions we ask people to help us answer that.
Impact volunteers call our older people and volunteers to ask questions and have a script to use as a guide. Whenever you are asked to call people, you will be given clear instructions by the impact team as to what the questions are, why we’re asking them, and what your approach should be.
As you’ll have gathered from the introduction to evaluation, we ask questions in a variety of ways, ranging from the very structured (such as questionnaires) to the less structured (such as focus group discussions).
Questionnaires
When people complete questionnaires their responses come to us exactly as they are written and when we evaluate the responses, we are not generally interpreting what people have said.
So, when we call people to get their responses to the same questions, we stick to certain protocols:
- we introduce the questions in the same way and ask them in the same order, using the same wording.
- where prompts are needed, we try to avoid phrasing that might encourage one type of answer.
- we write down the exact response, rather than paraphrasing or summarising.
Interviews
If we call people to interview them, though, our approach can be a bit more relaxed – we want to establish a rapport with people to give them an opportunity to tell us as much as they want to. We still need to follow protocol in terms of introducing who we are and why we’re calling, especially when we’re explaining how we’ll record their information and their right to stop the call or decline to answer a question. But we’re more interested in having something like an open conversation about the things we want to ask about.
Focus groups
In a group situation, we’ll also have a set of topics we want to talk about (called a discussion guide), but the nature of a group discussion means that conversation is more open to going in different directions. One of the benefits of speaking to people in groups is getting to see a consensus form (or not) and the viewpoints/arguments that come up in that process.
Example
One set of questions that you will be asking regularly is our evaluation and well-being questionnaire.
In the introduction to evaluation, we talked about measuring our impact. One of the ways that we do that is to survey our older people at regular intervals, we use our evaluation and wellbeing questionnaire, which has two parts.
The first part of the survey asks about someone’s experience of Re-engage, and includes asking people the extent to which they agree or disagree with a set of statements about the impact that joining Re-engage has had, such as 'As a result of joining Re-engage, I feel happier.'
This helps us understand the ways our services make a difference, to see if some services are more successful in one regard than others, and to identify if there are groups of people who do not seem to experience the same benefits as others when they join Re-engage.
The second part includes standardised questions about someone’s feelings of isolation and loneliness, life satisfaction and happiness, and their social relationships.
The questions are those used by the Office for National Statistics (ONS) in their UK population surveys, such as “How often do you feel that you lack companionship?”. People are asked to choose from three responses: ‘Often’, ‘Some of the time’, or ‘Hardly ever or never’.
By using standardised questions, we ensure that everyone is asked the same questions in the same way, and their responses are recorded in the same format. We can also compare our findings to the general population and to other interventions (benchmarking), giving us an accurate picture of our impact.
Re-engage never uses a person’s responses to these surveys to include or exclude them from what we do. We look at totals and averages rather than individuals, so that we can consider the extent to which our services make a difference to our older people’s lives, provide reliable and up-to-date evidence of those changes, and develop and improve our services.
Please ensure that you read the full guidance on evaluation and wellbeing surveys.