Our intentions may be good, but how do we know our services are making a difference?
At Crisis we have long believed that we transform the lives of the homeless people we help, but through the work we are currently commissioning we finally have robust evidence which demonstrates that we are making a difference and – importantly – enables us to maximise the positive impact our work has on the homeless people we help.
The final results of the independent evaluation of Crisis’ Skylight services by the Centre for Housing Policy at University of York will be available later this year. Today we publish interim findings from this major three-year longitudinal evaluation, one of Crisis’ most important investments in knowledge, our first looking at the impact of the whole service rather than discrete elements of the model.
At the heart of the study is a qualitative cohort study – tracking a group of 135 homeless people as they use Skylight centres (and after they leave), looking in detail at their experiences of progression. As part of the evaluation we’re also gathering the views and life stories from an additional 280 members via focus groups, talking to staff and external partner agencies as well as reviewing statistical information from Crisis’ client recording system.
The scale of the evaluation is probably unprecedented in the sector, and I can’t help but feel fortunate to work for an organisation which focuses on understanding and maximising impact.
The second interim findings report marks an important milestone in the life of the project as it captures the early results from the longitudinal analyses, showing what’s been happening to the participants that we’ve been tracking over time.
So what happened to cohort participants in the first two years? The report shows that the majority of people progressed and reported positive experiences of the service. Among those who had progressed, three typical pathways are identified: regaining progress, moving forward for the first time and punctuated progression.
Participants in the first group were those whose ‘normal’ or positive trajectories through life had been interrupted by homelessness – they had been working, in training or in further/higher education but this was disrupted by homelessness or threatened homelessness. For this group of people, the experience of progression with Skylight was centred on returning to their former situation or regaining progress. The individuals were likely to connect directly with training, volunteering and job seeking services, rather than use arts-based activities.
In the second group were people who had sustained experience of unemployment, poor educational attainment and had experienced stigmatisation and marginalisation from a young age. Skylight connected them to positive trajectories for the first time or the first time in a long time. This group of cohort participants were more likely to have higher support needs, sustained experiences of homelessness and history of problematic drug/alcohol use. Here, engagement with a Skylight would quite often begin with basic skills education or arts-based activity and the process of progression would build up over time.
Finally, there was a pattern of punctuated progression in which participants were able to progress but then experienced setbacks and required assistance again. The reasons for disrupted progression could be external; e.g., a job was secured, but was only a three or six month contract, then there could be some need to fall back on Skylight to help secure another job.
But what impact did our services have on participants? The report shows clear evidence of gains in health and well-being, particularly in relation to improvements in self-confidence. 88% of cohort participants reported progress in one of more areas (education, training, job-seeking, volunteering), despite often facing very significant barriers to progression. 23% found paid work, and 13% moved into training, further or higher education. The report concludes that two elements in the model are crucial to its effectiveness: one-to-one coaching (especially valued and outcomes were better when it was in place), the holistic nature of the service (can help with job search, housing, education, self-esteem), and the availability of multiple points of engagement (important in working with different groups of homeless people; e.g. being able to begin with arts-based activity or going straight to help with job seeking).
So we have a lot to feel pleased about in respect to how Skylight is working and what it is achieving, but the evaluations also identifies some learning points which we are now planning to use to improve further the quality of the services (a key aim of the project). Service drop out was an issue across all centres, which is not unsurprising given the nature of the population we work with but there’s no room for complacency. Although the comprehensiveness of Skylight is a huge strength, we don’t always engage and support people in a consistent manner both within and across centres.
We know that when the project concludes in summer 2016 the final report won’t provide all the answers. But what it will do is provide the strongest evidence base yet in the UK on the value of support designed specifically to transform the economic and social position of homeless people. For us impact is everything, and it is crucial to have strong evidence to demonstrate to others – our trustees, funders, volunteers and importantly service users themselves – the impact of our work and maximise the positive impact our services have on the homeless people we help.