Evaluation and Measuring Progress
“When I first joined…the key issue for the board and for the agency is that they really need to get outcomes. They had metric up the ying-yang and that is something that Covenant House does really well – it measures lots and lots and lots of things…what’s key is understanding how well we’re doing. So we needed to put together a key set of indictors, what we call key performance indicators, the outputs and the outcomes.” — John Harvey, Director of Program Services, Covenant House Vancouver
One of the ways an organization can measure whether it is achieving its mission and meeting its goals is through the development of ‘Key Performance Indicators’ (KPI). “Performance measurement is a process that systematically evaluates whether your efforts are making an impact on the clients you are serving or the problem you are targeting”.
While KPIs vary depending upon organization and program, a good Key Performance Indicator provides a measurement of success that is quantifiable. Being able to count allows for comparison between programs, between years or between organizations.
Performance Indicators generally measure ‘outputs’ – specific numerical measurements (clients served, number of youth rehoused, number of hygiene kits given out). Some will measure specific achievements – i.e. “80% of clients will successfully complete ROP within a year” whereas others measure progress towards overall organizational goals: “The average length of stay in shelters is reduced.”
KPIs allow an organization to determine what is and is not working and to adjust programming to improve outcomes. In selecting the correct KPI an organization has to have:
- a clear mission
- well-defined goals
- a set of desired outcomes
- prioritization of the most important factors.
An agency may have several goals and outcomes that it wishes to measure. By creating clear KPIs an organization can indicate which of these it views as most important. For example, a graduated transitional housing program may wish to see its clients move on to independent living within a certain time frame, say 12 or 18 months. However, it also wants its clients to complete all steps in the program before graduation (see Steps section for an idea of a graduated plan to completion). The agency needs to understand which of these is most important and therefore, which key performance indicator holds the most weight.
Therefore, if completion of all program steps is more important than moving on to independent living with a set number of months, success can be obtained when a high number of youth complete all steps even if it takes them longer to move into independent living. Since a step program would be aimed at providing youth with the necessary skills for independence, the likelihood of achieving long-term successful housing will be greater with this outcome than if youth are pushed through the program to meet the length of stay goal.
An effective evaluation program uses a logic model to join the key goals and outcomes together in a straightforward and organized fashion. The W.K. Kellogg Foundation defines a logic model as a “systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve.”
Logic Model Template
Resources dedicated to or consumed by the program
What the program does with the inputs to fulfills its mission
The direct products of program activities
Benefits for participants during and after program activities
Desired long-term result of the program
In 2006-2007 Covenant House Vancouver developed the use of Key Performance Indicators. After a pilot year, they committed to using the indicators for three years before doing an in-depth review of the data and making any programmatic adjustments. This began in 2008 officially.
“We just wanted to develop a pool of data so that we can say ‘ok, now what? What does this tell us after three year?’” — John Harvey, Director of Program Services, Covenant House Vancouver
Case Manager Lisa Ronaldson (ROP Vancouver) explains how the statistics that are collected at ROP are based on the youth’s individual case plan. “We look at what are they working on—employment, education. Are they stuck in a certain area? Health—so that could be mental health, it could be medical health, it could be dental health. Legal—if there’s any legal stuff and where they are in that continuum. Housing—if I’ve started to work with them on different types of housing, filling out applications.” Overall, she says, “it’s a way for us to keep track of the work that the youth is doing.”
The Efforts to Outcomes (ETO) database is a form of case management software, created by Social Solutions, a company based out of Baltimore, US. It is an electronic database system used to measure different program outcomes. With Covenant House, ETO has significantly improved the agency’s ability to “collect, monitor and analyze what we are doing, who we are doing it with, and how well we are doing it” (Covenant House Washington website).
ETO is a flexible system, as it can be tailored to suit an organization’s specific needs. An organization may want to track numerous elements of their services, such as program outcomes or demographics. The ETO can be built to track certain elements of service delivery, as indicated by the organization. Ultimately, ETO allows an agency to organize the different interactions that occur between employees and stakeholders. Once the specified interactions are documented and added to the database, the organization can evaluate the data, and interpret the results, which facilitates greater program success, as agencies are able to see the benefits or downfalls of certain programs. According to the Social Solutions website, ETO “brings a performance management approach to improving the lives of participants under your care”. It is a systematic, evidence-based method that fosters careful program evaluation and eventually leads to improvements in service provision, as organizations understand what elements of their programming are successful, or require development. In terms of funding, ETO is an effective measure to ensure accountability and demonstrate program success to possible donors.
ETO provides different kinds of database management, suited to certain kinds of organizations. Initially, ETO conducts a customization process, to determine which software is best suited to your organization’s needs. After implementation, ETO provides a training curriculum to address future changes, as an organization’s needs may be different over time.
For instance, Covenant House uses ETO to track a client’s progress, thus different reports are generated when a client enters a shelter, leaves a shelter and completes classes and programs. These interactions are then organized in the electronic database, along with other interactions of different clients at the same location, which allows Covenant House to produce data representing the trends and priorities of the clients at a specific Covenant House. As other Covenant House programs are also using ETO, this facilitates data sharing, as locations can compare and contrast the unique aspects of their centre with challenges encountered at other locations. Consequently, Covenant House is able to produce a comprehensive history of program participants, which illustrates how people have been helped over time, and identifies ways to improve Covenant House programming to successfully assist future participants.
Because ETO also serves as the ROP’s program database, ETO is able to do much more than just track data. The customizable nature of the program allows for basic forms to be created within ETO that facilitate electronic registration/intake and discharge, updating of case plans, tracking of referrals and key contacts etc.
A screenshot of an Efforts to Outcomes (ETO) intake form.
Covenant House Toronto uses the Youth Engagement Scale (YES) model to gather information about a youth’s engagement with various factors of the program. YES examines12 different “Points of Service” in a young person’s life using a 10-step scale that captures how engaged they are (actively working to make changes) on that particular issue including school, work, involvement in their program etc. The 10-step scale is linked to the Stage of Change transtheoretical model, which will be discussed in the Theories to Support the Work chapter.
The 12 different Points of Service are: Activities of Daily Living, Housing, Education, Employment, Motivation and Relationships, Support System/Community Connectivity, Strength Based Resiliency/Self-Efficacy, Legal, Medical, Mental Health, Substances and Financial.
According to CHT, in the area of employment for example, a youth being scored as Step 1 or Step 2 would fall into the Pre-Contemplation Stage of Change and their behaviour would include being “Unemployed and not interested in seeking employment, hostile or cold toward idea of employment and rejects offers of employment coaching/assistance”. As the youth moves up the scale and becomes more engaged with this Point of Service their behaviour would transition to a more positive outlook until they are “employed in a sustainable job providing a living income”.
CHT is in the process of rolling out visual tools to help staff and youth see how YES measures their progress. The graphs below are still in development, but provide a sample of how YES can show the growth and progress of each youth.
As these sample graphs show, a youth can progress in many ways. The system allows comparisons between several points (as shown in the charts on the left hand side) or between two specific points in time (as shown in the charts on the right hand side).
“[It] has really helped around just tracking in a better way the impact that we have on young people. We work with young people from 16-24 and sometimes it takes a number of years before you can really see that we’re having an impact. So this tool really helps us to check our progress.” — Carol Howes, Director of Program Services, Covenant House Toronto
The Case Management Team approach is very effective when combined with YES because while one worker might not see a youth’s engagement as very high, another may have had a detailed conversation about that specific factor and so a melding of scores can be obtained.
The YES model can also be used to demonstrate progress to a youth.
It is important to note that with the YES model, and with Outcomes Star described in the next section, progress is not always linear. Youth may make forward progress in Activities of Daily Living but backward progress in Employment. This is understood to be natural and both staff and youth are informed that this should not be seen as a failure in any way.
The Outcomes Star is an approach used to measure change when working with vulnerable populations, particularly homeless people. The model was developed by Triangle Consulting, originally for St. Mungo’s Broadway, a UK charity dedicated to supporting homeless people. There are several versions of the Outcomes Star adapted to different groups, such as the elderly and people suffering from mental illness, yet we will focus on the Outcomes Star approach adapted for homeless populations.
How it Works
The Outcomes Star method is based on the idea that people experiencing change move through several stages while transitioning from dependence to independence. The Outcomes Star approach identifies ten outcome areas: motivation and taking responsibility, self-care and living skills, managing money and personal administration, social networks and relationships, drug and alcohol misuse, physical health, emotional and mental health, meaningful use of time, managing tenancy and accommodation and offending. Within each outcome area, the Outcomes Star approach measures change through a specific ladder of change. The Outcomes Star Chart, pictured below, provides a visual representation of this process.
Outcomes Star Chart
Ladder of Change
The Ladder of Change combines the theory behind the Stages of Change model (discussed in the Theories to Support the Work chapter) with the Outcomes Star. The first stage of the ladder is “stuck”, which means that when people begin a change process, it is initially very difficult. Many people find it challenging to accept that they have a problem that requires change, and/or they may not want to accept help from others. The second stage is “accepting help”, which is when people seek outside assistance, as they no longer want to tackle their issues alone. The third stage is “believing” which is when people begin to acknowledge that they will be able to change. When someone begins to believe in their ability to change, they often anticipate future benefits, and are much more welcoming of outside assistance. The fourth stage of change is “learning”, which is when people try different strategies, in order to change. This is a challenging stage, as most people are not usually successful on their first attempt to change, which can be a discouraging process. The last stage of change is “self-reliance” which is when individuals are able to maintain change without assistance from others. Many individuals begin the ladder at different stages, and do not necessarily progress linearly, as it may take several attempts to reach the self-reliance stage.
Ladder of Change Model
As you complete the Ladder of Change for each outcome area, you are instructed to measure your progress on the Outcomes Star, by indicating which number you are at, for the different ladders of change. Once you have chosen a number for each outcome area, you link the different numbers together to create your own star, which represents your progress and room for further development.
The image below is a composite of several youth and their progress as viewed on the Outcomes Star. A profile of an individual youth would look similar.
Program Outcomes for Individual Youth
(8 February 2013 - 9 September 2014, Vancouver)
Homeless Hub Thoughts:
Overall, both organizations do a great job collecting and using data and making sure that all programs are evaluated. While many funders ask for this level of evaluation, few provide the financial supports to achieve it. Covenant House’s private funding base allows them to dedicate the resources necessary to build evaluation in to their work across the board.
Evaluation is certainly critical to any program operation. There are two ways to think about evaluation in a transitional housing program model: evaluation of the individual youth’s progress and success of the program activity as a whole. Both agencies have established a variety of measures, including individual surveys, focus groups, the use of Efforts to Outcomes and the two youth engagement tools: the Youth Engagement Scale and the Outcomes Star.
We like Vancouver’s method of setting up the parameters for data collection, creating an ongoing process for gathering information and then not reacting to the data for a set period of time. This is a good practice to ensure that you are noticing a trend rather than outliers. One of the keys to good data collection is patience.
It is also important, as Vancouver has emphasized, to not “chase numbers” but have a clearly defined logic model and understanding of what your key indicators are. This makes it easier for you to gather and use data without staff feeling that all they do is fill in forms and create numbers.
Some of the factors in your evaluation may be based upon the needs of your funders. If their primary goal is for you to house youth faster, then you will need to be able to compare how long it used to take to house youth and prove that your methods are making housing youth easier and faster. If funders are more concerned about the development of life skills, then your evaluation methods would most likely include a pre and post life skills assessment of the youth (maybe every few months depending upon length of stay), as well as tracking the types of sessions offered, the number of residents who participated in each one etc.
Listening to the youth and seeking their input was also discussed by both organizations. They also emphasized the need to listen to staff as they are the ones working in the program at the ground level. It is important to include youth and staff input in your data and evaluation methods. When creating your key indicators both groups should be part of that process. Ideally, hard numbers will also be supplemented by qualitative surveys, interviews or focus groups to help understand and expand upon the data. While singular anecdotes are not evidence, gathering stories can help explain trends.
Youth evaluation may form part of your overall data collection but can also be used in other ways. For example, as noted above, pre and post life skills assessments may meet the needs of a funder, but they can also be used to measure and support the progress of an individual youth.
Using tools such as the Youth Engagement Scale or the Outcomes Star measure success and progress by a youth but they can also identify areas of concern for a program. If only a few youth are progressing in a certain item on a scale it could be a sign that program changes are needed to improve service delivery in a certain area.
These tools are also integrated into the service delivery; they are not just separate, standalone evaluation measurements. For example, the areas being assessed in Toronto’s Youth Engagement Scale are all embedded into the goals of the program. They make up part of each youth’s case plan and are discussed in the Case Management Team meetings.