Evaluation and Measuring Progress

Evaluation and Measuring Progress

Success in Trade for Trades is measured a bit differently compared to many other homeless/at-risk youth programs, which are purely focused on measuring a youth’s progress. Instead, outcomes are examined in three major areas:

  1. Number of youth completing the program and where they transition to afterwards.
  2. Successful completion of quality work for clients (including passing city inspections, maintaining COR certification, client satisfaction, number of completed projects).
  3. Progress towards self-sustainability (including projects being taken on independently that do not require additional funding support).

“[Be] very purposeful about what you’re trying to achieve on both outcomes. For us, it’s a bit of a scenario where if we don’t have high outcomes for young people but we have high outcomes in terms of quality of our work. What’s the point? If we have high outcomes for young people but don’t have quality work, pretty soon [there’s going to] be no point, because this is going to disappear. So really, it’s about designing a program that can achieve both.”  —Sheldon Pollett, Executive Director, Choices for Youth

A formal evaluation of the project was done in Years 1 and 2 only. This means that there has not been an extension evaluation completed in several years. As the project moves into a social enterprise model (beginning April 1st 2015) there will be an increased emphasis on evaluation. This will be discussed in the Changes to Train for Trades for 2015-2016 section.

As an important component of continuous program development and ensuring they achieve high outcomes for youth, program staff maintain a high level of connectivity with past clients, whether they completed the program or not. As a result, staff are able to monitor the short and long term outcomes for youth, year after year. The following is a summary analysis of youth outcomes:

Outcome

Count

%

Complete

48

61%

Incomplete

30

38%

Incomplete due to Medical Reasons

1

1%

Totals

79

100% 

Since its inception, Train for Trades has worked with 79 youth (with an additional 19 youth currently participating in the program. The above table represents youth who are no longer in the program. Of the 79 youth, 48 completed the program successfully, while the remaining 31 are categorized as ‘incomplete’, which includes youth who voluntarily left the program prematurely, left the program for medical reasons, or were deemed needing more support than the program could provide at the time.

Current Status

Count

%

Employed

32

41%

Post-Secondary

4

5%

Program Enrolment[3]

3

4%

Unemployed

37

47%

Deceased

1

1%

Unknown

2

3%

Totals

79

100%

At first glance, the table above shows a youth ‘success’ rate of just below 50%, if success is defined by either securing employment or pursuing post-secondary education of some sort. However, digging deeper into the numbers demonstrates a very clear distinction between participants who have completed and those who have not completed the program, as outlined in the tables below.

Status of Participants Who Are No Longer with Train for Trades

Completed

%

Incomplete

%

Employed

26

55%

6

19%

Post-Secondary

4

9%

0

0%

Program Enrolment

1

2%

2

6%

Unemployed

13

28%

23

72%

Deceased

1

2%

0

0%

Unknown

2

4%

0

0%

Incomplete Due to Medical Reasons

0

0%

1

3%

Totals

47

100%

32

100%

The impact that participating in the Train for Trades Program has on the outcomes of young people is clear when looking at post-exit outcomes detailed in the tables above. When comparing youth who have completed the program to youth who have not, the pathways are very different. Of the youth who have completed, 66% have gone on to either pursue post-secondary education, secure employment, or transitioned into the next appropriate program, which is the true measure of outcome success for the Train for Trades team. Conversely, of the youth who have not completed the program, only 19% have secured employment; 72% are unemployed, with none of these youth pursuing educational opportunities.

Homeless Hub Thoughts:

The program itself is very successful and is clearly able to measure success through youth achievement. However, there is no formal assessment process in place, which is a weakness and presents risk to the ongoing success of the program. Funders, in particular, like numbers. They like to be able to concretely measure success and achievement to establish that their money is being used efficiently and effectively.

All data collected at T4T is done by staff. Every six months they follow-up with youth who have left the program by phone call, text or by reaching out to family/friends. When connection is made with a youth, staff get current contact information and obtain a progress update on the youth’s activities. Staff also get together and discuss current and past youth. The information is entered into an Excel spreadsheet so that it is current.

To us, this is merely a baseline of what needs to be done and as a result, we would classify Train for Trades as a promising practice. To obtain best practice status, agencies must be subject to evaluation, especially from external evaluators. The successes (and failures) must be examined. Successful replication of a program is also key to moving from an emerging or promising practice into a best practice.

An intervention is considered to be a promising practice when there is sufficient evidence to claim that the practice is proven effective at achieving a specific aim or outcome, consistent with the goals and objectives of the activity or program. Ideally, promising practices demonstrate their effectiveness through the most rigorous scientific research, however there is not enough generalizable evidence to label them ‘best practices’. They do however hold promise for other organizations and entities that wish to adapt the approaches based on the soundness of the evidence. For a more complete discussion of the differences between best, promising and emerging practices see: What Works and For Whom? A Framework for Promising Practices published by the Homeless Hub.

We also believe that data management software (as discussed in the Supports section) is important to help track data. Beyond the casual updates every six months we would also see value in extensive exit interviews with youth to determine what worked and did not work for them, as well as qualitative interviews, surveys and focus groups with staff and youth to help evaluate the program.

Evaluation is often an afterthought to program delivery. While funders want numbers and proof of success, they are also reluctant to fund extensive evaluation, which makes it hard for agencies to carry out the level of evaluation necessary. We encourage agencies to budget for evaluation funding and to work with academic partners in the community to obtain evaluations of their programs.


[3] Enrolled in a training or day program at Choices for Youth or another agency.

Train for Trades uses the slogan “On Time. On Code. On Budget.” as part of their evaluation process. This means that they finish their construction projects as scheduled while still meeting all code requirements and inspections. They also, especially as they move towards the social enterprise model, work to meet budget forecasts. A detailed analysis of how the construction budgets are reviewed is examined in the Changes to T4T for 2015/16 section.

Since 2008, the following work has been completed:

  • 240 basement retrofits for Newfoundland Labrador Housing Corporation
  • 4 M&I's (Modernization & Improvement) renovations completed, 2 almost complete (also for Newfoundland Labrador Housing Corporation)
  • Lilly Building Renovated for Choices for Youth
  • Duckworth Street Renovated for Choices for Youth
  • St. John's Native Friendship Centre Renovated
  • 24 X 30 Foot Extension Completed for MacMorran Community Centre
  • 16 X 20 ft. Garage Complete for Private Customer
  • 40 x 60 ft. horse barn completed for Waypoints
  • Various work completed for private construction company 

Train for Trades has had its Certificate of Recognition™ status since 2010.  As discussed previously, COR™ is a certification given to employers in the construction industry by the Newfoundland Labrador Construction Safety Association. Maintaining the standards of the COR™ requirements is key to the success of the program. The fact that T4T is able to renew their certification annually is a significant measurement of success.

As the biggest client of Train for Trades, the fact that the Newfoundland Labrador Housing Corporation (NLHC) has continued to renew its partnership and in fact, has expanded the scope of the work speaks to the success of the program. Dennis Kendell, Regional Operations Executive Director at NLHC, was very clear that the work being done is of high quality and meets both city and NLHC inspections.

Someone might buy something or a service from a social enterprise once because it’s a good cause. But the only way – just like any other business – they’re coming back two, three, four, ten, twenty times is if you do good quality work. You think about Newfoundland Labrador Housing for example. We’ve done a little over 240 units of energy retrofit at this point. All lived in, occupied homes. We’ve had zero complaints and have been consistently recognized for the quality of our work being as good, or better, than work done by the private sector in the same area.” —Sheldon Pollett, Executive Director, Choices for Youth

Train for Trades has been working towards sustainability for some time. The various independent projects they have taken on are key to moving towards self-sufficiency. The ultimate goal for sustainability in a social enterprise is to become 100% self-funding. That is, no government grants, donations or corporate funding would be required to operate the program.

In terms of evaluation, sustainability means looking to see how the program can grow and expand to increase its revenue streams. Additionally, examining the program to see what cuts can be made while still maintaining the success of the program is also important.

Sometimes, ongoing funding from government funders can make up a component of the budget. For example, Train for Trades provides Personal Protective Equipment and tools to youth. Some social assistance and job-seekers programs provide funding for start-up costs, which would be a way of defraying the overall costs.

Keep in mind that complete self-sufficiency is unlikely to happen immediately. Train for Trades received government and corporate funding that has enabled it to operate. While grants can make a program sustainable in the short-term, the unstable nature of funding means that depending upon them is risky.

Certainly this risk has been seen at Train for Trades. The lessons learned, and the progress towards sustainability, have prompted a recent shift in the program to address funding challenges that exist within the organization. These will be discussed in the Changes to T4T for 2015-2016 section.

Another way of measuring success is to look at impact in sharing the story of the work being done. Train for Trades has been recognized as a successful program/best practice numerous times including:

You can’t get a better opportunity for youth.” — Dylan, age 22, Tier 3 participant

Homeless Hub Thoughts:

It is important to be able to prove the success of a project in order to access grants and government funding. With a social enterprise, the method of evaluation may be different than in typical youth programs because the outcomes include both the work itself and the progress the youth have made. Your program will fail if you only have good outcomes in one area and not the other.

Much of this work can be measured simply – did this happen or did it not happen?

  • How many youth started and finished the program?
  • How many youth attended X workshop? X training class?
  • How many youth received their GED?
  • How many youth were accepted to post-secondary education or full-time employment?

However, it is advisable to develop a formal case management system to record the progress of youth though the program. Having a formal follow-up system in place with regular check-in points (i.e. six months after completion, a year after completion) would be useful for measuring long-term success of the intervention. Pre and post skills-based assessment surveys would also be useful to measure progress. While T4T has an in-depth application and interview process that can be used to establish a baseline for the youth they do not have a formal case management system nor do they complete post-training assessments. As discussed in the evaluation section, these can be extremely valuable. 

Table of Contents

Supporting Documents

The Canadian Observatory on Homelessness acknowledges with thanks the financial support of The Home Depot Canada Foundation. Thanks to the staff, partners and service users (past and present) of Choices for Youth and Train for Trades who assisted in the development of the toolkit by taking part in interviews, providing data and resources or reviewing information.

The Home Depot Canada Foundation logo
Media Folder: