Changes to Evaluation

Evaluation and Measuring Progress

As an important component of continuous program development and ensuring they achieve high outcomes for youth, program staff maintain a high level of connectivity with past clients, whether they completed the program or not. As a result, staff are able to monitor the short and long term outcomes for youth, year after year. The following is a summary analysis of youth outcomes:

Outcome

Count

%

Complete

48

61%

Incomplete

30

38%

Incomplete due to Medical Reasons

1

1%

Totals

79

100% 

Since its inception, Train for Trades has worked with 79 youth (with an additional 19 youth currently participating in the program. The above table represents youth who are no longer in the program. Of the 79 youth, 48 completed the program successfully, while the remaining 31 are categorized as ‘incomplete’, which includes youth who voluntarily left the program prematurely, left the program for medical reasons, or were deemed needing more support than the program could provide at the time.

Current Status

Count

%

Employed

32

41%

Post-Secondary

4

5%

Program Enrolment[3]

3

4%

Unemployed

37

47%

Deceased

1

1%

Unknown

2

3%

Totals

79

100%

At first glance, the table above shows a youth ‘success’ rate of just below 50%, if success is defined by either securing employment or pursuing post-secondary education of some sort. However, digging deeper into the numbers demonstrates a very clear distinction between participants who have completed and those who have not completed the program, as outlined in the tables below.

Status of Participants Who Are No Longer with Train for Trades

Completed

%

Incomplete

%

Employed

26

55%

6

19%

Post-Secondary

4

9%

0

0%

Program Enrolment

1

2%

2

6%

Unemployed

13

28%

23

72%

Deceased

1

2%

0

0%

Unknown

2

4%

0

0%

Incomplete Due to Medical Reasons

0

0%

1

3%

Totals

47

100%

32

100%

The impact that participating in the Train for Trades Program has on the outcomes of young people is clear when looking at post-exit outcomes detailed in the tables above. When comparing youth who have completed the program to youth who have not, the pathways are very different. Of the youth who have completed, 66% have gone on to either pursue post-secondary education, secure employment, or transitioned into the next appropriate program, which is the true measure of outcome success for the Train for Trades team. Conversely, of the youth who have not completed the program, only 19% have secured employment; 72% are unemployed, with none of these youth pursuing educational opportunities.

Homeless Hub Thoughts:

The program itself is very successful and is clearly able to measure success through youth achievement. However, there is no formal assessment process in place, which is a weakness and presents risk to the ongoing success of the program. Funders, in particular, like numbers. They like to be able to concretely measure success and achievement to establish that their money is being used efficiently and effectively.

All data collected at T4T is done by staff. Every six months they follow-up with youth who have left the program by phone call, text or by reaching out to family/friends. When connection is made with a youth, staff get current contact information and obtain a progress update on the youth’s activities. Staff also get together and discuss current and past youth. The information is entered into an Excel spreadsheet so that it is current.

To us, this is merely a baseline of what needs to be done and as a result, we would classify Train for Trades as a promising practice. To obtain best practice status, agencies must be subject to evaluation, especially from external evaluators. The successes (and failures) must be examined. Successful replication of a program is also key to moving from an emerging or promising practice into a best practice.

An intervention is considered to be a promising practice when there is sufficient evidence to claim that the practice is proven effective at achieving a specific aim or outcome, consistent with the goals and objectives of the activity or program. Ideally, promising practices demonstrate their effectiveness through the most rigorous scientific research, however there is not enough generalizable evidence to label them ‘best practices’. They do however hold promise for other organizations and entities that wish to adapt the approaches based on the soundness of the evidence. For a more complete discussion of the differences between best, promising and emerging practices see: What Works and For Whom? A Framework for Promising Practices published by the Homeless Hub.

We also believe that data management software (as discussed in the Supports section) is important to help track data. Beyond the casual updates every six months we would also see value in extensive exit interviews with youth to determine what worked and did not work for them, as well as qualitative interviews, surveys and focus groups with staff and youth to help evaluate the program.

Evaluation is often an afterthought to program delivery. While funders want numbers and proof of success, they are also reluctant to fund extensive evaluation, which makes it hard for agencies to carry out the level of evaluation necessary. We encourage agencies to budget for evaluation funding and to work with academic partners in the community to obtain evaluations of their programs.


[3] Enrolled in a training or day program at Choices for Youth or another agency.

Train for Trades uses the slogan “On Time. On Code. On Budget.” as part of their evaluation process. This means that they finish their construction projects as scheduled while still meeting all code requirements and inspections. They also, especially as they move towards the social enterprise model, work to meet budget forecasts. A detailed analysis of how the construction budgets are reviewed is examined in the Changes to T4T for 2015/16 section.

Since 2008, the following work has been completed:

  • 240 basement retrofits for Newfoundland Labrador Housing Corporation
  • 4 M&I's (Modernization & Improvement) renovations completed, 2 almost complete (also for Newfoundland Labrador Housing Corporation)
  • Lilly Building Renovated for Choices for Youth
  • Duckworth Street Renovated for Choices for Youth
  • St. John's Native Friendship Centre Renovated
  • 24 X 30 Foot Extension Completed for MacMorran Community Centre
  • 16 X 20 ft. Garage Complete for Private Customer
  • 40 x 60 ft. horse barn completed for Waypoints
  • Various work completed for private construction company 

Train for Trades has had its Certificate of Recognition™ status since 2010.  As discussed previously, COR™ is a certification given to employers in the construction industry by the Newfoundland Labrador Construction Safety Association. Maintaining the standards of the COR™ requirements is key to the success of the program. The fact that T4T is able to renew their certification annually is a significant measurement of success.

As the biggest client of Train for Trades, the fact that the Newfoundland Labrador Housing Corporation (NLHC) has continued to renew its partnership and in fact, has expanded the scope of the work speaks to the success of the program. Dennis Kendell, Regional Operations Executive Director at NLHC, was very clear that the work being done is of high quality and meets both city and NLHC inspections.

Someone might buy something or a service from a social enterprise once because it’s a good cause. But the only way – just like any other business – they’re coming back two, three, four, ten, twenty times is if you do good quality work. You think about Newfoundland Labrador Housing for example. We’ve done a little over 240 units of energy retrofit at this point. All lived in, occupied homes. We’ve had zero complaints and have been consistently recognized for the quality of our work being as good, or better, than work done by the private sector in the same area.” —Sheldon Pollett, Executive Director, Choices for Youth

Train for Trades has been working towards sustainability for some time. The various independent projects they have taken on are key to moving towards self-sufficiency. The ultimate goal for sustainability in a social enterprise is to become 100% self-funding. That is, no government grants, donations or corporate funding would be required to operate the program.

In terms of evaluation, sustainability means looking to see how the program can grow and expand to increase its revenue streams. Additionally, examining the program to see what cuts can be made while still maintaining the success of the program is also important.

Sometimes, ongoing funding from government funders can make up a component of the budget. For example, Train for Trades provides Personal Protective Equipment and tools to youth. Some social assistance and job-seekers programs provide funding for start-up costs, which would be a way of defraying the overall costs.

Keep in mind that complete self-sufficiency is unlikely to happen immediately. Train for Trades received government and corporate funding that has enabled it to operate. While grants can make a program sustainable in the short-term, the unstable nature of funding means that depending upon them is risky.

Certainly this risk has been seen at Train for Trades. The lessons learned, and the progress towards sustainability, have prompted a recent shift in the program to address funding challenges that exist within the organization. These will be discussed in the Changes to T4T for 2015-2016 section.

Another way of measuring success is to look at impact in sharing the story of the work being done. Train for Trades has been recognized as a successful program/best practice numerous times including:

You can’t get a better opportunity for youth.” — Dylan, age 22, Tier 3 participant

Homeless Hub Thoughts:

It is important to be able to prove the success of a project in order to access grants and government funding. With a social enterprise, the method of evaluation may be different than in typical youth programs because the outcomes include both the work itself and the progress the youth have made. Your program will fail if you only have good outcomes in one area and not the other.

Much of this work can be measured simply – did this happen or did it not happen?

  • How many youth started and finished the program?
  • How many youth attended X workshop? X training class?
  • How many youth received their GED?
  • How many youth were accepted to post-secondary education or full-time employment?

However, it is advisable to develop a formal case management system to record the progress of youth though the program. Having a formal follow-up system in place with regular check-in points (i.e. six months after completion, a year after completion) would be useful for measuring long-term success of the intervention. Pre and post skills-based assessment surveys would also be useful to measure progress. While T4T has an in-depth application and interview process that can be used to establish a baseline for the youth they do not have a formal case management system nor do they complete post-training assessments. As discussed in the evaluation section, these can be extremely valuable. 

  • Understand your community and the needs within in. What are the issues faced by youth in your area? What services exist already to support them? How can you create an opportunity that is unique but builds upon or expands offerings already in the community?
  • Understand the level of complexity in the population that you are targeting and develop the appropriate level of staff supports. Train for Trades uses an Intensive Case Management model to support the varied needs of their youth, including mental health issues, addictions, family breakdown, post-child welfare system, interactions with the criminal justice system etc.
  • Ensure that staff understand how to deal with conflict effectively and have good group management skills.
  • Create a quality product/deliver a quality service. You will not gain repeat business if your program does not deliver what it promises.
  • Determine what kind of program is going to provide the highest chances for later employment success for the youth. Is there a skilled labour shortage in a specific area? Do not create a construction program if there are too many unemployed construction workers already. Maybe the program could be food/catering, sewing, childcare, courier services etc.
  • Be very purposeful in developing successful outcomes for young people and delivering high quality work. Both have to be achieved for the project to be considered successful.
  • Educate potential partners and allow adequate time to bring everyone on board (i.e. union). This will ensure that everyone understands the benefits and that all fears have been addressed.
  • Start small and scale up. Learn what works and what does not. Take the time to work out any issues before trying to get too big. At the same time, plan for scale and understand that your initial resources are finite and you only have a certain amount of time to figure out how to make your program operational and sustainable. Do not simply chase funds – plan to make your program a social enterprise from the beginning.
  • Ensure you have sufficient start-up funds to cover all of the unique costs.
  • Reap maximum benefits – figure out the ways in which your program can contribute to the greater good while providing high quality supports for young people.
  • Do not lose track of the fact that this is a support program before it is an employment program. At the same time, do not forget this is an employment program and the work should be meaningful.
  • Make safety a priority (especially for this kind of work) including proper Personal Protective Equipment training and supervision.
  • Ensure that within your staff team you have someone with the ability to manage the business side and someone with the ability to manage the project in terms of scheduling, sub-contracting, material management etc.  More than a third of your time is often lost on construction projects due to scheduling.
  • Hire sub-contractors directly who understand the program and who are willing to mentor youth.
  • Have staff with a balance of youth worker skills and carpentry/construction skills, so they can manage both sides of the work.
  • Recognize and address the high start-up cost to the youth (i.e. transportation, Personal Protective Equipment, work clothes and safety equipment, etc.) 
  • A program such as this provides an opportunity to reduce the stigma that is often attached to youth and to people who live in social housing. It helps build a partnership with the community.
  • It gives teenagers and young adults the chance and the support they may need to better themselves. The high start-up costs mentioned above prohibit many individuals from entering the work force even if they have the skills/motivation to do so. When they are unsure about direction or how to navigate the system they often cannot get started without assistance.
  • A program that accepts applicants with significant barriers provides opportunities for young people who have been marginalized from the work force and who face challenges that are difficult to overcome. For example, many youth struggle to find employment when they have a criminal record.
  • The program creates a sense of autonomy, identity, empowerment and pride in the youth that it serves.
  • In many communities there is a significant skilled labour/trades shortage. Creating skilled employees who go on to further school or apprenticeships helps address that issue.
  • For a training institution, this program reaches an audience that might not normally be reached and gives them an opportunity to learn about and consider post-secondary education.
  • For a housing provider, it is a win in multiple areas. You are helping tenants and the organization because you are reducing long-term renovation and repair costs (i.e. due to mould prevention). At the same time, you are helping youth.

The program has decreased from 20 youth to 10 youth effective April 1st, 2015 (four youth are currently transitioning out of the project to meet the goal of 10).  Sheldon Pollett, Choices for Youth’s Executive Director says, “In the future, like any contracting company, or business, we will scale up and down according to the contact work we have on hand. We intend however, to use 10 [youth] as our base”. If new projects are obtained, the model allows for an increase in the number of youth hired. 

The tier model will continue to exist, but the youth will be merged into mixed-tier construction teams so that new Tier 1 youth can be mentored by youth in Tiers 2 and 3. This will hopefully make all of the teams more productive by reducing the overall learning curve of the project each time new youth are hired.

The work will be specialized to skill level, so teams with more experience/expertise will take on more of the individual and Modernization & Improvement contracts, while newer/more inexperienced teams will focus on the basement retrofits. While this sounds similar to the existing process, the biggest difference will be the mixing of tiers in each team with senior youth mentoring newer youth. 

T4T is a support program and an employment program. On the support side the goals has been to support a youth’s transition into the labour force by providing intensive case management supports. On the employment side, these changes also bring productivity to the forefront of operations. This models real world work environments to ensure youth are more successful when they transitional out of Train for Trades.  

Since the changes include a reduction in the number of youth served at one time there is a resultant modification to the staffing model. The position of the Construction/Site manager has been made redundant and the responsibilities will be shouldered by the Project Manager position. By having a Project Manager position and hiring an individual with a background in construction and subcontracting, project management and large scale construction projects, the goal is to bring more industry knowledge to the social enterprise concept. This individual will also have strong insight into the expectations of the industry and can help ensure output productivity meets an industry standard.

One of the two Youth Supports Coordinators positions is being made redundant. As the number of youth has decreased from 20 to 10, it is expected that one individual can coordinate and provide all necessary support to the youth.

One of the four Youth Support Worker positions was also made redundant. The three remaining positions were renamed “Lead Hand”. The new job descriptions for these positions reflect a greater range of experience in construction projects, while still combining youth support skills. 

In this new model, with 10 youth and six staff directly supporting youth, the ratio will be 1.66 youth to 1 staff. 

The Manager of Social Enterprise position is also being eliminated.  The Project Manager, the Fund Development and Communications Department and the Finance Department will assist the Project Coordinator in the roll out of the business plan.

See new Job Descriptions

Post-Hibernia fund expansion and the 2015-16 fiscal year saw the maturation into a social enterprise budget which captures the difference between earned income and untied funding, as well as a contribution margin analysis (these terms will be explained below).

In this model the proposed budget sits at $1,679,450.

Earned Revenues refers to money that is obtained through payments for carrying out the construction work. This is estimated to be $1.3 million and includes:

  • $543,000 (32% of revenue) in government contracts from Newfoundland Labrador Housing Corporation for basements and retrofits
  • $662,000 (39% of revenue) from Choices for Youth to cover a portion of a major capital build as well as ongoing maintenance of CFY buildings
  • $174,000 (10% of revenue) from private sector projects.

Untied Funding refers to grant funding that is not linked to a specific construction project. This revenue equals  $300,000 (18% of all revenues) and includes:

  • $250,000 proposed investment from Advanced Education and Skills
  • $50,000 draw down on the Hibernia fund (the remaining balance becomes a legacy account to serve as a contingency against future business risks).

Expenditures are separated into Cost of Goods sold (variable costs), enterprise costs, social program costs and administrative overhead and total $1.6 million. The budget therefore becomes balanced.

Variable Costs of $1.2million (or 72% of total budget), include:

  • 28% for direct labour costs of youth and Lead Hands allocated to projects
  • 33% for sub contractors on projects
  • 37% in project materials
  • ~1% for variable overhead (i.e. Insurance on specific jobs)

Enterprise Costs include fixed costs of operating the training program. This comes to $ 206,000 or 12% of the total expense budget. These include salaries for project management and business management, training, travel, IT, etc. This is similar to the 2011-12 budget.

Social Program Costs include fixed costs for operation of the social program. This is basically the direct, one-on-one support a youth can access through the enterprise and include a Youth Supports Coordinator, the GED program as well as some small operating expenses. The total cost is $72,000 or 4%.

Administrative Costs (similar to the 2011-12) includes office rental and administrative fees, but also includes the Coordinator and Administrative Assistant. The total cost is $192,000 or 11%.

There are two important metrics for T4T - the percentage spent on program versus administration and the contribution margin.

Program versus Administrative Costs: Since the variable costs and enterprise costs together make up the overall training program, together with the social program costs it can be said that 89% is spent on running the training program and providing supports. The administrative costs are 11%.

Contribution Margin: By taking the earned income only and subtracting the variable costs, the contribution margin can be established. This is to determine the ability of the enterprise to generate enough revenues to cover costs and attain sustainability. 

The contribution margin is:

  • Earned revenues: $1,379,450
  • Cost of Goods Sold: $1,209,457 (72%)
  • Contribution Margin: $169,994 (18%)

From the perspective of the Finance Department, any future project proposed by Train for Trades must meet or exceed the target of 18% Contribution Margin. There will be some flexibility if the project proves to be a great learning opportunity, but the goal is to meet or exceed this target annually.

Since the contribution margin is less then our carrying fixed and overhead costs, the untied funding is required for 2015-16. However, as the enterprise continues to develop, it is anticipated that enough work will be generated that T4T will be able to cover off the $300,000 in untied funding through growth in government or private sector contracts. Current projections suggest that T4T will be completely sustainable and free of untied funds within the next five years.

It is at this time that the enterprise will begin turning a profit (in the traditional sense) and producing revenues to be reinvested into future enterprise growth or the wider Choices organization.

As discussed in the Evaluation section, Train for Trades has not had a full formal evaluation since the first two years of operation. They intend to evaluate T4T as a social enterprise at the end of the first year. This will have two components: a financial evaluation and a programmatic evaluation.

The financial evaluation will be based on examining the success of T4T as a business model and creating a financial forecast for the next year. The programmatic evaluation will include qualitative research with youth and partners.

Homeless Hub Thoughts:

Train for Trades operated as a typical youth program for a number of years, with a high dependence on corporate and government grants. It developed a strong track record of success both on the youth support/employment side and on the construction end. As such, a decision was made to turn it into a social enterprise.

Running a social enterprise is a good way of doing business to reduce or eliminate reliance on outside funding for sustainability.  While the move towards self-sustainability and the social enterprise model was fast-tracked, T4T and Choices for Youth had always intended to move in that direction.

It is important to note that in the new model:

  • The level of youth support has actually increased. The staff to youth ratio moves from 2.5 to 1.6 staff for every youth. T4T has ensured that youth do not lose out by this restructuring.
  • The changes are designed to make T4T more productive and more sustainable. Hiring a Project Manager with significant construction management experience not only will improve job performance but will increase the future employability of youth by giving them real world work experience.
  • The program is designed to scale. If more contracts are obtained, it is easy to increase the number of staff and youth. If the existing number of contracts/workload are maintained, self-sufficiency is still a reality given the lower number of staff/youth.
  • With its focus on social enterprise, it is possible that T4T will become a revenue generator – not just revenue neutral – allowing it to build a buffer zone for slower work climates, or to support expansion.

It is important to note however, the recommendation from staff at CFY/T4T is that agencies wanting to develop their own employment program consider a pilot project first and use the pre-2015/16 model to develop their program and figure out how best to make it successful.

We again emphasize the importance of evaluation, especially during such a transition period. An extensive external evaluation that includes both quantitative and qualitative assessment will really help establish the success of Train for Trades as it moves forward with its social enterprise model. 

Table of Contents

Supporting Documents

The Canadian Observatory on Homelessness acknowledges with thanks the financial support of The Home Depot Canada Foundation. Thanks to the staff, partners and service users (past and present) of Choices for Youth and Train for Trades who assisted in the development of the toolkit by taking part in interviews, providing data and resources or reviewing information.

The Home Depot Canada Foundation logo
Media Folder: